## System Requirements {#sec-running-nixos-tests-requirements}
NixOS tests require virtualization support.
NixOS tests using QEMU virtual machine [`nodes`](#test-opt-nodes)require virtualization support.
This means that the machine must have `kvm` in its [system features](https://nixos.org/manual/nix/stable/command-ref/conf-file.html?highlight=system-features#conf-system-features) list, or `apple-virt` in case of macOS.
These features are autodetected locally, but `apple-virt` is only autodetected since Nix 2.19.0.
Features of **remote builders** must additionally be configured manually on the client, e.g. on NixOS with [`nix.buildMachines.*.supportedFeatures`](https://search.nixos.org/options?show=nix.buildMachines.*.supportedFeatures&sort=alpha_asc&query=nix.buildMachines) or through general [Nix configuration](https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds).
If you run the tests on a **macOS** machine, you also need a "remote" builder for Linux; possibly a VM. [nix-darwin](https://daiderd.com/nix-darwin/) users may enable [`nix.linux-builder.enable`](https://daiderd.com/nix-darwin/manual/index.html#opt-nix.linux-builder.enable) to launch such a VM.
NixOS tests using `systemd-nspawn`[`containers`](#test-opt-containers) require the Nix daemon to be
This applies to `systemctl`, `get_unit_info`, `wait_for_unit`,
`start_job` and `stop_job`.
For faster dev cycles it's also possible to disable the code-linters
(this shouldn't be committed though):
```nix
{
skipLint=true;
nodes.machine=
{config,pkgs,...}:
{
# configuration…
};
testScript=''
Python code…
'';
}
```
This will produce a Nix warning at evaluation time. To fully disable the
linter, wrap the test script in comment directives to disable the Black
linter directly (again, don't commit this within the Nixpkgs
repository):
```nix
{
testScript=''
# fmt: off
Python code…
# fmt: on
'';
}
```
Similarly, the type checking of test scripts can be disabled in the following
way:
```nix
{
skipTypeCheck=true;
nodes.machine=
{config,pkgs,...}:
{
# configuration…
};
}
```
## Failing tests early {#ssec-failing-tests-early}
### Failing tests early {#ssec-failing-tests-early}
To fail tests early when certain invariants are no longer met (instead of waiting for the build to time out), the decorator `polling_condition` is provided. For example, if we are testing a program `foo` that should not quit after being started, we might write the following:
@@ -255,7 +303,7 @@ def foo_running():
machine.succeed("pgrep -x foo")
```
## Adding Python packages to the test script {#ssec-python-packages-in-test-script}
### Adding Python packages to the test script {#ssec-python-packages-in-test-script}
When additional Python libraries are required in the test script, they can be
added using the parameter `extraPythonPackages`. For example, you could add
@@ -279,6 +327,61 @@ added using the parameter `extraPythonPackages`. For example, you could add
In that case, `numpy` is chosen from the generic `python3Packages`.
### Linting and type checking test scripts {#ssec-test-script-checks}
Test scripts are automatically linted with
[Pyflakes](https://pypi.org/project/pyflakes/) and type-checked with
[Mypy](https://mypy.readthedocs.io/en/stable/).
If there are any linting or type checking errors, the test will
fail to evaluate.
For faster dev cycles it's also possible to disable the code-linters
(this shouldn't be committed though):
```nix
{
skipLint=true;
nodes.machine=
{config,pkgs,...}:
{
# configuration…
};
testScript=''
Python code…
'';
}
```
This will produce a Nix warning at evaluation time. To fully disable the
linter, wrap the test script in comment directives to disable the Black
linter directly (again, don't commit this within the Nixpkgs
repository):
```nix
{
testScript=''
# fmt: off
Python code…
# fmt: on
'';
}
```
Similarly, the type checking of test scripts can be disabled in the following
way:
```nix
{
skipTypeCheck=true;
nodes.machine=
{config,pkgs,...}:
{
# configuration…
};
}
```
## Overriding a test {#sec-override-nixos-test}
The NixOS test framework returns tests with multiple overriding methods.
@@ -297,7 +400,7 @@ The NixOS test framework returns tests with multiple overriding methods.
: Evaluates the test with additional NixOS modules and/or arguments.
`module`
: A NixOS module to add to all the nodes in the test. Sets test option [`extraBaseModules`](#test-opt-extraBaseModules).
: A NixOS module to add to all the machines in the test. Sets test option [`extraBaseModules`](#test-opt-extraBaseModules).
`specialArgs`
: An attribute set of arguments to pass to all NixOS modules. These override the existing arguments, as well as any `_module.args.<name>` that the modules may define. Sets test option [`node.specialArgs`](#test-opt-node.specialArgs).
@@ -345,29 +448,20 @@ list-id: test-options-list
source: @NIXOS_TEST_OPTIONS_JSON@
```
## Accessing VMs in the sandbox with SSH {#sec-test-sandbox-breakpoint}
::: {.note}
For debugging with SSH access into the machines, it's recommended to try using
[the interactive driver](#sec-running-nixos-tests-interactively) with its
[SSH backdoor](#sec-nixos-test-ssh-access) first.
This feature is mostly intended to debug flaky test failures that aren't
reproducible elsewhere.
:::
As explained in [](#sec-nixos-test-ssh-access), it's possible to configure an
SSH backdoor based on AF_VSOCK. This can be used to SSH into a VM of a running
build in a sandbox.
## Debugging test machines {#sec-test-sandbox-breakpoint}
This can be done when something in the test fails, e.g.
You can set the [`enableDebugHook`](#test-opt-enableDebugHook) option to pause
a test on the first failure and have it print instructions on how to enter the
sandbox shell of the test. Suppose you have the following test module:
```nix
{
name="foo";
nodes.machine={};
sshBackdoor.enable=true;
enableDebugHook=true;
sshBackdoor.enable=true;
testScript=''
start_all()
@@ -376,31 +470,76 @@ This can be done when something in the test fails, e.g.
}
```
For the AF_VSOCK feature to work, `/dev/vhost-vsock` is needed in the sandbox
which can be done with e.g.
The test will fail with an output like this:
```
nix-build -A nixosTests.foo --option sandbox-paths /dev/vhost-vsock
vm-test-run-foo> !!! Breakpoint reached, run 'sudo /nix/store/eeeee-attach/bin/attach <PID>'
```
This will halt the test execution on a test-failure and print instructions
on how to enter the sandbox shell of the VM test. Inside, one can log into
e.g. `machine` with
You can then enter the sandbox shell:
```
ssh -F ./ssh_config vsock/3
$ sudo /nix/store/eeeee-attach/bin/attach <PID>
bash#
```
There, you can attach to a [`pdb`](https://docs.python.org/3/library/pdb.html) session
to step through the Python test script:
```
bash# telnet 127.0.0.1 4444
pdb$
```
Note that it is also possible to set breakpoints in the test script using `debug.breakpoint()`.
### SSH access to test VMs {#sec-test-vm-ssh-access}
::: {.note}
For debugging with SSH access into the machines, it's recommended to try using
[the interactive driver](#sec-running-nixos-tests-interactively) with its
[SSH backdoor](#sec-nixos-test-ssh-access) first.
This feature is mostly intended to debug flaky test failures that aren't
reproducible elsewhere.
:::
If you set the [`sshBackdoor.enable`](#test-opt-sshBackdoor.enable) option,
QEMU virtual machines will open an SSH backdoor based on AF_VSOCK
(see [](#sec-nixos-test-ssh-access)).
Once you are in the sandbox shell, you can access the VMs (for example, `machine`)
with SSH over vsock:
```
bash# ssh -F ./ssh_config vsock/3
```
For the AF_VSOCK feature to work, `/dev/vhost-vsock` is needed in the sandbox
which can be done with e.g.
```
nix-build -A nixosTests.foo --option sandbox-paths /dev/vhost-vsock
```
As described in [](#sec-nixos-test-ssh-access), the numbers for vsock start at
`3` instead of `1`. So the first VM in the network (sorted alphabetically) can
be accessed with `vsock/3`.
Alternatively, it's possible to explicitly set a breakpoint with
`debug.breakpoint()`. This also has the benefit, that one can step through
`testScript` with `pdb` like this:
### SSH access to test containers {#sec-test-container-ssh-access}
If you set the [`sshBackdoor.enable`](#test-opt-sshBackdoor.enable) option,
each `systemd-nspawn` container will open an SSH backdoor.
Once the container starts,
it will print instructions on how to log into the container via SSH.
If the test fails,
attach to the sandbox as described above,
and then use the provided SSH command to log into the container.
