Commit graph

855 commits

Author SHA1 Message Date
Ryan Lahfa
5a7e90bc95
Merge pull request #228801 from NixOS/make-disk-image-faster
nixos/lib/make-disk-image: do not compile a full arch QEMU to convert images
2023-05-15 13:24:08 +02:00
Robert Hensing
5c3e59b6d6
Merge pull request #230523 from hercules-ci/fast-nixos-test-eval
Fast nixos test eval
2023-05-11 17:34:46 +02:00
Robert Hensing
b0e17891f2 nixos/testing/nodes.nix: Do not rely on disabledModules
It's just not necessary.
2023-05-11 16:24:01 +02:00
Robert Hensing
0f83261f0e nixos/testing: Add node.pkgsReadOnly escape hatch
By adding this option indirection, a test can declare all by itself
that it needs a custom nixpkgs. This is a more convenient way of
going about this when the caller of the test framework receives a
`node.pkgs` unconditionally.
2023-05-11 16:24:01 +02:00
Robert Hensing
f659db7ba2 nixos/testing: Add node.pkgs option
By factoring out this logic, it's easier for other projects to make
use of it this optimization too (and do it correctly).
2023-05-11 16:24:00 +02:00
Robert Hensing
b213791e7e nixos/all-tests.nix: Add readOnlyPkgs module 2023-05-11 16:24:00 +02:00
Robert Hensing
cd358fe24e nixos/all-tests.nix: Set nixpkgs.system 2023-05-10 15:55:09 +02:00
Robert Hensing
693e2c3287 nixos/eval-config: Remove statically known mkIf
mkIf is unnecessary when the condition is statically known - that is
knowable before entering the module evaluation.

By changing this to a precomputed module, we support changing the
defined options to readOnly options.
2023-05-10 15:55:09 +02:00
Jacek Galowicz
b7dfa5082a
Merge pull request #228220 from R-VdP/test_driver_guest_shell_timeout
nixos-test-driver: include a timeout for the recv call, do not assume sh == bash
2023-05-09 13:05:28 +02:00
Robert Hensing
8054785157 lib/modules: Move class out of specialArgs 2023-05-06 18:32:58 +02:00
Robert Hensing
79703eef08 nixos,nixpkgs: Add module classes
This allows modules that declare their class to be checked.
While that's not most user modules, frameworks can take advantage
of this by setting declaring the module class for their users.
That way, the mistake of importing a module into the wrong hierarchy
can be reported more clearly in some cases.
2023-05-06 18:29:04 +02:00
Raito Bezarius
a22826f46a nixos/lib/make-(multi|single)-disk-zfs-image: use qemu_kvm (host arch) instead of qemu (all arches) 2023-04-28 23:50:44 +02:00
Raito Bezarius
77c0b522b7 nixos/lib/make-disk-image: do not compile a full arch QEMU to convert images 2023-04-28 23:40:57 +02:00
r-vdp
4147b878bc
nixos-test-driver: include a timeout for the recv call, do not assume sh == bash 2023-04-26 01:05:53 +02:00
Alyssa Ross
d6e84a4574
nixosTest: remove hostname limitations 2023-04-25 08:55:30 +00:00
Leon Barrett
15c760d6b8 nixos/make-disk-image: fix contents dir paths
`make-disk-image` is a tool for creating VM images. It takes an argument
`contents` that allows one to specify files and directories that should
be copied into the VM image. However, directories end up not at the
specified target, but instead at a subdirectory of the target, with a
nix-store-like path, e.g.
`/target/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-source`. See issue
https://github.com/NixOS/nixpkgs/issues/226203 .

This change adds a test for make-disk-image's contents directory
handling and adds a fix (appending `/` to rsync input directory names).

This closes issue https://github.com/NixOS/nixpkgs/issues/226203 .
2023-04-16 09:54:45 -07:00
Sandro
df9856c9b3
Merge pull request #222110 from SuperSandro2000/root-slice
systemd-lib: fix building -.slice (root slice)
2023-03-23 14:10:24 +01:00
Alyssa Ross
27f97953e0
nixos/test-driver: accept non-\w* filenames
What the code was trying to do was helpfully add a directory and
extension if none were specified, but it did this by checking whether
the filename was composed of a very limited character set that didn't
even include dashes.

With this change, the intention of the code is clearer, and I can put
dashes in my screenshot names.
2023-03-22 15:54:06 +00:00
Sandro Jäckel
a10f3197fc
systemd-lib: fix building -.slice (root slice)
The error was:

dirname: invalid option -- '.'
Try 'dirname --help' for more information.
2023-03-20 01:11:23 +01:00
Astro
64a4ae6f37 nixos/test-driver: remove allow_reboot state from Machine, make it a start() kwarg 2023-03-16 01:50:15 +01:00
Astro
9fc85a85d5 nixos/test-driver: implement Machine.reboot() 2023-03-16 01:41:32 +01:00
Astro
95d121e798 nixos/test-driver: fix allow_reboot 2023-03-15 12:48:01 +01:00
Nick Bathum
519958c94b
nixos/lib/make-multi-disk-zfs-image: expose memSize parameter
closes #178095
2023-02-26 20:18:04 -05:00
Nick Bathum
59462da917
nixos/lib/make-disk-image: expose memSize parameter
Generating large disk images can fail from out-of-memory in vm-run.
2023-02-26 12:08:58 -05:00
pennae
417dd2ad16 nixos-render-docs: add options asciidoc converter
same reasoning as for the earlier commonmark converter.
2023-02-21 18:19:00 +01:00
pennae
4d3aef762f nixos-render-docs: add options commonmark converter
the old method of pasting parts of options.json into a markdown document
and hoping for the best no longer works now that options.json contains
more than just docbook. given the infrastructure we have now we can
actually render options.md properly, so we may as well do that.
2023-02-21 18:19:00 +01:00
pennae
81cf173256 nixos-render-docs: use multiprocessing for options
options processing is pretty slow right now, mostly because the
markdown-it-py parser is pure python (and with performance
pessimizations at that). options parsing *is* embarassingly parallel
though, so we can just fork out all the work to worker processes and
collect the results.

multiprocessing probably has a greater benefit on linux than on darwin
since the worker spawning method darwin uses is less efficient than
fork() on linux. this hasn't been tested on darwin, only on linux, but
if anything darwin will be faster with its preferred method.
2023-02-18 13:17:57 +01:00
K900
254426ec4b nixos/lib/testing: set default timeout for VM tests 2023-02-16 19:17:53 +03:00
pennae
8fe19590c3 nixos/make-options-doc: fix related packages link label
the pkgs path literal surely shouldn't include the title, right?
2023-02-08 15:23:34 +01:00
Jacek Galowicz
7f88d9c34c
Merge pull request #214910 from rnhmjoj/pr-gnupg-test
nixos/tests/gnupg: init
2023-02-07 09:17:06 +01:00
rnhmjoj
067d688b16
nixos/test-driver: handle decoding errors in Machine.execute
The output of a command is not guaranteed to be valid UTF-8, so the
decoding can fail raising UnicodeDecodeError. If this happens during a
`succeeds` the check will be erroneously marked failed.

This changes the error handling to the "replace" mode, where invalid
codepoints are replaced with � (REPLACEMENT CHARACTER U+FFFD) and the
decoding can go on.
2023-02-07 08:47:14 +01:00
rnhmjoj
f2929eb949
nixos/test-driver: drop logging from Machine.send_monitor_command
Several machine operations, like `send_chars` and `send_key`, are
implemented by calling `send_monitor_command`, possibly multiple times.
This generates a huge amount of unnecessary noise in the log, because
`send_monitor_command` is a low-level operation and an implementation
detail.

Here's an excerpt from a highlighted log before and afte the change.

Before:

    subtest: Can generate a PGP key
    machine: sending keys 'gpg --gen-key\n'
    machine: sending monitor command: sendkey g
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey g, in 0.00 seconds)
    machine: sending monitor command: sendkey p
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey p, in 0.00 seconds)
    machine: sending monitor command: sendkey g
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey g, in 0.00 seconds)
    machine: sending monitor command: sendkey spc
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey spc, in 0.00 seconds)
    machine: sending monitor command: sendkey 0x0C
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey 0x0C, in 0.00 seconds)
    machine: sending monitor command: sendkey 0x0C
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey 0x0C, in 0.00 seconds)
    machine: sending monitor command: sendkey g
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey g, in 0.00 seconds)
    machine: sending monitor command: sendkey e
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey e, in 0.00 seconds)
    machine: sending monitor command: sendkey n
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey n, in 0.00 seconds)
    machine: sending monitor command: sendkey 0x0C
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey 0x0C, in 0.00 seconds)
    machine: sending monitor command: sendkey k
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey k, in 0.00 seconds)
    machine: sending monitor command: sendkey e
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey e, in 0.00 seconds)
    machine: sending monitor command: sendkey y
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey y, in 0.00 seconds)
    machine: sending monitor command: sendkey ret
    machine: waiting for monitor prompt
    (finished: waiting for monitor prompt, in 0.00 seconds)
    (finished: sending monitor command: sendkey ret, in 0.00 seconds)
    (finished: sending keys 'gpg --gen-key\n', in 0.15 seconds)

After:

    subtest: Can generate a PGP key
    machine: sending keys 'gpg --gen-key\n'
    (finished: sending keys 'gpg --gen-key\n', in 0.15 seconds)
2023-02-07 08:40:54 +01:00
Alyssa Ross
32d066dce3
channel: add --show-trace to nix-env command
Otherwise, when this fails on Hydra, we have no way of seeing what
went wrong.
2023-02-03 23:02:47 +00:00
pennae
5b6dcece88
Merge pull request #212684 from pennae/nixos-render-docs
nixos-render-docs: init, use for some manual rendering to docbook
2023-01-30 19:26:07 +01:00
Vladimír Čunát
23ce77d76e Revert #178290: nixos/virtualisation: add option
...for explicitly named network interfaces

This reverts commit 6ae3e7695e.
(and evaluation fixups 08d26bbb72 7aed90a969)
Some of the tests fail or time out after the merge.
2023-01-30 07:55:50 -08:00
pennae
e8c5618b67 nixos-render-docs: add some better CLI infrastructure
using environment variables isn't great once multiple input or output
formats get involved (which will happen soon). now is a good time to set
a pattern for future converters.
2023-01-27 20:07:34 +01:00
pennae
be6a25368f nixos-render-docs: init from optionsToDocbook.py
this new package shall eventually contain the rendering code necessary
to produce the entirety of the nixos (not nixpkgs) manual, in all of its
various output formats.
2023-01-27 20:07:33 +01:00
Jörg Thalheim
e80e3878b3
Merge pull request #178533 from Mic92/nixos-tests-shell
nixos/tests: extend shell_interact to accept alternative socat addresses
2023-01-27 12:24:08 +00:00
Jörg Thalheim
29db54c373 nixos/tests: extend shell_interact to accept alternative socat addresses
`shell_interact()` is currently not nice to use.  If you try to cancel
the socat process, it will also break the nixos test. Furthermore
ptpython creates it's own terminal that subprocesses are running in,
which breaks some of the terminal features of socat.
Hence this commit extends `shell_interact` to allow also to connect to
arbitrary servers i.e. tcp servers started by socat.
2023-01-27 13:09:25 +01:00
pennae
9ff987764c nixos/make-options-doc: enable smartquotes and replacements
the rest of the nixos manual has them enabled, so we should enable them
here too for consistency.

this changes rendered output pervasively. changes also include quotes in
types (eg in `strings concatenated with "\n"`), but since those are not
code this is probably fine. if not we can probably add a myst role to
inhibit replacements.
2023-01-26 00:38:06 +01:00
pennae
de22a26b4c nixos/make-options-doc: render option types through md
no changes to rendered output, but options.xml loses a few spaces.
2023-01-26 00:38:06 +01:00
pennae
6b677d9148 nixos/make-options-doc: disable inline html
we target not only html, but also manpages. inline html is not helpful
for the latter.
2023-01-26 00:38:06 +01:00
pennae
ba1f533134 nixos/make-option-docs: wrap md parser+renderer into Converter
the new rendering tool will consist of a number of different Converters,
each with a possibly specialized Renderer.
2023-01-26 00:38:06 +01:00
pennae
3e45994a3b nixos/make-options-doc: don't use inspect for optionsToDocbook
the rules are fixed, and we want to support all of them (or throw a
useful error message). this will also become the base for a generic
renderer system, so let's just list all the rules statically.
2023-01-26 00:38:06 +01:00
Robert Hensing
d255493e8f nixos/optionsToDocbook.py: Add docbook ns for when the appendix tag is omitted 2023-01-26 00:32:56 +01:00
pennae
0175a91aa3 nixos/make-options-doc: split docbook conversion from mergeJSON
this restores mergeJSON to its former glory if…merging json, and
extracts the MD rendering into a new script that will run instead of the
py+nix+xslt pipeline we previously ran to convert options.json to docbook.

this change alone gives a noticable performance boost when building
docs (18s instead of 27s to build optionsDocBook).

no changes to rendered output, except for a single example in the
rsnapshot module that uses hard tabs for indentation instead of spaces.
this probably isn't important.

docbook warnings remain with mergeJSON since the other processing steps
output single files instead of directories. since we'll only keep the
check until 23.11 this is probably also not important to fix.

also contains a few improvements to error reporting in the MD renderers.
2023-01-26 00:32:56 +01:00
pennae
381dcd7f9d nixos/make-options-doc: remove program argument to xslt
this doesn't seem to be set anywhere, not from make-options-doc nor
anywhere else.
2023-01-26 00:32:56 +01:00
Andrew Hoff
6ae3e7695e nixos/virtualisation: add option for explicitly named network interfaces
Adds a new option to the virtualisation modules that enables specifying
explicitly named network interfaces in QEMU VMs. The existing
`virtualisation.vlans` is still supported for cases where the name of
the network interface is irrelevant.
2023-01-22 12:20:40 -05:00
pennae
e2019c49d4 nixos/make-options-doc: use markdown-it-py for rendering
only whitespace changes (mostly empty descriptions rendered as literal
line breaks and trailing space toPretty generates, but that were dropped
by mistune).
2023-01-22 17:50:09 +01:00
pennae
248e1b957d nixos/make-options-doc: render related packages with md
don't generate docbook for related packages, generate markdown instead.
this could be extended further to not even generate markdown but have
mergeJSON handle all of the rendering. markdown will work fine for now
though.
2023-01-22 17:50:09 +01:00