nixos-rebuild-ng: init (#354029)

      - any-glob-to-any-file:

        - nixos/**/*

        - pkgs/by-name/sw/switch-to-configuration-ng/**/*

        - pkgs/by-name/ni/nixos-rebuild-ng/**/*

        - pkgs/os-specific/linux/nixos-rebuild/**/*

"6.topic: nixos-container":

/nixos/tests/amazon-ssm-agent.nix                 @arianvp

/nixos/modules/system/boot/grow-partition.nix     @arianvp

# nixos-rebuild-ng

/pkgs/by-name/ni/nixos-rebuild-ng                 @thiagokokada

# Updaters

# nixos-rebuild-ng

Work-in-Progress rewrite of

[`nixos-rebuild`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.sh).

## Why the rewrite?

The current state of `nixos-rebuild` is dare: it is one of the most critical

piece of code we have in NixOS, but it has tons of issues:

- The code is written in Bash, and while this by itself is not necessary bad,

  it means that it is difficult to do refactorings due to the lack of tooling

  for the language

- The code itself is a hacky mess. Changing even one line of code can cause

  issues that affects dozens of people

- Lack of proper testing (we do have some integration tests, but no unit tests

  and coverage is probably pitiful)

- The code predates some of the improvements `nix` had over the years, e.g.: it

  builds Flakes inside a temporary directory and read the resulting symlink

  since the code seems to predate `--print-out-paths` flag

Given all of those above, improvements in the `nixos-rebuild` are difficult to

do. A full rewrite is probably the easier way to improve the situation since

this can be done in a separate package that will not break anyone. So this is

an attempt of the rewrite.

## Why Python?

- It is the language of choice for many critical things inside `nixpkgs`, like

  the `NixOSTest` and `systemd-boot-builder.py` activation scripts

- It is a language with great tooling, e.g.: `mypy` for type checking, `ruff`

  for linting, `pytest` for unit testing

- It is a scripting language that fits well with the scope of this project

- Python's standard library is great and it means we will need a low number of

  external dependencies for this project. For example, `nixos-rebuild`

  currently depends in `jq` for JSON parsing, while Python has `json` in

  standard library

## Do's and Don'ts

- Do: be as much of a drop-in replacement as possible

- Do: fix obvious bugs

- Do: improvements that are non-breaking

- Don't: change logic in breaking ways even if this would be an improvement

## How to use

```nix

{ pkgs, ... }:

{

  environment.systemPackages = [ pkgs.nixos-rebuild-ng ];

}

```

And use `nixos-rebuild-ng` instead of `nixos-rebuild`.

## Current caveats

- For now we will install it in `nixos-rebuild-ng` path by default, to avoid

  conflicting with the current `nixos-rebuild`. This means you can keep both in

  your system at the same time, but it also means that a few things like bash

  completion are broken right now (since it looks at `nixos-rebuild` binary)

- `_NIXOS_REBUILD_EXEC` is **not** implemented yet, so different from

  `nixos-rebuild`, this will use the current version of `nixos-rebuild-ng` in

  your `PATH` to build/set profile/switch, while `nixos-rebuild` builds the new

  version (the one that will be switched) and re-exec to it instead. This means

  that in case of bugs in `nixos-rebuild-ng`, the only way that you will get

  them fixed is **after** you switch to a new version

- `nix` bootstrap is also **not** implemented yet, so this means that you will

  eval with an old version of Nix instead of a newer one. This is unlikely to

  cause issues, because the build will happen in the daemon anyway (that is

  only changed after the switch), and unless you are using bleeding edge `nix`

  features you will probably have zero problems here. You can basically think

  that using `nixos-rebuild-ng` is similar to running `nixos-rebuild --fast`

  right now

- Ignore any performance advantages of the rewrite right now, because of the 2

  caveats above

- `--target-host` and `--build-host` are not implemented yet and this is

  probably the thing that will be most difficult to implement. Help here is

  welcome

- Bugs in the profile manipulation can cause corruption of your profile that

  may be difficult to fix, so right now I only recommend using

  `nixos-rebuild-ng` if you are testing in a VM or in a filesystem with

  snapshots like btrfs or ZFS. Those bugs are unlikely to be unfixable but the

  errors can be difficult to understand. If you want to go anyway,

  `nix-collect-garbage -d` and `nix store repair` are your friends

## TODO

- [ ] Remote host/builders (via SSH)

- [ ] Improve nix arguments handling (e.g.: `nixFlags` vs `copyFlags` in the

  old `nixos-rebuild`)

- [ ] `_NIXOS_REBUILD_EXEC`

- [ ] Port `nixos-rebuild.passthru.tests`

- [ ] Change module system to allow easier opt-in, like

  `system.switch.enableNg` for `switch-to-configuration-ng`

- [ ] Improve documentation

- [ ] `nixos-rebuild repl` (calling old `nixos-rebuild` for now)

- [ ] `nix` build/bootstrap

- [ ] Reduce build closure

{

  lib,

  installShellFiles,

  nix,

  nixos-rebuild,

  python3,

  withNgSuffix ? true,

}:

python3.pkgs.buildPythonApplication {

  pname = "nixos-rebuild-ng";

  version = "0.0.0";

  src = ./src;

  pyproject = true;

  build-system = with python3.pkgs; [

    setuptools

  ];

  dependencies = with python3.pkgs; [

    tabulate

    types-tabulate

  ];

  nativeBuildInputs = [

    installShellFiles

  ];

  propagatedBuildInputs = [

    # Make sure that we use the Nix package we depend on, not something

    # else from the PATH for nix-{env,instantiate,build}. This is

    # important, because NixOS defaults the architecture of the rebuilt

    # system to the architecture of the nix-* binaries used. So if on an

    # amd64 system the user has an i686 Nix package in her PATH, then we

    # would silently downgrade the whole system to be i686 NixOS on the

    # next reboot.

    # The binary will be included in the wrapper for Python.

    nix

  ];

  preBuild = ''

    substituteInPlace nixos_rebuild/__init__.py \

      --subst-var-by nixos_rebuild ${lib.getExe nixos-rebuild}

  '';

  postInstall =

    ''

      installManPage ${nixos-rebuild}/share/man/man8/nixos-rebuild.8

      installShellCompletion \

        --bash ${nixos-rebuild}/share/bash-completion/completions/_nixos-rebuild

    ''

    + lib.optionalString withNgSuffix ''

      mv $out/bin/nixos-rebuild $out/bin/nixos-rebuild-ng

    '';

  nativeCheckInputs = with python3.pkgs; [

    pytestCheckHook

    mypy

    ruff

  ];

  pytestFlagsArray = [ "-vv" ];

  postCheck = ''

    echo -e "\x1b[32m## run mypy\x1b[0m"

    mypy nixos_rebuild tests

    echo -e "\x1b[32m## run ruff\x1b[0m"

    ruff check nixos_rebuild tests

    echo -e "\x1b[32m## run ruff format\x1b[0m"

    ruff format --check nixos_rebuild tests

  '';

  meta = {

    description = "Rebuild your NixOS configuration and switch to it, on local hosts and remote";

    homepage = "https://github.com/NixOS/nixpkgs/tree/master/pkgs/by-name/ni/nixos-rebuild-ng";

    license = lib.licenses.mit;

    maintainers = [ lib.maintainers.thiagokokada ];

    mainProgram = if withNgSuffix then "nixos-rebuild-ng" else "nixos-rebuild";

  };

}

from __future__ import annotations

import argparse

import json

import os

import sys

from subprocess import run

from typing import assert_never

from tabulate import tabulate

from .models import Action, Flake, NRError, Profile

from .nix import (

    edit,

    list_generations,

    nixos_build,

    nixos_build_flake,

    rollback,

    rollback_temporary_profile,

    set_profile,

    switch_to_configuration,

    upgrade_channels,

)

from .utils import info

VERBOSE = False

def parse_args(argv: list[str]) -> tuple[argparse.Namespace, list[str]]:

    parser = argparse.ArgumentParser(

        prog="nixos-rebuild",

        description="Reconfigure a NixOS machine",

        add_help=False,

        allow_abbrev=False,

    )

    parser.add_argument("--help", action="store_true")

    parser.add_argument("--file", "-f")

    parser.add_argument("--attr", "-A")

    parser.add_argument("--flake", nargs="?", const=True)

    parser.add_argument("--no-flake", dest="flake", action="store_false")

    parser.add_argument("--install-bootloader", action="store_true")

    # TODO: add deprecated=True in Python >=3.13

    parser.add_argument("--install-grub", action="store_true")

    parser.add_argument("--profile-name", "-p", default="system")

    parser.add_argument("--specialisation", "-c")

    parser.add_argument("--rollback", action="store_true")

    parser.add_argument("--upgrade", action="store_true")

    parser.add_argument("--upgrade-all", action="store_true")

    parser.add_argument("--json", action="store_true")

    parser.add_argument("action", choices=Action.values(), nargs="?")

    args, remainder = parser.parse_known_args(argv[1:])

    global VERBOSE

    # Manually parse verbose flag since this is a nix flag that also affect

    # the script

    VERBOSE = any(v == "--verbose" or v.startswith("-v") for v in remainder)

    # https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.sh#L56

    if args.action == Action.DRY_RUN.value:

        args.action = Action.DRY_BUILD.value

    if args.install_grub:

        info(

            f"{parser.prog}: warning: --install-grub deprecated, use --install-bootloader instead"

        )

        args.install_bootloader = True

    if args.action == Action.EDIT.value and (args.file or args.attr):

        parser.error("--file and --attr are not supported with 'edit'")

    if args.flake and (args.file or args.attr):

        parser.error("--flake cannot be used with --file or --attr")

    if args.help or args.action is None:

        r = run(["man", "8", "nixos-rebuild"], check=False)

        parser.exit(r.returncode)

    return args, remainder

def execute(argv: list[str]) -> None:

    args, nix_flags = parse_args(argv)

    profile = Profile.from_name(args.profile_name)

    flake = Flake.from_arg(args.flake)

    if args.upgrade or args.upgrade_all:

        upgrade_channels(bool(args.upgrade_all))

    match action := Action(args.action):

        case Action.SWITCH | Action.BOOT:

            info("building the system configuration...")

            if args.rollback:

                path_to_config = rollback(profile)

            elif flake:

                path_to_config = nixos_build_flake(

                    "toplevel",

                    flake,

                    nix_flags,

                    no_link=True,

                )

                set_profile(profile, path_to_config)

            else:

                path_to_config = nixos_build(

                    "system",

                    args.attr,

                    args.file,

                    nix_flags,

                    no_out_link=True,

                )

                set_profile(profile, path_to_config)

            switch_to_configuration(

                path_to_config,

                action,

                specialisation=args.specialisation,

                install_bootloader=args.install_bootloader,

            )

        case Action.TEST | Action.BUILD | Action.DRY_BUILD | Action.DRY_ACTIVATE:

            info("building the system configuration...")

            dry_run = action == Action.DRY_BUILD

            if args.rollback and action in (Action.TEST, Action.BUILD):

                maybe_path_to_config = rollback_temporary_profile(profile)

                if maybe_path_to_config:  # kinda silly but this makes mypy happy

                    path_to_config = maybe_path_to_config

                else:

                    raise NRError("could not find previous generation")

            elif flake:

                path_to_config = nixos_build_flake(

                    "toplevel",

                    flake,

                    nix_flags,

                    keep_going=True,

                    dry_run=dry_run,

                )

            else:

                path_to_config = nixos_build(

                    "system",

                    args.attr,

                    args.file,

                    nix_flags,

                    keep_going=True,

                    dry_run=dry_run,

                )

            if action in (Action.TEST, Action.DRY_ACTIVATE):

                switch_to_configuration(

                    path_to_config,

                    action,

                    specialisation=args.specialisation,

                )

        case Action.BUILD_VM | Action.BUILD_VM_WITH_BOOTLOADER:

            info("building the system configuration...")

            attr = "vm" if action == Action.BUILD_VM else "vmWithBootLoader"

            if flake:

                path_to_config = nixos_build_flake(

                    attr,

                    flake,

                    nix_flags,

                    keep_going=True,

                )

            else:

                path_to_config = nixos_build(

                    attr,

                    args.attr,

                    args.file,

                    nix_flags,

                    keep_going=True,

                )

            vm_path = next(path_to_config.glob("bin/run-*-vm"), "./result/bin/run-*-vm")

            print(f"Done. The virtual machine can be started by running '{vm_path}'")

        case Action.EDIT:

            edit(flake, nix_flags)

        case Action.DRY_RUN:

            assert False, "DRY_RUN should be a DRY_BUILD alias"

        case Action.LIST_GENERATIONS:

            generations = list_generations(profile)

            if args.json:

                print(json.dumps(generations, indent=2))

            else:

                headers = {

                    "generation": "Generation",

                    "date": "Build-date",

                    "nixosVersion": "NixOS version",

                    "kernelVersion": "Kernel",

                    "configurationRevision": "Configuration Revision",

                    "specialisations": "Specialisation",

                    "current": "Current",

                }

                # Not exactly the same format as legacy nixos-rebuild but close

                # enough

                table = tabulate(

                    generations,

                    headers=headers,

                    tablefmt="plain",

                    numalign="left",

                    stralign="left",

                    disable_numparse=True,

                )

                print(table)

        case Action.REPL:

            # For now just redirect it to `nixos-rebuild` instead of

            # duplicating the code

            os.execv(

                "@nixos_rebuild@",

                argv,

            )

        case _:

            assert_never(action)

def main() -> None:

    try:

        execute(sys.argv)

    except (Exception, KeyboardInterrupt) as ex:

        if VERBOSE:

            raise ex

        else:

            sys.exit(str(ex))

if __name__ == "__main__":

    main()