Unverified Commit 9f95c2ac authored by Maximilian Bosch's avatar Maximilian Bosch
Browse files

linux: new workflow for kernel changes 

See https://github.com/NixOS/nixpkgs/issues/437208#issuecomment-3288623669
Depends on https://github.com/NixOS/org/pull/172

As documented below, the idea is to essentially group all changes
rebuilding all VM tests with kernel updates and merge them together into
`master` whenever the Linux kernels get updated.

This documents the workflow of updates in the nixpkgs manual. While at
it, I removed the README from the packages because

* it's horribly outdated
* I didn't even know it exists which confirms that its discoverability
  was very poor

and added the relevant portions into the nixpkgs manual as well.
parent eabbe2ae

.github/workflows/periodic-merge-6h.yml

+2 −0
Original line number Diff line number Diff line
@@ -35,6 +35,8 @@ jobs: 
            into: staging-next

          - from: staging-next

            into: staging

          - from: master

            into: staging-nixos

    uses: ./.github/workflows/periodic-merge.yml

    with:

      from: ${{ matrix.pairs.from }}

CONTRIBUTING.md

+30 −8
Original line number Diff line number Diff line
@@ -430,20 +430,22 @@ gitGraph 


Here's an overview of the different branches:



| branch | `master` | `staging-next` | `staging` |

| --- | --- | --- | --- |

| Used for development | ✔️ | ❌ | ✔️ |

| Built by Hydra | ✔️ | ✔️ | ❌ |

| [Mass rebuilds][mass-rebuild] | ❌ | ⚠️  Only to fix Hydra builds | ✔️ |

| Critical security fixes | ✔️ for non-mass-rebuilds | ✔️ for mass-rebuilds | ❌ |

| Automatically merged into | `staging-next` | `staging` | - |

| Manually merged into | - | `master` | `staging-next` |

| branch | `master` | `staging-next` | `staging` | [`staging-nixos`](#test-driver-rebuild) |

| --- | --- | --- | --- | --- |

| Used for development | ✔️ | ❌ | ✔️ | ✔️ |

| Built by Hydra | ✔️ | ✔️ | ❌ | ❌ |

| [Mass rebuilds][mass-rebuild] | ❌ | ⚠️  Only to fix Hydra builds | ✔️ | ❌[^1]  |

| Critical security fixes | ✔️ for non-mass-rebuilds | ✔️ for mass-rebuilds | ❌ | ✔️ |

| Automatically merged into | `staging-next` & `staging-nixos` | `staging` | - | - |

| Manually merged into | - | `master` | `staging-next` | `master` |



The staging workflow is used for all stable branches with corresponding names:

- `master`/`release-YY.MM`

- `staging`/`staging-YY.MM`

- `staging-next`/`staging-next-YY.MM`



[^1]: Except changes that cause no more rebuilds than kernel updates



# Conventions



## Branch conventions
@@ -495,6 +497,26 @@ In order to help the decision, CI automatically assigns [`rebuild` labels](https 
As a rule of thumb, if the number of rebuilds is **500 or more**, consider targeting the `staging` branch instead of `master`; if the number is **1000 or more**, the pull request causes a mass rebuild, and should target the `staging` branch.

See [previously merged pull requests to the staging branches](https://github.com/NixOS/nixpkgs/issues?q=base%3Astaging+-base%3Astaging-next+is%3Amerged) to get a sense for what changes are considered mass rebuilds.



Please note that changes to the Linux kernel are an exception to this rule.

These PRs go to `staging-nixos`, see [the next section for more context](#changes-rebuilding-all-tests).



### Changes rebuilding all NixOS tests

[test-driver-rebuild]: #changes-rebuilding-all-tests



Changes causing a rebuild of all NixOS tests get a special [`10.rebuild-nixos-tests`](https://github.com/NixOS/nixpkgs/issues?q=state%3Aopen%20label%3A10.rebuild-nixos-tests) label.

These changes pose a significant impact on the build infrastructure.



Hence, these PRs should either target a `staging`-branch or `staging-nixos`, provided one of following conditions applies:



* The label `10.rebuild-nixos-tests` is set, or

* The PR is a change affecting the Linux kernel.



The branch gets merged whenever mainline kernel updates or critical security fixes land on the branch.

This usually happens on a weekly basis.



Backports are not handled by such a branch.

The relevant PRs from this branch must be backported manually.



## Commit conventions

[commit-conventions]: #commit-conventions

ci/supportedBranches.js

+2 −0
Original line number Diff line number Diff line
@@ -8,6 +8,7 @@ const typeConfig = { 
  release: ['development', 'primary'],

  staging: ['development', 'secondary'],

  'staging-next': ['development', 'secondary'],

  'staging-nixos': ['development', 'secondary'],

  'haskell-updates': ['development', 'secondary'],

  nixos: ['channel'],

  nixpkgs: ['channel'],
@@ -19,6 +20,7 @@ const orderConfig = { 
  master: 0,

  release: 1,

  staging: 2,

  'staging-nixos': 2,

  'haskell-updates': 3,

  'staging-next': 4,

}

doc/packages/linux.section.md

+53 −0
Original line number Diff line number Diff line
@@ -105,3 +105,56 @@ $ make -C $dev/lib/modules/*/build M=$(pwd)/drivers/net/ethernet/mellanox module 
```



:::



## Maintainer information {#sec-linux-kernel-maintainer-information}



### Updating kernels {#sec-linux-updates}



Updating all kernels can be done with the following script:



```ShellSession

$ pkgs/os-specific/linux/kernel/update.sh

```



The change gets submitted like this:



* File a PR against `staging-nixos`.

  * Add a `backport release-XX.XX` label for an automated backport.

    We don't expect many other changes on that branch to require a backport, hence there's no such branch for stable.

    By using an additional PR, we get the automatic backport against stable without manual cherry-picks.

* Merge into `staging-nixos`.

* File as PR from `staging-nixos` against `master`.

* When all status checks are green, merge.



### Add a new (major) version of the Linux kernel {#sec-linux-add-new-kernel-version}



* When running `./pkgs/os-specific/linux/kernel/update.sh`, new kernel majors get discovered automatically.

* Prepare all Nix expressions for the new kernel

  * Instantiate the new kernel in `pkgs/top-level/linux-kernels.nix` in the `kernels`-section.

    ```nix

    {

      linux_X_Y = callPackage ../os-specific/linux/kernel/mainline.nix {

        branch = "X.Y";

        kernelPatches = [

          # any new patches required (it makes to look which patches are used by its predecessor)

        ];

      };

    }

    ```

  * Instantiate the package-set in `vanillaPackages`:

    ```nix

    {

      linux_X_Y = recurseIntoAttrs (packagesFor kernels.linux_X_Y);

    }

    ```

  * Update `linux_latest` to the new attribute.

* __SQUASH__ the changes into the `linux: init at …` commit.

* If a new hardened is available:

  * Instantiate a `linux_X_Y_hardened = hardenedKernelsFor kernels.linux_X_Y { };` in `kernels` and

    `linux_X_Y_hardened = hardenedKernelFor kernels.linux_X_Y { };` in the `packages`-section.

  * Make sure to remove the hardened variant of the previous kernel version unless it's LTS.

    We only support the latest and latest LTS version of hardened.

* If no new hardened kernel is available:

  * Keep the previously latest kernel until its mainline counterpart gets removed.

    After that `linux_hardened` points to the latest LTS supported by hardened.

* __SQUASH__ the changes into the `linux_X_Y_hardened: init at …` commit.

doc/redirects.json

+9 −0
Original line number Diff line number Diff line
@@ -4306,6 +4306,15 @@ 
  "sec-linux-kernel": [

    "index.html#sec-linux-kernel"

  ],

  "sec-linux-kernel-maintainer-information": [

    "index.html#sec-linux-kernel-maintainer-information"

  ],

  "sec-linux-updates": [

    "index.html#sec-linux-updates"

  ],

  "sec-linux-add-new-kernel-version": [

    "index.html#sec-linux-add-new-kernel-version"

  ],

  "ex-overriding-kernel-derivation": [

    "index.html#ex-overriding-kernel-derivation"

  ],