Commit cc422e32 authored by Silvan Mosberger's avatar Silvan Mosberger
Browse files

workflows/check-by-name: Pin nixpkgs-check-by-name tool 

Before this, the tool for CI would update when nixos-unstable updated,
which is kind of terrible because you don't know when it happens, and it
might break master.

In fact, the tooling _right now_ has a serious bug and shouldn't be used!

This PR addresses this by _pinning_ the tooling in Nixpkgs itself.

Updating the tooling now requires two PRs:
- The first PR to update the tooling source
- (wait for Hydra to build and publish it in nixos-unstable)
- The second PR to update the pinned tooling

In turn you know exactly when the changes are going to take effect.

This change however has additional benefits:
- It makes CI more reproducible, because it doesn't depend on the state
  of nixos-unstable anymore
- Updates to the tooling can be tested with the workflow itself,
  because PRs that update the pinned tool will be tested on the updated
  version
- CI gets a sizable speed boost, because there's no need to download and
  evaluate a channel anymore
- It makes it more realistic to move the source of the tool into a
  separate repository
- It removes the brittle branch-specific logic that was previously
  needed to ensure that release branches use their own version of the
  tooling.
parent 9122ead9

.github/workflows/check-by-name.yml

+1 −1
Original line number Diff line number Diff line
@@ -92,7 +92,7 @@ jobs: 
          echo "base=$base" >> "$GITHUB_ENV"

      - uses: cachix/install-nix-action@7ac1ec25491415c381d9b62f0657c7a028df52a7 # v24

      - name: Fetching the tool

        run: pkgs/test/nixpkgs-check-by-name/scripts/fetch-tool.sh "$GITHUB_BASE_REF" result

        run: pkgs/test/nixpkgs-check-by-name/scripts/fetch-pinned-tool.sh result

      - name: Running nixpkgs-check-by-name

        run: |

          if result/bin/nixpkgs-check-by-name --base "$base" .; then

pkgs/test/nixpkgs-check-by-name/scripts/README.md

+11 −4
Original line number Diff line number Diff line 
# CI-related Scripts



This directory contains scripts used and related to the CI running the `pkgs/by-name` checks in Nixpkgs. See also the [CI GitHub Action](../../../../.github/workflows/check-by-name.yml).

This directory contains scripts and files used and related to the CI running the `pkgs/by-name` checks in Nixpkgs.

See also the [CI GitHub Action](../../../../.github/workflows/check-by-name.yml).



## `./run-local.sh BASE_BRANCH [REPOSITORY]`
@@ -15,12 +16,18 @@ Arguments: 
- `BASE_BRANCH`: The base branch to use, e.g. master or release-23.11

- `REPOSITORY`: The repository to fetch the base branch from, defaults to https://github.com/NixOS/nixpkgs.git



## `./fetch-tool.sh BASE_BRANCH OUTPUT_PATH`

## `./update-pinned-tool.sh`



Fetches the Hydra-prebuilt nixpkgs-check-by-name to use from the NixOS channel corresponding to the given base branch.

Updates the pinned CI tool in [`./pinned-tool.json`](./pinned-tool.json) to the

[latest version from the `nixos-unstable` channel](https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.tests.nixpkgs-check-by-name.x86_64-linux)



This script is called manually once the CI tooling needs to be updated.



## `./fetch-pinned-tool.sh OUTPUT_PATH`



Fetches the pinned tooling specified in [`./pinned-tool.json`](./pinned-tool.json).



This script is used both by [`./run-local.sh`](#run-local-sh-base-branch-repository) and CI.



Arguments:

- `BASE_BRANCH`: The base branch to use, e.g. master or release-23.11

- `OUTPUT_PATH`: The output symlink path for the tool

pkgs/test/nixpkgs-check-by-name/scripts/fetch-pinned-tool.sh

0 → 100755
+30 −0
Original line number Diff line number Diff line 
#!/usr/bin/env bash

# Try to not use nix-shell here to avoid fetching Nixpkgs,

# especially since this is used in CI

# The only dependency is `jq`, which in CI is implicitly available

# And when run from ./run-local.sh is provided by that parent script



set -o pipefail -o errexit -o nounset



trace() { echo >&2 "$@"; }



SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )



pin_file=$SCRIPT_DIR/pinned-tool.json



if (( $# < 1 )); then

    trace "Usage: $0 fetch OUTPUT_PATH"

    trace "OUTPUT_PATH: The output symlink path for the tool"

    exit 1

fi

output=$1



trace "Reading $pin_file.. "

rev=$(jq -r .rev "$SCRIPT_DIR"/pinned-tool.json)

trace -e "Git revision is \e[34m$rev\e[0m"

path=$(jq -r .path "$SCRIPT_DIR"/pinned-tool.json)

trace "Tooling path is $path"



trace -n "Fetching the prebuilt version of nixpkgs-check-by-name.. "

nix-store --add-root "$output" -r "$path" >/dev/null

realpath "$output"

pkgs/test/nixpkgs-check-by-name/scripts/fetch-tool.sh

+7 −35
Original line number Diff line number Diff line 
#!/usr/bin/env bash

# Fetches the prebuilt nixpkgs-check-by-name to use from

# the NixOS channel corresponding to the given base branch



set -o pipefail -o errexit -o nounset

# Legacy script to make CI work for the PR that replaces this

# Needed due to `.github/workflows/check-by-name.yml` using `pull_request_target`,

# which uses the workflow from the base branch, which still uses this script.

# This file can be removed after the PR replacing it is merged.



trace() { echo >&2 "$@"; }



if (( $# < 2 )); then

    trace "Usage: $0 BASE_BRANCH OUTPUT_PATH"

    trace "BASE_BRANCH: The base branch to use, e.g. master or release-23.11"

    trace "BASE_BRANCH (unused): The base branch to use, e.g. master or release-23.11"

    trace "OUTPUT_PATH: The output symlink path for the tool"

    exit 1

fi

baseBranch=$1

output=$2



trace -n "Determining the channel to use for PR base branch $baseBranch.. "

if [[ "$baseBranch" =~ ^(release|staging|staging-next)-([0-9][0-9]\.[0-9][0-9])$ ]]; then

  # Use the release channel for all PRs to release-XX.YY, staging-XX.YY and staging-next-XX.YY

  preferredChannel=nixos-${BASH_REMATCH[2]}

else

  # Use the nixos-unstable channel for all other PRs

  preferredChannel=nixos-unstable

fi



# Check that the channel exists. It doesn't exist for fresh release branches

if curl -fSs "https://channels.nixos.org/$preferredChannel"; then

    channel=$preferredChannel

    trace "$channel"

else

    # Fall back to nixos-unstable, makes sense for fresh release branches

    channel=nixos-unstable

    trace -e "\e[33mWarning: Preferred channel $preferredChannel could not be fetched, using fallback: $channel\e[0m"

fi



trace -n "Fetching latest version of channel $channel.. "

# This is probably the easiest way to get Nix to output the path to a downloaded channel!

nixpkgs=$(nix-instantiate --find-file nixpkgs -I nixpkgs=channel:"$channel")

trace "$nixpkgs"



# This file only exists in channels

trace -e "Git revision of channel $channel is \e[34m$(<"$nixpkgs/.git-revision")\e[0m"

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )



trace -n "Fetching the prebuilt version of nixpkgs-check-by-name.. "

nix-build -o "$output" "$nixpkgs" -A tests.nixpkgs-check-by-name -j 0 >/dev/null

realpath "$output" >&2
 
"$SCRIPT_DIR"/fetch-pinned-tool.sh "$output"

pkgs/test/nixpkgs-check-by-name/scripts/pinned-tool.json

0 → 100644
+4 −0
Original line number Diff line number Diff line 
{

  "rev": "9b19f5e77dd906cb52dade0b7bd280339d2a1f3d",

  "path": "/nix/store/qlls5ca8q88qpyygg9ddi60gl1nmvpij-nixpkgs-check-by-name"

}