check-by-name: Update contributor docs

# When you make changes to this workflow, also update pkgs/test/nixpkgs-check-by-name/scripts/run-local.sh adequately

name: Check pkgs/by-name

# The pre-built tool is fetched from a channel,

# making it work predictable on all PRs.

# The tool is pinned to a pre-built version on Hydra,

# see pkgs/test/nixpkgs-check-by-name/scripts/README.md

on:

  # Using pull_request_target instead of pull_request avoids having to approve first time contributors

  pull_request_target

          echo "base=$base" >> "$GITHUB_ENV"

      - uses: cachix/install-nix-action@7ac1ec25491415c381d9b62f0657c7a028df52a7 # v24

      - name: Fetching the pinned tool

        # Update the pinned version using pkgs/test/nixpkgs-check-by-name/scripts/update-pinned-tool.sh

        run: |

          # Get the direct /nix/store path from the pin to avoid having to evaluate Nixpkgs

          toolPath=$(jq -r '."ci-path"' pkgs/test/nixpkgs-check-by-name/scripts/pinned-tool.json)

        (onSystems ["x86_64-linux"] "nixpkgs.mesa_i686") # i686 sanity check + useful

        ["nixpkgs.tarball"]

        # Ensure that nixpkgs-check-by-name is available in all release channels and nixos-unstable,

        # so that a pre-built version can be used in CI for PR's on the corresponding development branches.

        # Ensure that nixpkgs-check-by-name is available in nixos-unstable,

        # so that a pre-built version can be used in CI for PR's

        # See ../pkgs/test/nixpkgs-check-by-name/README.md

        (onSystems ["x86_64-linux"] "nixpkgs.tests.nixpkgs-check-by-name")

      ];

# Nixpkgs pkgs/by-name checker

This directory implements a program to check the [validity](#validity-checks) of the `pkgs/by-name` Nixpkgs directory.

It is being used by [this GitHub Actions workflow](../../../.github/workflows/check-by-name.yml).

This is part of the implementation of [RFC 140](https://github.com/NixOS/rfcs/pull/140).

A [pinned version](./scripts/pinned-tool.json) of this tool is used by [this GitHub Actions workflow](../../../.github/workflows/check-by-name.yml).

See [./scripts](./scripts/README.md#update-pinned-toolsh) for how to update the pinned version.

The source of the tool being right inside Nixpkgs allows any Nixpkgs committer to make updates to it.

## Interface

The interface of the tool is shown with `--help`:

- `expected` (optional):

  A file containing the expected standard output.

  The default is expecting an empty standard output.

## Hydra builds

This program will always be available pre-built for `x86_64-linux` on the `nixos-unstable` channel and `nixos-XX.YY` channels.

This is ensured by including it in the `tested` jobset description in [`nixos/release-combined.nix`](../../../nixos/release-combined.nix).

This allows CI for PRs to development branches `master` and `release-XX.YY` to fetch the pre-built program from the corresponding channel and use that to check the PR. This has the following benefits:

- It allows CI to check all PRs, even if they would break the CI tooling.

- It makes the CI check very fast, since no Nix builds need to be done, even for mass rebuilds.

- It improves security, since we don't have to build potentially untrusted code from PRs.

  The tool only needs a very minimal Nix evaluation at runtime, which can work with [readonly-mode](https://nixos.org/manual/nix/stable/command-ref/opt-common.html#opt-readonly-mode) and [restrict-eval](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-restrict-eval).

- It allows anybody to make updates to the tooling and for those updates to be automatically used by CI without needing a separate release mechanism.

The tradeoff is that there's a delay between updates to the tool and those updates being used by CI.

This needs to be considered when updating the [API](#api).

## `./update-pinned-tool.sh`

Updates the pinned CI tool in [`./pinned-tool.json`](./pinned-tool.json) to the

[latest version from the `nixos-unstable` channel](https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.tests.nixpkgs-check-by-name.x86_64-linux)

[latest version from the `nixos-unstable` channel](https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.tests.nixpkgs-check-by-name.x86_64-linux).

This script needs to be called manually when the CI tooling needs to be updated.

The `pinned-tool.json` file gets populated with both:

- The `/nix/store` path for `x86_64-linux`, such that CI doesn't have to evaluate Nixpkgs and can directly fetch it from the cache instead.

- The Nixpkgs revision, such that the `./run-local.sh` script can be used to run the checks locally on any system.

To ensure that the tool is always pre-built for `x86_64-linux` in the `nixos-unstable` channel,

it's included in the `tested` jobset description in [`nixos/release-combined.nix`](../../../nixos/release-combined.nix).

Why not just build the tooling right from the PRs Nixpkgs version?

- Because it allows CI to check all PRs, even if they would break the CI tooling.

- Because it makes the CI check very fast, since no Nix builds need to be done, even for mass rebuilds.

- Because it improves security, since we don't have to build potentially untrusted code from PRs.

  The tool only needs a very minimal Nix evaluation at runtime, which can work with [readonly-mode](https://nixos.org/manual/nix/stable/command-ref/opt-common.html#opt-readonly-mode) and [restrict-eval](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-restrict-eval).

This script is called manually once the CI tooling needs to be updated.