Unverified Commit 00b1aef9 authored by Sergei Zimmerman's avatar Sergei Zimmerman
Browse files

ctestCheckHook: init 

Motivation for this hook is simple: there's no single documented
way to do trivial things with ctest:

1. Pass additional flags to ctest invocation.
2. Selectively disable tests in a mechanism similar to python's
   `disabledTests` or rust's composable skips in `checkFlags`.
3. Disable parallel checking.

Current state of things has lead to several different solutions:

1. Completely overriding `checkPhase` [1] and invoking ctest manually
   with the necessary flags. This is most often coupled with `-E` for
   disabling test or setting parallel level.
2. Wrangling with weird double string/regex escaping and trying to stuff
   additional parameters and/or exclusion regex via `CMAKE_CTEST_ARGUMENTS`.
   This approach is especially painful when test names have spaces. This is
   the reason I originally decided to implement this hook after wrangling with
   failing darwin tests here [2].
3. Stuffing additional arguments into `checkFlagsArray` with the
   `ARGS` makefile parameter [3].

I don't see any reason to keep the status-quo. Doing something along these
lines has been suggested [4] for both `ctest` and `meson`. Meson setup-hook
has switched from `ninja` to `meson` in [5] with little friction. Doing
the same for cmake in a single sweep would prove problematic due to the
aforementioned zoo of workarounds and hacks for `ctest`. Doing it via
a separate hook would allow us to refactor things piecemeal and without
going through staging. The benefit of the hook is immediately clear and it
would allow to drive the refactor tractor at a comfortable pace.

[1]: pd/pdal/package.nix:117, cc/ccache/package.nix:108, gl/glog/package.nix:79
[2]: https://www.github.com/NixOS/nixpkgs/pull/375955
[3]: op/open62541/package.nix:114
[4]: https://www.github.com/NixOS/nixpkgs/issues/113829
[5]: https://www.github.com/NixOS/nixpkgs/pull/213845
parent 5ca2c4b2

doc/hooks/cmake.section.md

+18 −0
Original line number Diff line number Diff line
@@ -33,3 +33,21 @@ The default value is `build`. 
#### `dontUseCmakeConfigure` {#dont-use-cmake-configure}



When set to true, don't use the predefined `cmakeConfigurePhase`.



## Controlling CTest invocation {#cmake-ctest}



By default tests are run by make in [`checkPhase`](#ssec-check-phase) or by [ninja](#ninja) if `ninja` is

available in `nativeBuildInputs`. Makefile and Ninja generators produce the `test` target, which invokes `ctest` under the hood.

This makes passing additional arguments to `ctest` difficult, so it's possible to invoke it directly in `checkPhase`

by adding `ctestCheckHook` to `nativeCheckInputs`.



### CTest Variables {#cmake-ctest-variables}



#### `disabledTests` {#cmake-ctest-disabled-tests}



Allows to disable running a list of tests. Note that regular expressions are not supported by `disabledTests`, but

it can be combined with `--exclude-regex` option.



#### `ctestFlags` {#cmake-ctest-flags}



Additional options passed to `ctest` together with `checkFlags`.

doc/redirects.json

+12 −0
Original line number Diff line number Diff line
@@ -5,6 +5,18 @@ 
  "chap-release-notes": [

    "release-notes.html#chap-release-notes"

  ],

  "cmake-ctest": [

    "index.html#cmake-ctest"

  ],

  "cmake-ctest-disabled-tests": [

    "index.html#cmake-ctest-disabled-tests"

  ],

  "cmake-ctest-flags": [

    "index.html#cmake-ctest-flags"

  ],

  "cmake-ctest-variables": [

    "index.html#cmake-ctest-variables"

  ],

  "ex-build-helpers-extendMkDerivation": [

    "index.html#ex-build-helpers-extendMkDerivation"

  ],

pkgs/by-name/ct/ctestCheckHook/ctest-check-hook.sh

0 → 100644
+44 −0
Original line number Diff line number Diff line 
# shellcheck shell=bash disable=SC2154



ctestCheckHook() {

    echo "Executing ctestCheckHook"



    runHook preCheck



    local buildCores=1



    if [ "${enableParallelChecking-1}" ]; then

        buildCores="$NIX_BUILD_CORES"

    fi



    local flagsArray=(

        "-j$buildCores"

        # This is enabled by the cmakeConfigurePhase by exporting

        # CTEST_OUTPUT_ON_FAILURE, but it makes sense it enable it globally here

        # as well.

        "--output-on-failure"

    )



    local disabledTestsArray=()

    concatTo disabledTestsArray disabledTests



    if [ ${#disabledTestsArray[@]} -ne 0 ]; then

        local ctestExcludedTestsFile=$NIX_BUILD_TOP/.ctest-excluded-tests

        disabledTestsString="$(concatStringsSep "\n" disabledTestsArray)"

        echo -e "$disabledTestsString" >"$ctestExcludedTestsFile"

        flagsArray+=("--exclude-from-file" "$ctestExcludedTestsFile")

    fi



    concatTo flagsArray ctestFlags checkFlags checkFlagsArray



    echoCmd 'ctest flags' "${flagsArray[@]}"

    ctest "${flagsArray[@]}"



    echo "Finished ctestCheckHook"



    runHook postCheck

}



if [ -z "${dontUseCTestCheck-}" ] && [ -z "${checkPhase-}" ]; then

    checkPhase=ctestCheckHook

fi

pkgs/by-name/ct/ctestCheckHook/package.nix

0 → 100644
+9 −0
Original line number Diff line number Diff line 
{

  makeSetupHook,

  cmake,

}:



makeSetupHook {

  name = "ctestCheckHook";

  propagatedBuildInputs = [ cmake ];

} ./ctest-check-hook.sh