Merge pull request #256417 from tweag/fileset.trace

- > The file set library is currently somewhat limited but is being expanded to include more functions over time.

  in [the manual](../../doc/functions/fileset.section.md)

- Once a tracing function exists, `__noEval` in [internal.nix](./internal.nix) should mention it

- If/Once a function to convert `lib.sources` values into file sets exists, the `_coerce` and `toSource` functions should be updated to mention that function in the error when such a value is passed

- If/Once a function exists that can optionally include a path depending on whether it exists, the error message for the path not existing in `_coerce` should mention the new function

    _coerceMany

    _toSourceFilter

    _unionMany

    _printFileset

    ;

  inherit (builtins)

    isList

    isPath

    pathExists

    seq

    typeOf

    ;

        _unionMany

      ];

  /*

    Incrementally evaluate and trace a file set in a pretty way.

    This function is only intended for debugging purposes.

    The exact tracing format is unspecified and may change.

    This function takes a final argument to return.

    In comparison, [`traceVal`](#function-library-lib.fileset.traceVal) returns

    the given file set argument.

    This variant is useful for tracing file sets in the Nix repl.

    Type:

      trace :: FileSet -> Any -> Any

    Example:

      trace (unions [ ./Makefile ./src ./tests/run.sh ]) null

      =>

      trace: /home/user/src/myProject

      trace: - Makefile (regular)

      trace: - src (all files in directory)

      trace: - tests

      trace:   - run.sh (regular)

      null

  */

  trace =

    /*

    The file set to trace.

    This argument can also be a path,

    which gets [implicitly coerced to a file set](#sec-fileset-path-coercion).

    */

    fileset:

    let

      # "fileset" would be a better name, but that would clash with the argument name,

      # and we cannot change that because of https://github.com/nix-community/nixdoc/issues/76

      actualFileset = _coerce "lib.fileset.trace: argument" fileset;

    in

    seq

      (_printFileset actualFileset)

      (x: x);

  /*

    Incrementally evaluate and trace a file set in a pretty way.

    This function is only intended for debugging purposes.

    The exact tracing format is unspecified and may change.

    This function returns the given file set.

    In comparison, [`trace`](#function-library-lib.fileset.trace) takes another argument to return.

    This variant is useful for tracing file sets passed as arguments to other functions.

    Type:

      traceVal :: FileSet -> FileSet

    Example:

      toSource {

        root = ./.;

        fileset = traceVal (unions [

          ./Makefile

          ./src

          ./tests/run.sh

        ]);

      }

      =>

      trace: /home/user/src/myProject

      trace: - Makefile (regular)

      trace: - src (all files in directory)

      trace: - tests

      trace:   - run.sh (regular)

      "/nix/store/...-source"

  */

  traceVal =

    /*

    The file set to trace and return.

    This argument can also be a path,

    which gets [implicitly coerced to a file set](#sec-fileset-path-coercion).

    */

    fileset:

    let

      # "fileset" would be a better name, but that would clash with the argument name,

      # and we cannot change that because of https://github.com/nix-community/nixdoc/issues/76

      actualFileset = _coerce "lib.fileset.traceVal: argument" fileset;

    in

    seq

      (_printFileset actualFileset)

      # We could also return the original fileset argument here,

      # but that would then duplicate work for consumers of the fileset, because then they have to coerce it again

      actualFileset;

}

    isString

    pathExists

    readDir

    typeOf

    seq

    split

    trace

    typeOf

    ;

  inherit (lib.attrsets)

    attrNames

    attrValues

    mapAttrs

    setAttrByPath

  ];

  _noEvalMessage = ''

    lib.fileset: Directly evaluating a file set is not supported. Use `lib.fileset.toSource` to turn it into a usable source instead.'';

    lib.fileset: Directly evaluating a file set is not supported.

      To turn it into a usable source, use `lib.fileset.toSource`.

      To pretty-print the contents, use `lib.fileset.trace` or `lib.fileset.traceVal`.'';

  # The empty file set without a base path

  _emptyWithoutBase = {

    # The one and only!

    _internalIsEmptyWithoutBase = true;

    # Double __ to make it be evaluated and ordered first

    __noEval = throw _noEvalMessage;

    # Due to alphabetical ordering, this is evaluated last,

    # which makes the nix repl output nicer than if it would be ordered first.

    # It also allows evaluating it strictly up to this error, which could be useful

    _noEval = throw _noEvalMessage;

  };

  # Create a fileset, see ./README.md#fileset

      _internalBaseComponents = components parts.subpath;

      _internalTree = tree;

      # Double __ to make it be evaluated and ordered first

      __noEval = throw _noEvalMessage;

      # Due to alphabetical ordering, this is evaluated last,

      # which makes the nix repl output nicer than if it would be ordered first.

      # It also allows evaluating it strictly up to this error, which could be useful

      _noEval = throw _noEvalMessage;

    };

  # Coerce a value to a fileset, erroring when the value cannot be coerced.

      // value;

  /*

    Simplify a filesetTree recursively:

    - Replace all directories that have no files with `null`

    A normalisation of a filesetTree suitable filtering with `builtins.path`:

    - Replace all directories that have no files with `null`.

      This removes directories that would be empty

    - Replace all directories with all files with `"directory"`

    - Replace all directories with all files with `"directory"`.

      This speeds up the source filter function

    Note that this function is strict, it evaluates the entire tree

    Type: Path -> filesetTree -> filesetTree

  */

  _simplifyTree = path: tree:

  _normaliseTreeFilter = path: tree:

    if tree == "directory" || isAttrs tree then

      let

        entries = _directoryEntries path tree;

        simpleSubtrees = mapAttrs (name: _simplifyTree (path + "/${name}")) entries;

        subtreeValues = attrValues simpleSubtrees;

        normalisedSubtrees = mapAttrs (name: _normaliseTreeFilter (path + "/${name}")) entries;

        subtreeValues = attrValues normalisedSubtrees;

      in

      # This triggers either when all files in a directory are filtered out

      # Or when the directory doesn't contain any files at all

      else if all isString subtreeValues then

        "directory"

      else

        simpleSubtrees

        normalisedSubtrees

    else

      tree;

  /*

    A minimal normalisation of a filesetTree, intended for pretty-printing:

    - If all children of a path are recursively included or empty directories, the path itself is also recursively included

    - If all children of a path are fully excluded or empty directories, the path itself is an empty directory

    - Other empty directories are represented with the special "emptyDir" string

      While these could be replaced with `null`, that would take another mapAttrs

    Note that this function is partially lazy.

    Type: Path -> filesetTree -> filesetTree (with "emptyDir"'s)

  */

  _normaliseTreeMinimal = path: tree:

    if tree == "directory" || isAttrs tree then

      let

        entries = _directoryEntries path tree;

        normalisedSubtrees = mapAttrs (name: _normaliseTreeMinimal (path + "/${name}")) entries;

        subtreeValues = attrValues normalisedSubtrees;

      in

      # If there are no entries, or all entries are empty directories, return "emptyDir".

      # After this branch we know that there's at least one file

      if all (value: value == "emptyDir") subtreeValues then

        "emptyDir"

      # If all subtrees are fully included or empty directories

      # (both of which are coincidentally represented as strings), return "directory".

      # This takes advantage of the fact that empty directories can be represented as included directories.

      # Note that the tree == "directory" check allows avoiding recursion

      else if tree == "directory" || all (value: isString value) subtreeValues then

        "directory"

      # If all subtrees are fully excluded or empty directories, return null.

      # This takes advantage of the fact that empty directories can be represented as excluded directories

      else if all (value: isNull value || value == "emptyDir") subtreeValues then

        null

      # Mix of included and excluded entries

      else

        normalisedSubtrees

    else

      tree;

  # Trace a filesetTree in a pretty way when the resulting value is evaluated.

  # This can handle both normal filesetTree's, and ones returned from _normaliseTreeMinimal

  # Type: Path -> filesetTree (with "emptyDir"'s) -> Null

  _printMinimalTree = base: tree:

    let

      treeSuffix = tree:

        if isAttrs tree then

          ""

        else if tree == "directory" then

          " (all files in directory)"

        else

          # This does "leak" the file type strings of the internal representation,

          # but this is the main reason these file type strings even are in the representation!

          # TODO: Consider removing that information from the internal representation for performance.

          # The file types can still be printed by querying them only during tracing

          " (${tree})";

      # Only for attribute set trees

      traceTreeAttrs = prevLine: indent: tree:

        foldl' (prevLine: name:

          let

            subtree = tree.${name};

            # Evaluating this prints the line for this subtree

            thisLine =

              trace "${indent}- ${name}${treeSuffix subtree}" prevLine;

          in

          if subtree == null || subtree == "emptyDir" then

            # Don't print anything at all if this subtree is empty

            prevLine

          else if isAttrs subtree then

            # A directory with explicit entries

            # Do print this node, but also recurse

            traceTreeAttrs thisLine "${indent}  " subtree

          else

            # Either a file, or a recursively included directory

            # Do print this node but no further recursion needed

            thisLine

        ) prevLine (attrNames tree);

      # Evaluating this will print the first line

      firstLine =

        if tree == null || tree == "emptyDir" then

          trace "(empty)" null

        else

          trace "${toString base}${treeSuffix tree}" null;

    in

    if isAttrs tree then

      traceTreeAttrs firstLine "" tree

    else

      firstLine;

  # Pretty-print a file set in a pretty way when the resulting value is evaluated

  # Type: fileset -> Null

  _printFileset = fileset:

    if fileset._internalIsEmptyWithoutBase then

      trace "(empty)" null

    else

      _printMinimalTree fileset._internalBase

        (_normaliseTreeMinimal fileset._internalBase fileset._internalTree);

  # Turn a fileset into a source filter function suitable for `builtins.path`

  # Only directories recursively containing at least one files are recursed into

  # Type: Path -> fileset -> (String -> String -> Bool)

    let

      # Simplify the tree, necessary to make sure all empty directories are null

      # which has the effect that they aren't included in the result

      tree = _simplifyTree fileset._internalBase fileset._internalTree;

      tree = _normaliseTreeFilter fileset._internalBase fileset._internalTree;

      # The base path as a string with a single trailing slash

      baseString =