Merge pull request #197547 from hercules-ci/module-class-and-flake-import

  # NB: This file describes the Nixpkgs manual, which happens to use module

  #     docs infra originally developed for NixOS.

  optionsDoc = pkgs.nixosOptionsDoc {

    inherit (pkgs.lib.evalModules { modules = [ ../../pkgs/top-level/config.nix ]; }) options;

    inherit (pkgs.lib.evalModules {

      modules = [ ../../pkgs/top-level/config.nix ];

      class = "nixpkgsConfig";

    }) options;

    documentType = "none";

    transformOptions = opt:

      opt // {

  <xi:include href="using/configuration.chapter.xml" />

  <xi:include href="using/overlays.chapter.xml" />

  <xi:include href="using/overrides.chapter.xml" />

 </part>

 <part>

  <title>Nixpkgs <code>lib</code></title>

  <xi:include href="functions.xml" />

  <xi:include href="module-system/module-system.chapter.xml" />

 </part>

 <part xml:id="part-stdenv">

  <title>Standard environment</title>

# Module System {#module-system}

## Introduction {#module-system-introduction}

The module system is a language for handling configuration, implemented as a Nix library.

Compared to plain Nix, it adds documentation, type checking and composition or extensibility.

::: {.note}

This chapter is new and not complete yet. For a gentle introduction to the module system, in the context of NixOS, see [Writing NixOS Modules](https://nixos.org/manual/nixos/unstable/index.html#sec-writing-modules) in the NixOS manual.

:::

## `lib.evalModules` {#module-system-lib-evalModules}

Evaluate a set of modules. This function is typically only used once per application (e.g. once in NixOS, once in Home Manager, ...).

### Parameters {#module-system-lib-evalModules-parameters}

#### `modules` {#module-system-lib-evalModules-param-modules}

A list of modules. These are merged together to form the final configuration.

<!-- TODO link to section about merging, TBD -->

#### `specialArgs` {#module-system-lib-evalModules-param-specialArgs}

An attribute set of module arguments that can be used in `imports`.

This is in contrast to `config._module.args`, which is only available after all `imports` have been resolved.

#### `class` {#module-system-lib-evalModules-param-class}

If the `class` attribute is set and non-`null`, the module system will reject `imports` with a different `_class` declaration.

The `class` value should be a string in lower [camel case](https://en.wikipedia.org/wiki/Camel_case).

If applicable, the `class` should match the "prefix" of the attributes used in (experimental) [flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake.html#description). Some examples are:

 - `nixos` as in `flake.nixosModules`

 - `nixosTest`: modules that constitute a [NixOS VM test](https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests)

<!-- We've only just started with `class`. You're invited to add a few more. -->

#### `prefix` {#module-system-lib-evalModules-param-prefix}

A list of strings representing the location at or below which all options are evaluated. This is used by `types.submodule` to improve error reporting and find the implicit `name` module argument.

### Return value {#module-system-lib-evalModules-return-value}

The result is an attribute set with the following attributes:

#### `options` {#module-system-lib-evalModules-return-value-options}

The nested attribute set of all option declarations.

#### `config` {#module-system-lib-evalModules-return-value-config}

The nested attribute set of all option values.

#### `type` {#module-system-lib-evalModules-return-value-type}

A module system type. This type is an instance of `types.submoduleWith` containing the current [`modules`](#module-system-lib-evalModules-param-modules).

The option definitions that are typed with this type will extend the current set of modules, like [`extendModules`](#module-system-lib-evalModules-return-value-extendModules).

However, the value returned from the type is just the [`config`](#module-system-lib-evalModules-return-value-config), like any submodule.

If you're familiar with prototype inheritance, you can think of this `evalModules` invocation as the prototype, and usages of this type as the instances.

This type is also available to the [`modules`](#module-system-lib-evalModules-param-modules) as the module argument `moduleType`.

<!-- TODO: document the module arguments. Using moduleType is like saying: suppose this configuration was extended. -->

#### `extendModules` {#module-system-lib-evalModules-return-value-extendModules}

A function similar to `evalModules` but building on top of the already passed [`modules`](#module-system-lib-evalModules-param-modules). Its arguments, `modules` and `specialArgs` are added to the existing values.

If you're familiar with prototype inheritance, you can think of the current, actual `evalModules` invocation as the prototype, and the return value of `extendModules` as the instance.

This functionality is also available to modules as the `extendModules` module argument.

::: {.note}

**Evaluation Performance**

`extendModules` returns a configuration that shares very little with the original `evalModules` invocation, because the module arguments may be different.

So if you have a configuration that has been (or will be) largely evaluated, almost none of the computation is shared with the configuration returned by `extendModules`.

The real work of module evaluation happens while computing the values in `config` and `options`, so multiple invocations of `extendModules` have a particularly small cost, as long as only the final `config` and `options` are evaluated.

If you do reference multiple `config` (or `options`) from before and after `extendModules`, evaluation performance is the same as with multiple `evalModules` invocations, because the new modules' ability to override existing configuration fundamentally requires constructing a new `config` and `options` fixpoint.

:::

#### `_module` {#module-system-lib-evalModules-return-value-_module}

A portion of the configuration tree which is elided from `config`.

<!-- TODO: when markdown migration is complete, make _module docs visible again and reference _module docs. Maybe move those docs into this chapter? -->

#### `_type` {#module-system-lib-evalModules-return-value-_type}

A nominal type marker, always `"configuration"`.

#### `class` {#module-system-lib-evalModules-return-value-_configurationClass}

The [`class` argument](#module-system-lib-evalModules-param-class).

          decls

      ));

in

rec {

  /*

    Evaluate a set of modules.  The result is a set with the attributes:

      ‘options’: The nested set of all option declarations,

      ‘config’: The nested set of all option values.

      ‘type’: A module system type representing the module set as a submodule,

            to be extended by configuration from the containing module set.

            This is also available as the module argument ‘moduleType’.

      ‘extendModules’: A function similar to ‘evalModules’ but building on top

            of the module set. Its arguments, ‘modules’ and ‘specialArgs’ are

            added to the existing values.

            Using ‘extendModules’ a few times has no performance impact as long

            as you only reference the final ‘options’ and ‘config’.

            If you do reference multiple ‘config’ (or ‘options’) from before and

            after ‘extendModules’, performance is the same as with multiple

            ‘evalModules’ invocations, because the new modules' ability to

            override existing configuration fundamentally requires a new

            fixpoint to be constructed.

            This is also available as a module argument.

      ‘_module’: A portion of the configuration tree which is elided from

            ‘config’. It contains some values that are mostly internal to the

            module system implementation.

  /* See https://nixos.org/manual/nixpkgs/unstable/#module-system-lib-evalModules

     or file://./../doc/module-system/module-system.chapter.md

     !!! Please think twice before adding to this argument list! The more

     that is specified here instead of in the modules themselves the harder

                  # there's _module.args. If specialArgs.modulesPath is defined it will be

                  # used as the base path for disabledModules.

                  specialArgs ? {}

                , # This would be remove in the future, Prefer _module.args option instead.

                  args ? {}

                , # `class`:

                  # A nominal type for modules. When set and non-null, this adds a check to

                  # make sure that only compatible modules are imported.

                  # This would be remove in the future, Prefer _module.args option instead.

                  class ? null

                , args ? {}

                , # This would be remove in the future, Prefer _module.check option instead.

                  check ? true

                }:

      merged =

        let collected = collectModules

          class

          (specialArgs.modulesPath or "")

          (regularModules ++ [ internalModule ])

          ({ inherit lib options config specialArgs; } // specialArgs);

        prefix ? [],

        }:

          evalModules (evalModulesArgs // {

            inherit class;

            modules = regularModules ++ modules;

            specialArgs = evalModulesArgs.specialArgs or {} // specialArgs;

            prefix = extendArgs.prefix or evalModulesArgs.prefix or [];

          });

      type = lib.types.submoduleWith {

        inherit modules specialArgs;

        inherit modules specialArgs class;

      };

      result = withWarnings {

        _type = "configuration";

        options = checked options;

        config = checked (removeAttrs config [ "_module" ]);

        _module = checked (config._module);

        inherit extendModules type;

        class = class;

      };

    in result;

  # collectModules :: (modulesPath: String) -> (modules: [ Module ]) -> (args: Attrs) -> [ Module ]

  # collectModules :: (class: String) -> (modulesPath: String) -> (modules: [ Module ]) -> (args: Attrs) -> [ Module ]

  #

  # Collects all modules recursively through `import` statements, filtering out

  # all modules in disabledModules.

  collectModules = let

  collectModules = class: let

      # Like unifyModuleSyntax, but also imports paths and calls functions if necessary

      loadModule = args: fallbackFile: fallbackKey: m:

        if isFunction m || isAttrs m then

          unifyModuleSyntax fallbackFile fallbackKey (applyModuleArgsIfFunction fallbackKey m args)

        if isFunction m then

          unifyModuleSyntax fallbackFile fallbackKey (applyModuleArgs fallbackKey m args)

        else if isAttrs m then

          if m._type or "module" == "module" then

            unifyModuleSyntax fallbackFile fallbackKey m

          else if m._type == "if" || m._type == "override" then

            loadModule args fallbackFile fallbackKey { config = m; }

          else

            throw (

              "Could not load a value as a module, because it is of type ${lib.strings.escapeNixString m._type}"

              + lib.optionalString (fallbackFile != unknownModule) ", in file ${toString fallbackFile}."

              + lib.optionalString (m._type == "configuration") " If you do intend to import this configuration, please only import the modules that make up the configuration. You may have to create a `let` binding, file or attribute to give yourself access to the relevant modules.\nWhile loading a configuration into the module system is a very sensible idea, it can not be done cleanly in practice."

               # Extended explanation: That's because a finalized configuration is more than just a set of modules. For instance, it has its own `specialArgs` that, by the nature of `specialArgs` can't be loaded through `imports` or the the `modules` argument. So instead, we have to ask you to extract the relevant modules and use those instead. This way, we keep the module system comparatively simple, and hopefully avoid a bad surprise down the line.

            )

        else if isList m then

          let defs = [{ file = fallbackFile; value = m; }]; in

          throw "Module imports can't be nested lists. Perhaps you meant to remove one level of lists? Definitions: ${showDefs defs}"

        else unifyModuleSyntax (toString m) (toString m) (applyModuleArgsIfFunction (toString m) (import m) args);

      checkModule =

        if class != null

        then

          m:

            if m._class != null -> m._class == class

            then m

            else

              throw "The module ${m._file or m.key} was imported into ${class} instead of ${m._class}."

        else

          m: m;

      /*

      Collects all modules recursively into the form

          };

        in parentFile: parentKey: initialModules: args: collectResults (imap1 (n: x:

          let

            module = loadModule args parentFile "${parentKey}:anon-${toString n}" x;

            module = checkModule (loadModule args parentFile "${parentKey}:anon-${toString n}" x);

            collectedImports = collectStructuredModules module._file module.key module.imports args;

          in {

            key = module.key;

        else config;

    in

    if m ? config || m ? options then

      let badAttrs = removeAttrs m ["_file" "key" "disabledModules" "imports" "options" "config" "meta" "freeformType"]; in

      let badAttrs = removeAttrs m ["_class" "_file" "key" "disabledModules" "imports" "options" "config" "meta" "freeformType"]; in

      if badAttrs != {} then

        throw "Module `${key}' has an unsupported attribute `${head (attrNames badAttrs)}'. This is caused by introducing a top-level `config' or `options' attribute. Add configuration attributes immediately on the top level instead, or move all of them (namely: ${toString (attrNames badAttrs)}) into the explicit `config' attribute."

      else

        { _file = toString m._file or file;

          _class = m._class or null;

          key = toString m.key or key;

          disabledModules = m.disabledModules or [];

          imports = m.imports or [];

      # shorthand syntax

      lib.throwIfNot (isAttrs m) "module ${file} (${key}) does not look like a module."

      { _file = toString m._file or file;

        _class = m._class or null;

        key = toString m.key or key;

        disabledModules = m.disabledModules or [];

        imports = m.require or [] ++ m.imports or [];

        options = {};

        config = addFreeformType (removeAttrs m ["_file" "key" "disabledModules" "require" "imports" "freeformType"]);

        config = addFreeformType (removeAttrs m ["_class" "_file" "key" "disabledModules" "require" "imports" "freeformType"]);

      };

  applyModuleArgsIfFunction = key: f: args@{ config, options, lib, ... }: if isFunction f then

  applyModuleArgsIfFunction = key: f: args@{ config, options, lib, ... }:

    if isFunction f then applyModuleArgs key f args else f;

  applyModuleArgs = key: f: args@{ config, options, lib, ... }:

    let

      # Module arguments are resolved in a strict manner when attribute set

      # deconstruction is used.  As the arguments are now defined with the

      # context on the explicit arguments of "args" too. This update

      # operator is used to make the "args@{ ... }: with args.lib;" notation

      # works.

    in f (args // extraArgs)

  else

    f;

    in f (args // extraArgs);

  /* Merge a list of modules.  This will recurse over the option

     declarations in all modules, combining them into a single set.

    _file = file;

    config = lib.importTOML file;

  };

  private = lib.mapAttrs

    (k: lib.warn "External use of `lib.modules.${k}` is deprecated. If your use case isn't covered by non-deprecated functions, we'd like to know more and perhaps support your use case well, instead of providing access to these low level functions. In this case please open an issue in https://github.com/nixos/nixpkgs/issues/.")

    {

      inherit

        applyModuleArgsIfFunction

        dischargeProperties

        evalOptionValue

        mergeModules

        mergeModules'

        pushDownProperties

        unifyModuleSyntax

        ;

      collectModules = collectModules null;

    };

in

private //

{

  # NOTE: not all of these functions are necessarily public interfaces; some

  #       are just needed by types.nix, but are not meant to be consumed

  #       externally.

  inherit

    defaultOrderPriority

    defaultOverridePriority

    defaultPriority

    doRename

    evalModules

    filterOverrides

    filterOverrides'

    fixMergeModules

    fixupOptionType  # should be private?

    importJSON

    importTOML

    mergeDefinitions

    mergeOptionDecls  # should be private?

    mkAfter

    mkAliasAndWrapDefinitions

    mkAliasAndWrapDefsWithPriority

    mkAliasDefinitions

    mkAliasIfDef

    mkAliasOptionModule

    mkAliasOptionModuleMD

    mkAssert

    mkBefore

    mkChangedOptionModule

    mkDefault

    mkDerivedConfig

    mkFixStrictness

    mkForce

    mkIf

    mkImageMediaOverride

    mkMerge

    mkMergedOptionModule

    mkOptionDefault

    mkOrder

    mkOverride

    mkRemovedOptionModule

    mkRenamedOptionModule

    mkRenamedOptionModuleWith

    mkVMOverride

    setDefaultModuleLocation

    sortProperties;

}

checkConfigOutput '^true$' "$@" ./define-module-check.nix

# Check coerced value.

set --

checkConfigOutput '^"42"$' config.value ./declare-coerced-value.nix

checkConfigOutput '^"24"$' config.value ./declare-coerced-value.nix ./define-value-string.nix

checkConfigError 'A definition for option .* is not.*string or signed integer convertible to it.*. Definition values:\n\s*- In .*: \[ \]' config.value ./declare-coerced-value.nix ./define-value-list.nix

## Freeform modules

# Assigning without a declared option should work

checkConfigOutput '^"24"$' config.value ./freeform-attrsOf.nix ./define-value-string.nix

# Shorthand modules interpret `meta` and `class` as config items

checkConfigOutput '^true$' options._module.args.value.result ./freeform-attrsOf.nix ./define-freeform-keywords-shorthand.nix

# No freeform assignments shouldn't make it error

checkConfigOutput '^{ }$' config ./freeform-attrsOf.nix

# but only if the type matches

# because of an `extendModules` bug, issue 168767.

checkConfigOutput '^1$' config.sub.specialisation.value ./extendModules-168767-imports.nix

# Class checks, evalModules

checkConfigOutput '^{ }$' config.ok.config ./class-check.nix

checkConfigOutput '"nixos"' config.ok.class ./class-check.nix

checkConfigError 'The module .*/module-class-is-darwin.nix was imported into nixos instead of darwin.' config.fail.config ./class-check.nix

checkConfigError 'The module foo.nix#darwinModules.default was imported into nixos instead of darwin.' config.fail-anon.config ./class-check.nix

# Class checks, submoduleWith

checkConfigOutput '^{ }$' config.sub.nixosOk ./class-check.nix

checkConfigError 'The module .*/module-class-is-darwin.nix was imported into nixos instead of darwin.' config.sub.nixosFail.config ./class-check.nix

# submoduleWith type merge with different class

checkConfigError 'error: A submoduleWith option is declared multiple times with conflicting class values "darwin" and "nixos".' config.sub.mergeFail.config ./class-check.nix

# _type check

checkConfigError 'Could not load a value as a module, because it is of type "flake", in file .*/module-imports-_type-check.nix' config.ok.config ./module-imports-_type-check.nix

checkConfigOutput '^true$' "$@" config.enable ./declare-enable.nix ./define-enable-with-top-level-mkIf.nix

checkConfigError 'Could not load a value as a module, because it is of type "configuration", in file .*/import-configuration.nix.*please only import the modules that make up the configuration.*' config ./import-configuration.nix

# doRename works when `warnings` does not exist.

checkConfigOutput '^1234$' config.c.d.e ./doRename-basic.nix

# doRename adds a warning.