Merge pull request #251006 from Uthar/doc-lisp001

# lisp-modules {#lisp}

This document describes the Nixpkgs infrastructure for building Common Lisp

libraries that use ASDF (Another System Definition Facility). It lives in

`pkgs/development/lisp-modules`.

systems that use [ASDF](https://asdf.common-lisp.dev/) (Another System

Definition Facility). It lives in `pkgs/development/lisp-modules`.

## Overview {#lisp-overview}

The main entry point of the API are the Common Lisp implementation packages

(e.g. `abcl`, `ccl`, `clasp-common-lisp`, `clisp` `ecl`, `sbcl`)

themselves. They have the `pkgs` and `withPackages` attributes, which can be

used to discover available packages and to build wrappers, respectively.

The `pkgs` attribute set contains packages that were automatically imported from

Quicklisp, and any other manually defined ones. Not every package works for all

the CL implementations (e.g. `nyxt` only makes sense for `sbcl`).

The `withPackages` function is of primary utility. It is used to build runnable

wrappers, with a pinned and pre-built ASDF FASL available in the `ASDF`

environment variable, and `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS`

configured to find the desired systems on runtime.

With a few exceptions, the primary thing that the infrastructure does is to run

`asdf:load-system` for each system specified in the `systems` argument to

`build-asdf-system`, and save the FASLs to the Nix store. Then, it makes these

FASLs available to wrappers. Any other use-cases, such as producing SBCL

executables with `sb-ext:save-lisp-and-die`, are achieved via overriding the

`buildPhase` etc.

themselves (e.g. `abcl`, `ccl`, `clasp-common-lisp`, `clisp`, `ecl`,

`sbcl`). They have the `pkgs` and `withPackages` attributes, which can be used

to discover available packages and to build wrappers, respectively.

The `pkgs` attribute set contains packages that were automatically

[imported](#lisp-importing-packages-from-quicklisp) from Quicklisp, and any

other [manually defined](#lisp-defining-packages-inside) ones. Not every package

works for all the CL implementations (e.g. `nyxt` only makes sense for `sbcl`).

The `withPackages` function is of primary utility. It is used to build

[runnable wrappers](#lisp-building-wrappers), with a pinned and pre-built

[ASDF FASL](#lisp-loading-asdf) available in the `ASDF` environment variable,

and `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` configured to

[find the desired systems on runtime](#lisp-loading-systems).

In addition, Lisps have the `withOverrides` function, which can be used to

substitute any package in the scope of their `pkgs`. This will be useful

together with `overrideLispAttrs` when dealing with slashy ASDF systems, because

they should stay in the main package and be build by specifying the `systems`

[substitute](#lisp-including-external-pkg-in-scope) any package in the scope of

their `pkgs`. This will also be useful together with `overrideLispAttrs` when

[dealing with slashy systems](#lisp-dealing-with-slashy-systems), because they

should stay in the main package and be built by specifying the `systems`

argument to `build-asdf-system`.

## The 90% use case example {#lisp-use-case-example}

Then, in a shell:

```

$ result/bin/sbcl

$ sbcl

* (load (sb-ext:posix-getenv "ASDF"))

* (asdf:load-system 'alexandria)

```

let

  sbcl' = sbcl.withPackages (ps: [ ps.alexandria ]);

in mkShell {

  buildInputs = [ sbcl' ];

  packages = [ sbcl' ];

}

```

## Importing packages from Quicklisp {#lisp-importing-packages-from-quicklisp}

The library is able to very quickly import all the packages distributed by

Quicklisp by parsing its `releases.txt` and `systems.txt` files. These files are

available from [http://beta.quicklisp.org/dist/quicklisp.txt].

To save some work of writing Nix expressions, there is a script that imports all

the packages distributed by Quicklisp into `imported.nix`. This works by parsing

its `releases.txt` and `systems.txt` files, which are published every couple of

months on [quicklisp.org](http://beta.quicklisp.org/dist/quicklisp.txt).

The import process is implemented in the `import` directory as Common Lisp

functions in the `org.lispbuilds.nix` ASDF system. To run the script, one can

code in the `org.lispbuilds.nix` ASDF system. To run the script, one can

execute `ql-import.lisp`:

```

cd pkgs/development/lisp-modules

nix-shell --run 'sbcl --script ql-import.lisp'

```

The script will:

1. Download the latest Quicklisp `systems.txt` and `releases.txt` files

2. Generate an SQLite database of all QL systems in `packages.sqlite`

2. Generate a temporary SQLite database of all QL systems in `packages.sqlite`

3. Generate an `imported.nix` file from the database

The maintainer's job there is to:

(The `packages.sqlite` file can be deleted at will, because it is regenerated

each time the script runs.)

1. Re-run the `ql-import.lisp` script

2. Add missing native dependencies in `ql.nix`

3. For packages that still don't build, package them manually in `packages.nix`

The maintainer's job is to:

1. Re-run the `ql-import.lisp` script when there is a new Quicklisp release

2. [Add any missing native dependencies](#lisp-quicklisp-adding-native-dependencies) in `ql.nix`

3. For packages that still don't build, [package them manually](#lisp-defining-packages-inside) in `packages.nix`

Also, the `imported.nix` file **must not be edited manually**! It should only be

generated as described in this section.

generated as described in this section (by running `ql-import.lisp`).

### Adding native dependencies {#lisp-quicklisp-adding-native-dependencies}

The previous implementation of `lisp-modules` didn't fully trust the Quicklisp

data, because there were times where the dependencies specified were not

complete, and caused broken builds. It instead used a `nix-shell` environment to

complete and caused broken builds. It instead used a `nix-shell` environment to

discover real dependencies by using the ASDF APIs.

The current implementation has chosen to trust this data, because it's faster to

During Quicklisp import:

- `+` in names are converted to `_plus{_,}`: `cl+ssl`->`cl_plus_ssl`, `alexandria+`->`alexandria_plus`

- `.` to `_dot_`: `iolib.base`->`iolib_dot_base`

- `+` in names is converted to `_plus{_,}`: `cl+ssl`->`cl_plus_ssl`, `alexandria+`->`alexandria_plus`

- `.` in names is converted to `_dot_`: `iolib.base`->`iolib_dot_base`

- names starting with a number have a `_` prepended (`3d-vectors`->`_3d-vectors`)

- `_` in names is converted to `__` for reversibility

## Defining packages manually inside Nixpkgs {#lisp-defining-packages-inside}

New packages, that for some reason are not in Quicklisp, and so cannot be

auto-imported, can be written in the `packages.nix` file.

Packages that for some reason are not in Quicklisp, and so cannot be

auto-imported, or don't work straight from the import, are defined in the

`packages.nix` file.

In that file, use the `build-asdf-system` function, which is a wrapper around

`mkDerivation` for building ASDF systems. Various other hacks are present, such

as `build-with-compile-into-pwd` for systems which create files during

compilation.

compilation (such as cl-unicode).

The `build-asdf-system` function is documented with comments in

`nix-cl.nix`. Also, `packages.nix` is full of examples of how to use it.

The `build-asdf-system` function is documented

[here](#lisp-defining-packages-outside). Also, `packages.nix` is full of

examples of how to use it.

## Defining packages manually outside Nixpkgs {#lisp-defining-packages-outside}

Lisp derivations (`abcl`, `sbcl` etc.) also export the `buildASDFSystem`

function, which is the same as `build-asdf-system`, except for the `lisp`

argument which is set to the given CL implementation.

function, which is similar to `build-asdf-system` from `packages.nix`, but is

part of the public API.

It takes the following arguments:

- `pname`: the package name

- `version`: the package version

- `src`: the package source

- `patches`: patches to apply to the source before build

- `nativeLibs`: native libraries used by CFFI and grovelling

- `javaLibs`: Java libraries for ABCL

- `lispLibs`: dependencies on other packages build with `buildASDFSystem`

- `systems`: list of systems to build

It can be used to define packages outside Nixpkgs, and, for example, add them

into the package scope with `withOverrides` which will be discussed later on.

into the package scope with `withOverrides`.

### Including an external package in scope {#lisp-including-external-pkg-in-scope}

})

```

## Overriding packages in scope {#lisp-overriding-packages-in-scope}

Packages can be woven into a new scope by using `withOverrides`:

```

let

  sbcl' = sbcl.withOverrides (self: super: {

    alexandria = super.alexandria.overrideLispAttrs (oldAttrs: rec {

      pname = "alexandria";

      version = "1.4";

      src = fetchFromGitLab {

        domain = "gitlab.common-lisp.net";

        owner = "alexandria";

        repo = "alexandria";

        rev = "v${version}";

        hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";

      };

    });

  });

in builtins.elemAt sbcl'.pkgs.bordeaux-threads.lispLibs 0

```

### Dealing with slashy systems {#lisp-dealing-with-slashy-systems}

Slashy (secondary) systems should not exist in their own packages! Instead, they

})

```

See the respective section on using `withOverrides` for how to weave it back

into `ecl.pkgs`.

See the [respective section](#lisp-including-external-pkg-in-scope) on using

`withOverrides` for how to weave it back into `ecl.pkgs`.

Note that sometimes the slashy systems might not only have more dependencies

than the main one, but create a circular dependency between `.asd`

implementations (`abcl`, `ecl`, `sbcl` etc.):

```

sbcl.withPackages (ps: [ ps.alexandria ps.bordeaux-threads ])

nix-shell -p 'sbcl.withPackages (ps: [ ps.alexandria ps.bordeaux-threads ])'

```

Such a wrapper can then be executed like this:

Such a wrapper can then be used like this:

```

result/bin/sbcl

$ sbcl

* (load (sb-ext:posix-getenv "ASDF"))

* (asdf:load-system 'alexandria)

* (asdf:load-system 'bordeaux-threads)

```

### Loading ASDF {#lisp-loading-asdf}