Skip to content
Snippets Groups Projects
README.md 2.94 KiB
Newer Older
Belhorn, Matt's avatar
Belhorn, Matt committed
# Setting up Spack for Plato on Summit

Spack generally requires special configuration to work correctly on Summit,
primarily because Summit's architecture is uncommon among the authors of spack
packages.

The necessary spack configs are in `/etc/spack` of this repo. A fork of the
OpenBLAS package that has been extended to work better on PPC64LE architecture
is also included in `/repos/ppc64le` which is referenced by the spack
`repos.yaml` configuration provided using relative paths.

The script `setup-spack-for-plato.sh` walks through the process of setting up
a spack instance and is annotated to provide a little explanation for each step. Each of
the spack configs are also lightly annotated to explain why some settings should
be what they are.

It should be entirely possible to, on Summit, call:

```
git clone https://code.ornl.gov/m9b/plato-on-summit.git plato
cd plato
./setup-spack-for-plato.sh
```

to build Plato on Summit using gcc v4.8.5 without the CUDA variant.

Using the resulting spack instance in the future to build other variants of
Plato interactively can be done with:

```
cd plato # directory where repo was cloned
module purge
# mv $HOME/.spack $HOME/.spack.backup # optional if having problems
. ./spack/share/spack/setup-env.sh
spack install ...
```

## Regarding OpenBLAS in Specs

The comments in `setup-spack-for-plato.sh` regarding OpenBLAS are important.

The default compiler used when no compiler is given as part of the spec is set
in `packages.yaml` to be the OS-provided GCC v4.8.5 toolchain in `/usr/bin`.
This is done because packages built with the OS toolchain don't require compiler
modules to be loaded due to libstdc++ issues. Spack builds use RPATHs and are
generally less affected by dependency modules, but spack builds are not immune
from occasionally needing compiler-specific environment settings provided
through environment modules at runtime.

However, OpenBLAS 0.3.6 introduced Power9 optimizations which use flags
unavailable to GCC4. Therefore, it is necessary to build OpenBLAS v0.3.5 with
`cpu_target=POWER8` when using GCC v5.X or older.

The default version/cpu_target option can be set in `packages.yaml`
if the user knows which toolchain (GCC v6 or newer or GCC v5.x or older) they
will generally be using. The default OpenBLAS options set by the provided
`packages.yaml` config file assumes the use of GCC6 or newer when not explicitly
set via a package spec.

Because the default toolchain is GCC 4.8.5 and the default options for OpenBLAS
assume the use of GCC6 or newer, it is generally necessary to declare the
OpenBLAS version and cpu_target OR compiler (if GCC6 or newer) in the package
spec if it depends on BLAS.

Examples:

No compiler spec, so OpenBLAS version and target must be specified because GCC4
will be used.
```
spack spec -lINt platoengine ^ppc64le.openblas@0.3.5 cpu_target=POWER8
```

GCC7 being used, so OpenBLAS defaults in `packages.yaml` take effect and are valid.
```
spack spec -lINt platoengine%gcc@7.4.0
```