Skip to content
  • Belhorn, Matt's avatar
    Finalizes multi-tiered migrations. · 4c0b6754
    Belhorn, Matt authored
    Resolves #129
    Resolves #121
    Resolves #82
    Resolves #269 (possibly)
    
    Applies migration fixes for each system
    ----------------------------------------
    
    Fixes OpenSSL packages on all systems
    
    Improves `provision` tmp space usage
    ------------------------------------
    
    Wipes staging area between builds in `cmd.provision` to maintain adequate space
    in limited C3/C4 `tmp` ramdisks.
    
    Adds host-specific post-deploy action scripts.
    ----------------------------------------------
    
    In particular, this resolves #231 by allowing the last summitdev deploy job to
    run an LMOD cache update script after the job exits successfully.
    
    This same behavior can be customized and generalized to each host that needs
    post-deploy commands to be run.
    
    Reorganizes job names
    ---------------------
    
    New naming scheme:
    - avoids unintentional job collapses in pipeline view
    - allows for jobs that do not require a specific tier identifier to be
      constructed in the same manner as staging and deploy jobs. In particular, it
      is no longer necessary to set a generic SWCI_TIER variable in the job
      environment.
    
    Job names must take the form of:
    
    ```
    {PREFIX}{SWCI_HOST}.{SWCI_TIER}{SUFFIX}
    ```
    
    Where
    
    - `{PREFIX}` can be any (including empty) string. If the prefix contains two
      numbers matching the regex `[\d]*[:\/\s][\d]*\s*` then the job listing in the
      pipeline view will be collapsed/collapsable provided the only difference
      between the set of jobs is the numbers.
    - `{SWCI_HOST}` must match the variable set for the job exactly.
    - `{SWCI_TIER}` must match the tier configuration directory name exactly. In the
      case where the job does not need a specific tier to run, the tier name can be
      `none`. It must be separated from `{SWCI_HOST}` with a single `.` (period) and
      must only contain the characters `[a-zA-Z0-9_-]`.
    - `{SUFFIX}` is optional, but must start with a character outside the set of
      `[a-zA-Z0-9_-]`. For example, any of ` .,:@/` that is permitted by YAML.
    
    This commit also
    - Adds staging wipe job for all hosts.
    - Enables jobs for migrated systems not added in initial tiered spack migration.
      - Enables jobs for eslogin-c3 and eslogin-c4.
      - Enables jobs for the lDTNs.
      - Enables jobs for OpenDTNs.
      - Enables jobs for Eos compute nodes.
    
    Cleans up CI scripts
    --------------------
    
    - Gives failed-build-log artifacts unique names.
    - Removes unused scripts to reduce clutter.
    - Moves all CI scripts into common CI bin directory.
    - Makes CI script names consistent. All scripts used for CI should start with
      `swci_` prefix and have no shell suffix.
    - Moves host environment `before_script` commands to sourceable external scripts
      so they can be re-used in development environments.
    - Generalizes SWCI scripts to be usable interactively in development
      environments.
    - Adds post failure job to cleanup and collect artifacts.
    - Adds CI script to create a mirror of as many packages known to SWCI as
      possible.
    - Configures NCRC machines to use uamntr web mirror.
    - Adds 'create mirror' and NCRC staging wipeout jobs.
    - Adds initial changes to documentation (Resolves #215, possibly)
    - Removes all exit 0 statuses and lets the exit codes of the contents speak for
      themselves.
    
    Adds deployment of runtime environment scripts
    ----------------------------------------------
    
    Adds scripts needed for runtime shell initialization by host-wide configurations
    like `/etc/profile` and the environment module system to `/sw/{HOST}/etc` for
    their consumption.
    
    Mirrors
    -------
    
    Hosting the mirror on 'http://users.nccs.gov/~uamntr/mirror` requires a host on
    which `/ccs/wwwusers` is mounted.  Ideally, this would be done from a DTN or an
    administrative spack instance on a non-production resource, but that is not
    possible at this time.  In the meantime, we will conscript Rhea for this
    purpose.
    
    Either way, the spack instance used must be fully configured for use as creating
    a mirror occasionally invokes concretization.
    
    The NCCS web mirror is added to all hosts.
    
    Host specifc environment settings
    ---------------------------------
    
    Migrates host-specific environment settings out of gitlab CI job `before_script`
    blocks in favor of separate external scripts `swci_setup_host_env` sourced for
    each machine. The scripts are stored in a heirarchy that is added to the PATH in
    order of precidence:
    
    1. `${spack}/etc/nccs-sw-ci/${SWCI_HOST}/${SWCI_TIER}/bin`
    2. `${spack}/etc/nccs-sw-ci/${SWCI_HOST}/bin`
    3. `${pack}/.ci/bin`
    
    where the lowest priority script is simply a no-op.
    
    Updates documentation
    ---------------------
    
    Adds greater detail to tiered spack instance documentation.
    
    Removes outdated READMEs.
    
    Standardizes compiler specs
    ---------------------------
    
    Adds missing compiler modules on Eos.
    
    Adds extra_rpaths field to compilers used to provide external packages. This
    especially affects systems with CrayPE.
    
    Cray systems with raw compilers in `/opt/` need to add to the RPATHs the
    libraries in `/opt` so that packages built with these compilers can be used
    without needing to load the compiler modulefile. This happens, for instance,
    when the compiler was used to build an 'external' build-time dependency for a
    higher tier:
    
      cmake at tier 0-core on eos used to build lapack at tier 1-compute on eos with
      PGI.
    
    Refactors manifest specs
    ------------------------
    
    Refactors manifest specs to match Spack's normalized pattern.
    
    Amends explicit python v2.7.12 dependency for adios.
    
    Adds missing version to lapack spec.
    
    Fixes OpenDTN manifest yaml typo.
    
    Limits external python to PGI builds.
    
    Allows python to be buildable, when not using PGI on summitdev.
    4c0b6754