Loading README.md +40 −0 Original line number Diff line number Diff line Loading @@ -78,6 +78,16 @@ the Python Package Index, *PyPI*, using the tool `pip` (aka `pip2`) or `pip3` depending on which version of python is being used. These commands are added to your `PATH` when the base python environment module is loaded. When the version of `pip` being used is outdated, it may refuse to connect to `pypi.python.org` due to changed SSL/TLS certificates. Please inform `help@olcf.ornl.gov` when this is the case. However, to work around the issue in the meantime, add the flag `--trusted-host pypi.python.org` to your `pip` incantations: ``` pip install --trusted-host pypi.python.org --user <PKG> ``` Alternatively, many packages can be installed by running a `setup.py` script provided with the package. These scripts use a number of distribution tools that are provided with the core python installation. Loading @@ -94,6 +104,36 @@ binaries and wheels and instead compile any binaries for the current environment using the system compiler. Packages installed this way are known to the python interpreter without any extra setup. However, because your home directory is shared with multiple resources, you may wish to override the default user site-packages directory. If you choose to do this, the path to your customized user-site base must be added to the environment everytime you change the python module. To do this, manually export the environment variable: ``` export PYTHONUSERBASE="$HOME/.local/<HOSTNAME>/python<VERSION> ``` The path you choose is arbitrary and can be put under `/ccs/proj/<PROJECTID>` if you with to make the packages installed there available on Cray compute nodes. It is recommended that the chosen path reflects at least both the host and python interperetor used to install packages there. In the above example, `<HOSTNAME>` and `python<VERSION>` allow you to separate packages for more than one host and python interperetor. In some cases, it may even be helpful to organize custom user site base prefixes by compiler environment or a particular application. You can organize these user site directories to your liking. I recommend: ``` PYVERSION=$(python -c "import platform; print 'python{0}'.format(platform.python_version()))" MYHOST="${HOSTNAME%%-*}" export PYTHONUSERBASE="$HOME/.local/$MYHOST/$PYVERSION ``` **Again, if you choose to change the default in this manner, the environment variable will need to be set, updated, or unset manually every time the python environment module is loaded or changed.** ## Virtual Environments A robust way to build a customized python stack is to build it from Loading install_anaconda_on_cray.sh 0 → 100755 +134 −0 Original line number Diff line number Diff line #!/bin/bash # Install Anaconda. # Setup the Environment ## Get rid of any python currently loaded. . $MODULESHOME/init/bash module unload python xalt ## We must use the GNU PE. if [ -n "$CRAYPE_VERSION" ]; then module unload PrgEnv-pgi PrgEnv-gnu PrgEnv-intel PrgEnv-cray module load PrgEnv-gnu dynamic-link else module unload PE-intel PE-gnu module load PE-gnu fi # Use a particular anaconda installer version. # Index from 1 in zsh, index from 0 in bash. URL_BASE="https://repo.continuum.io/archive" AVAIL=( $(curl "${URL_BASE}/" 2>/dev/null | \ tr '\n' ' ' | \ sed -e 's/\(<tr>\)\s*/\n\1/g' \ -e 's/\s{2,}//g' \ -e 's/<a href="\([^"]*\)">[^>]*>/\1/g' | \ awk -F "</*td[^>]*>|</*tr>" '/<\/*t[rd]>/ {printf "%s:%s\n", $3, $9 }' | \ grep "Linux-x86_64" | \ grep -E "Anaconda[23]" | \ sort) ) lc() { echo "$(tr "[:upper:]" "[:lower:]" <<< "$1")" } select version in ${AVAIL[@]%%.sh:*}; do if [ -n "${version}" ]; then for i in "${AVAIL[@]}"; do if [[ "${i}" =~ ${version} ]]; then _ver="${version%-Linux*}" ANACONDA_INSTALLER_NAME="${version}.sh" ANACONDA_INSTALLER_HASH="${i##*:}" ANACONDA_PYTHON_VERSION="$(lc ${version%%-*})" PACKAGE_VERSION="${_ver#*-}" PACKAGE_NAME="python_${ANACONDA_PYTHON_VERSION}" break fi done else echo "ERROR: Section invalid." exit 1 fi break done HOST="${HOSTNAME%%-*}" PREFIX_BASE="/sw/.testing/$USER/python/${HOST}" PREFIX="${PREFIX_BASE}/opt/${PACKAGE_NAME}/${PACKAGE_VERSION}" echo "Installing $ANACONDA_PYTHON_VERSION ($PACKAGE_VERSION) to ${PREFIX}" # Install anaconda. mkdir -p "/tmp/$USER" INSTALLER="/tmp/$USER/$ANACONDA_INSTALLER_NAME" if [ ! -e "${INSTALLER}" ]; then wget "${URL_BASE}/${ANACONDA_INSTALLER_NAME}" -O "${INSTALLER}" _dl_hash="$(md5sum ${INSTALLER} 2>&1 | grep -m 1 -oe "^[^ ]*")" [ "$_dl_hash" != "$ANACONDA_INSTALLER_HASH" ] && return 1 chmod a+x "${INSTALLER}" fi if [ ! -d "${PREFIX}" ]; then ${INSTALLER} -b -p "${PREFIX}" else return 1 fi # Add anaconda deploy to the PATH if [ -z "$OPATH" ]; then # Anaconda to the PATH if it hasn't been already. OPATH="$PATH" export PATH="$PREFIX/bin:$PATH" fi # Update pip to latest version pip install -v --upgrade pip # If on a Cray, if [ -n "${CRAYPE_VERSION}" ]; then # Install mpi4py in anaconda root, built against cray-mpich CC=cc MPICC=cc pip install -v --no-binary :all: mpi4py else pip install -v --no-binary :all: mpi4py fi # Make the site install read-only. This is useful when installed in a location # that normally has group write access to prevent others from modifying the # installation. chmod -R go-w "${PREFIX}" ## By making the install prefix (and everything below it) read-only, conda will ## allow users to install custom envs at `$HOME/.conda/envs` by default. ## ## Users can over-ride with `$HOME/.condarc` containing: ## |envs_dirs: ## | - /ccs/proj/<projid>/... ## | - ~/.conda/envs # Reset the PATH. [ -n "${OPATH}" ] && export PATH="${OPATH}" && unset OPATH # Generate a modulefile. MODULEFILE="${PREFIX_BASE}/modulefiles/${PACKAGE_NAME}/${PACKAGE_VERSION}" mkdir -p $(dirname ${MODULEFILE}) cat << EOF > ${MODULEFILE} #%Module proc ModulesHelp { } { puts stderr "${PACKAGE_NAME} ${PACKAGE_VERSION}" puts stderr "" } # One line description module-whatis "${PACKAGE_NAME} ${PACKAGE_VERSION}" set HOST ${HOST} set PREFIX ${PREFIX} conflict python* prepend-path PATH \$PREFIX/bin unsetenv PYTHONSTARTUP setenv PYTHONUSERBASE \$env(HOME)/.local/\$HOST/${PACKAGE_NAME}/${PACKAGE_VERSION} setenv CONDALIBS \$PREFIX/lib EOF Loading
README.md +40 −0 Original line number Diff line number Diff line Loading @@ -78,6 +78,16 @@ the Python Package Index, *PyPI*, using the tool `pip` (aka `pip2`) or `pip3` depending on which version of python is being used. These commands are added to your `PATH` when the base python environment module is loaded. When the version of `pip` being used is outdated, it may refuse to connect to `pypi.python.org` due to changed SSL/TLS certificates. Please inform `help@olcf.ornl.gov` when this is the case. However, to work around the issue in the meantime, add the flag `--trusted-host pypi.python.org` to your `pip` incantations: ``` pip install --trusted-host pypi.python.org --user <PKG> ``` Alternatively, many packages can be installed by running a `setup.py` script provided with the package. These scripts use a number of distribution tools that are provided with the core python installation. Loading @@ -94,6 +104,36 @@ binaries and wheels and instead compile any binaries for the current environment using the system compiler. Packages installed this way are known to the python interpreter without any extra setup. However, because your home directory is shared with multiple resources, you may wish to override the default user site-packages directory. If you choose to do this, the path to your customized user-site base must be added to the environment everytime you change the python module. To do this, manually export the environment variable: ``` export PYTHONUSERBASE="$HOME/.local/<HOSTNAME>/python<VERSION> ``` The path you choose is arbitrary and can be put under `/ccs/proj/<PROJECTID>` if you with to make the packages installed there available on Cray compute nodes. It is recommended that the chosen path reflects at least both the host and python interperetor used to install packages there. In the above example, `<HOSTNAME>` and `python<VERSION>` allow you to separate packages for more than one host and python interperetor. In some cases, it may even be helpful to organize custom user site base prefixes by compiler environment or a particular application. You can organize these user site directories to your liking. I recommend: ``` PYVERSION=$(python -c "import platform; print 'python{0}'.format(platform.python_version()))" MYHOST="${HOSTNAME%%-*}" export PYTHONUSERBASE="$HOME/.local/$MYHOST/$PYVERSION ``` **Again, if you choose to change the default in this manner, the environment variable will need to be set, updated, or unset manually every time the python environment module is loaded or changed.** ## Virtual Environments A robust way to build a customized python stack is to build it from Loading
install_anaconda_on_cray.sh 0 → 100755 +134 −0 Original line number Diff line number Diff line #!/bin/bash # Install Anaconda. # Setup the Environment ## Get rid of any python currently loaded. . $MODULESHOME/init/bash module unload python xalt ## We must use the GNU PE. if [ -n "$CRAYPE_VERSION" ]; then module unload PrgEnv-pgi PrgEnv-gnu PrgEnv-intel PrgEnv-cray module load PrgEnv-gnu dynamic-link else module unload PE-intel PE-gnu module load PE-gnu fi # Use a particular anaconda installer version. # Index from 1 in zsh, index from 0 in bash. URL_BASE="https://repo.continuum.io/archive" AVAIL=( $(curl "${URL_BASE}/" 2>/dev/null | \ tr '\n' ' ' | \ sed -e 's/\(<tr>\)\s*/\n\1/g' \ -e 's/\s{2,}//g' \ -e 's/<a href="\([^"]*\)">[^>]*>/\1/g' | \ awk -F "</*td[^>]*>|</*tr>" '/<\/*t[rd]>/ {printf "%s:%s\n", $3, $9 }' | \ grep "Linux-x86_64" | \ grep -E "Anaconda[23]" | \ sort) ) lc() { echo "$(tr "[:upper:]" "[:lower:]" <<< "$1")" } select version in ${AVAIL[@]%%.sh:*}; do if [ -n "${version}" ]; then for i in "${AVAIL[@]}"; do if [[ "${i}" =~ ${version} ]]; then _ver="${version%-Linux*}" ANACONDA_INSTALLER_NAME="${version}.sh" ANACONDA_INSTALLER_HASH="${i##*:}" ANACONDA_PYTHON_VERSION="$(lc ${version%%-*})" PACKAGE_VERSION="${_ver#*-}" PACKAGE_NAME="python_${ANACONDA_PYTHON_VERSION}" break fi done else echo "ERROR: Section invalid." exit 1 fi break done HOST="${HOSTNAME%%-*}" PREFIX_BASE="/sw/.testing/$USER/python/${HOST}" PREFIX="${PREFIX_BASE}/opt/${PACKAGE_NAME}/${PACKAGE_VERSION}" echo "Installing $ANACONDA_PYTHON_VERSION ($PACKAGE_VERSION) to ${PREFIX}" # Install anaconda. mkdir -p "/tmp/$USER" INSTALLER="/tmp/$USER/$ANACONDA_INSTALLER_NAME" if [ ! -e "${INSTALLER}" ]; then wget "${URL_BASE}/${ANACONDA_INSTALLER_NAME}" -O "${INSTALLER}" _dl_hash="$(md5sum ${INSTALLER} 2>&1 | grep -m 1 -oe "^[^ ]*")" [ "$_dl_hash" != "$ANACONDA_INSTALLER_HASH" ] && return 1 chmod a+x "${INSTALLER}" fi if [ ! -d "${PREFIX}" ]; then ${INSTALLER} -b -p "${PREFIX}" else return 1 fi # Add anaconda deploy to the PATH if [ -z "$OPATH" ]; then # Anaconda to the PATH if it hasn't been already. OPATH="$PATH" export PATH="$PREFIX/bin:$PATH" fi # Update pip to latest version pip install -v --upgrade pip # If on a Cray, if [ -n "${CRAYPE_VERSION}" ]; then # Install mpi4py in anaconda root, built against cray-mpich CC=cc MPICC=cc pip install -v --no-binary :all: mpi4py else pip install -v --no-binary :all: mpi4py fi # Make the site install read-only. This is useful when installed in a location # that normally has group write access to prevent others from modifying the # installation. chmod -R go-w "${PREFIX}" ## By making the install prefix (and everything below it) read-only, conda will ## allow users to install custom envs at `$HOME/.conda/envs` by default. ## ## Users can over-ride with `$HOME/.condarc` containing: ## |envs_dirs: ## | - /ccs/proj/<projid>/... ## | - ~/.conda/envs # Reset the PATH. [ -n "${OPATH}" ] && export PATH="${OPATH}" && unset OPATH # Generate a modulefile. MODULEFILE="${PREFIX_BASE}/modulefiles/${PACKAGE_NAME}/${PACKAGE_VERSION}" mkdir -p $(dirname ${MODULEFILE}) cat << EOF > ${MODULEFILE} #%Module proc ModulesHelp { } { puts stderr "${PACKAGE_NAME} ${PACKAGE_VERSION}" puts stderr "" } # One line description module-whatis "${PACKAGE_NAME} ${PACKAGE_VERSION}" set HOST ${HOST} set PREFIX ${PREFIX} conflict python* prepend-path PATH \$PREFIX/bin unsetenv PYTHONSTARTUP setenv PYTHONUSERBASE \$env(HOME)/.local/\$HOST/${PACKAGE_NAME}/${PACKAGE_VERSION} setenv CONDALIBS \$PREFIX/lib EOF
