Libfabric Dependency Detection

Created by: philip-davis

I am having an issue building on Cori with libfabric support. I believe this comes down to something I am doing wrong in FindLIBFABRIC.cmake, but I can't figure out what. I am seeing this:

[ 40%] Linking CXX executable ../../../../../bin/sst_conn_tool
/usr/bin/ld: cannot find -lpkgcfg_lib_PC_LIBFABRIC_atomic-NOTFOUND
/usr/bin/ld: /global/homes/p/pdavis/cori/sw/libfabric/gnu/lib/libfabric.a(src_libfabric_la-fabric.o): in function `fi_ini':
fabric.c:(.text+0xaa2): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
collect2: error: ld returned 1 exit status
make[2]: *** [source/adios2/toolkit/sst/util/CMakeFiles/sst_conn_tool.dir/build.make:100: bin/sst_conn_tool] Error 1
make[1]: *** [CMakeFiles/Makefile2:2007: source/adios2/toolkit/sst/util/CMakeFiles/sst_conn_tool.dir/all] Error 2
make: *** [Makefile:141: all] Error 2

Which looks to me like cmake is inducing a failure because it thought it needed to find the absolute path libatomic using pkg-config, but couldn't. I see this in CMakeCache.txt:

//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_alpslli:FILEPATH=/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64/libalpslli.a

//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_alpsutil:FILEPATH=/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64/libalpsutil.a

//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_atomic:FILEPATH=pkgcfg_lib_PC_LIBFABRIC_atomic-NOTFOUND

//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_dl:FILEPATH=/usr/lib64/libdl.a

//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_fabric:FILEPATH=/global/homes/p/pdavis/cori/sw/libfabric/gnu/lib/libfabric.a

//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_pthread:FILEPATH=/usr/lib64/libpthread.a

//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_udreg:FILEPATH=/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64/libudreg.a

//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_ugni:FILEPATH=/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64/libugni.a

libfabric.pc looks like:

prefix=/global/homes/p/pdavis/cori/sw/libfabric/gnu
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: libfabric
Description: OFI-WG libfabric
URL: https://github.com/ofiwg/libfabric.git
Version: 1.7.2
Requires:
Cflags: -I${includedir}
Libs: -L${libdir} -lfabric   -L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64 -lxpmem -L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64 -ludreg -L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64 -lalpsutil -lalpslli -L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64 -lugni -latomic -lpthread -ldl
Libs.private:  -L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64 -lxpmem -L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64 -ludreg -L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64 -lalpsutil -lalpslli -L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64 -lugni -latomic -lpthread -ldl
Requires.private:

The absolute path of libatomic is /opt/gcc/8.2.0/snos/lib64/libatomic.a, so there's not enough information in libfabric.pc to find it, however, /opt/gcc/8.2.0/snos/lib64 is in the LIBRARY_PATH of the cc wrapper, and I can link against libatomic:

pdavis@cori02:~/tests> cc -v atomic.c -latomic
Using built-in specs.
COLLECT_GCC=/opt/gcc/8.2.0/bin/../snos/bin/gcc
COLLECT_LTO_WRAPPER=/opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/lto-wrapper
Target: x86_64-suse-linux
Configured with: ../cray-gcc-8.2.0-201811010913.df0113f60eb17/configure --prefix=/opt/gcc/8.2.0/snos --disable-nls --libdir=/opt/gcc/8.2.0/snos/lib --enable-languages=c,c++,fortran --with-gxx-include-dir=/opt/gcc/8.2.0/snos/include/g++ --with-slibdir=/opt/gcc/8.2.0/snos/lib --with-system-zlib --enable-shared --enable-__cxa_atexit --build=x86_64-suse-linux --with-ppl --with-cloog --disable-multilib
Thread model: posix
gcc version 8.2.0 20180726 (Cray Inc.) (GCC)
COLLECT_GCC_OPTIONS='-march=core-avx2' '-static' '-D' '__CRAYXC' '-D' '__CRAY_HASWELL' '-D' '__CRAYXT_COMPUTE_LINUX_TARGET' '-u' 'pthread_mutex_destroy' '-D' '__TARGET_LINUX__' '-v' '-I' '/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include' '-I' '/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include' '-I' '/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include' '-I' '/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include' '-I' '/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include' '-I' '/opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include' '-I' '/opt/cray/pe/pmi/5.0.14/include' '-I' '/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include' '-I' '/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include' '-I' '/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include' '-I' '/opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include' '-I' '/opt/cray-hss-devel/9.0.0/include' '-L/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/lib' '-L/opt/cray/dmapp/default/lib64' '-L/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/lib' '-L/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/lib64' '-L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64' '-L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64' '-L/opt/cray/pe/pmi/5.0.14/lib64' '-L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64' '-L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64' '-L/opt/cray/pe/atp/2.1.3/libApp' '-L/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/lib64'
 /opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/cc1 -quiet -v -I /opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include -I /opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include -I /opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include -I /opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include -I /opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include -I /opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include -I /opt/cray/pe/pmi/5.0.14/include -I /opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include -I /opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include -I /opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include -I /opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include -I /opt/cray-hss-devel/9.0.0/include -D __CRAYXC -D __CRAY_HASWELL -D __CRAYXT_COMPUTE_LINUX_TARGET -D __TARGET_LINUX__ atomic.c -quiet -dumpbase atomic.c -march=core-avx2 -auxbase atomic -version -o /tmp/ccwl9GA7.s
GNU C17 (GCC) version 8.2.0 20180726 (Cray Inc.) (x86_64-suse-linux)
	compiled by GNU C version 8.2.0 20180726 (Cray Inc.), GMP version 6.0.0, MPFR version 3.1.3, MPC version 1.0.3, isl version isl-0.15-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring nonexistent directory "/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/../../../../x86_64-suse-linux/include"
#include "..." search starts here:
#include <...> search starts here:
 /opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include
 /opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include
 /opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include
 /opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include
 /opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include
 /opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include
 /opt/cray/pe/pmi/5.0.14/include
 /opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include
 /opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include
 /opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include
 /opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include
 /opt/cray-hss-devel/9.0.0/include
 /opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/include
 /usr/local/include
 /opt/gcc/8.2.0/snos/include
 /opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/include-fixed
 /usr/include
End of search list.
GNU C17 (GCC) version 8.2.0 20180726 (Cray Inc.) (x86_64-suse-linux)
	compiled by GNU C version 8.2.0 20180726 (Cray Inc.), GMP version 6.0.0, MPFR version 3.1.3, MPC version 1.0.3, isl version isl-0.15-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: af69ce7fa3d5dbba875fe0994824dbac
COLLECT_GCC_OPTIONS='-march=core-avx2' '-static' '-D' '__CRAYXC' '-D' '__CRAY_HASWELL' '-D' '__CRAYXT_COMPUTE_LINUX_TARGET' '-u' 'pthread_mutex_destroy' '-D' '__TARGET_LINUX__' '-v' '-I' '/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include' '-I' '/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include' '-I' '/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include' '-I' '/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include' '-I' '/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include' '-I' '/opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include' '-I' '/opt/cray/pe/pmi/5.0.14/include' '-I' '/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include' '-I' '/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include' '-I' '/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include' '-I' '/opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include' '-I' '/opt/cray-hss-devel/9.0.0/include' '-L/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/lib' '-L/opt/cray/dmapp/default/lib64' '-L/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/lib' '-L/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/lib64' '-L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64' '-L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64' '-L/opt/cray/pe/pmi/5.0.14/lib64' '-L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64' '-L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64' '-L/opt/cray/pe/atp/2.1.3/libApp' '-L/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/lib64'
 as -v -I /opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include -I /opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include -I /opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include -I /opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include -I /opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include -I /opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include -I /opt/cray/pe/pmi/5.0.14/include -I /opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include -I /opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include -I /opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include -I /opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include -I /opt/cray-hss-devel/9.0.0/include --64 -o /tmp/ccg3Klvj.o /tmp/ccwl9GA7.s
GNU assembler version 2.31.1 (x86_64-suse-linux) using BFD version (GNU Binutils; SUSE Linux Enterprise 15) 2.31.1.20180828-6.3
COMPILER_PATH=/opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/:/opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/:/opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/:/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/:/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/
LIBRARY_PATH=/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/:/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-march=core-avx2' '-static' '-D' '__CRAYXC' '-D' '__CRAY_HASWELL' '-D' '__CRAYXT_COMPUTE_LINUX_TARGET' '-u' 'pthread_mutex_destroy' '-D' '__TARGET_LINUX__' '-v' '-I' '/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include' '-I' '/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include' '-I' '/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include' '-I' '/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include' '-I' '/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include' '-I' '/opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include' '-I' '/opt/cray/pe/pmi/5.0.14/include' '-I' '/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include' '-I' '/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include' '-I' '/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include' '-I' '/opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include' '-I' '/opt/cray-hss-devel/9.0.0/include' '-L/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/lib' '-L/opt/cray/dmapp/default/lib64' '-L/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/lib' '-L/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/lib64' '-L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64' '-L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64' '-L/opt/cray/pe/pmi/5.0.14/lib64' '-L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64' '-L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64' '-L/opt/cray/pe/atp/2.1.3/libApp' '-L/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/lib64'
 /opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/collect2 -plugin /opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/liblto_plugin.so -plugin-opt=/opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/lto-wrapper -plugin-opt=-fresolution=/tmp/ccCWTErv.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lc -m elf_x86_64 -static -u pthread_mutex_destroy /usr/lib/../lib64/crt1.o /usr/lib/../lib64/crti.o /opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/crtbeginT.o -L/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/lib -L/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/lib64 -L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64 -L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64 -L/opt/cray/pe/pmi/5.0.14/lib64 -L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64 -L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64 -L/opt/cray/pe/atp/2.1.3/libApp -L/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/lib64 -L/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0 -L/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/../../.. /tmp/ccg3Klvj.o -latomic -rpath=/opt/cray/pe/gcc-libs --no-as-needed -lAtpSigHandler -lAtpSigHCommData --undefined=_ATP_Data_Globals --undefined=__atpHandlerInstall -lpthread -Ttext-segment=0x20000000 -zmax-page-size=0x20000000 --whole-archive -lhugetlbfs --no-whole-archive -lsci_gnu_82_mpi -lsci_gnu_82 -lpthread -lhugetlbfs -lmpich_gnu_82 -lrt -lugni -lpthread -lpmi -lpthread -lalpslli -lpthread -lwlm_detect -lalpsutil -lpthread -lrca -lxpmem -lugni -lpthread -ludreg --as-needed -lgfortran -lquadmath --no-as-needed --as-needed -lmvec --no-as-needed --as-needed -lm --no-as-needed --as-needed -lpthread --no-as-needed --start-group -lgcc -lgcc_eh -lc --end-group /opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/crtend.o /usr/lib/../lib64/crtn.o
COLLECT_GCC_OPTIONS='-march=core-avx2' '-static' '-D' '__CRAYXC' '-D' '__CRAY_HASWELL' '-D' '__CRAYXT_COMPUTE_LINUX_TARGET' '-u' 'pthread_mutex_destroy' '-D' '__TARGET_LINUX__' '-v' '-I' '/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include' '-I' '/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include' '-I' '/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include' '-I' '/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include' '-I' '/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include' '-I' '/opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include' '-I' '/opt/cray/pe/pmi/5.0.14/include' '-I' '/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include' '-I' '/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include' '-I' '/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include' '-I' '/opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include' '-I' '/opt/cray-hss-devel/9.0.0/include' '-L/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/lib' '-L/opt/cray/dmapp/default/lib64' '-L/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/lib' '-L/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/lib64' '-L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64' '-L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64' '-L/opt/cray/pe/pmi/5.0.14/lib64' '-L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64' '-L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64' '-L/opt/cray/pe/atp/2.1.3/libApp' '-L/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/lib64'
pdavis@cori02:~/tests>

My cmake line is cmake .. -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn -DCMAKE_CXX_COMPILER=CC, and my modules are:

  1) modules/3.2.11.1                                 13) pmi/5.0.14
  2) nsg/1.2.0                                        14) dmapp/7.1.1-7.0.0.1_5.20__g25e5077.ari
  3) gcc/8.2.0                                        15) gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari
  4) craype-haswell                                   16) xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari
  5) craype-hugepages2M                               17) job/2.2.4-7.0.0.1_3.30__g36b56f4.ari
  6) craype-network-aries                             18) dvs/2.11_2.2.137-7.0.0.1_10.1__g04d35b33
  7) craype/2.5.18                                    19) alps/6.6.50-7.0.0.1_3.35__g962f7108.ari
  8) cray-mpich/7.7.6                                 20) rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari
  9) altd/2.0                                         21) atp/2.1.3
 10) cray-libsci/19.02.1                              22) PrgEnv-gnu/6.0.5
 11) udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari           23) cmake/3.14.4
 12) ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari

As a possibly unrelated aside, I had a terrible time getting libtool to statically link libatomic on Cori when building libfabric. I had to do make LDFLAGS=-all-static to force static linking, despite configuring libfabric with --enable-static --disable-shared. It seems like this is a known feature of libtool in certain circumstances.

I can work around this by editing CMakeCache.txt to insert the absolute path of libatomic, and I can successfully build. Something is clearly wrong somewhere, but I'm not sure what I've done wrong.