Libfabric Dependency Detection
Created by: philip-davis
I am having an issue building on Cori with libfabric support. I believe this comes down to something I am doing wrong in FindLIBFABRIC.cmake, but I can't figure out what. I am seeing this:
[ 40%] Linking CXX executable ../../../../../bin/sst_conn_tool
/usr/bin/ld: cannot find -lpkgcfg_lib_PC_LIBFABRIC_atomic-NOTFOUND
/usr/bin/ld: /global/homes/p/pdavis/cori/sw/libfabric/gnu/lib/libfabric.a(src_libfabric_la-fabric.o): in function `fi_ini':
fabric.c:(.text+0xaa2): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
collect2: error: ld returned 1 exit status
make[2]: *** [source/adios2/toolkit/sst/util/CMakeFiles/sst_conn_tool.dir/build.make:100: bin/sst_conn_tool] Error 1
make[1]: *** [CMakeFiles/Makefile2:2007: source/adios2/toolkit/sst/util/CMakeFiles/sst_conn_tool.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
Which looks to me like cmake is inducing a failure because it thought it needed to find the absolute path libatomic using pkg-config, but couldn't. I see this in CMakeCache.txt
:
//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_alpslli:FILEPATH=/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64/libalpslli.a
//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_alpsutil:FILEPATH=/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64/libalpsutil.a
//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_atomic:FILEPATH=pkgcfg_lib_PC_LIBFABRIC_atomic-NOTFOUND
//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_dl:FILEPATH=/usr/lib64/libdl.a
//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_fabric:FILEPATH=/global/homes/p/pdavis/cori/sw/libfabric/gnu/lib/libfabric.a
//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_pthread:FILEPATH=/usr/lib64/libpthread.a
//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_udreg:FILEPATH=/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64/libudreg.a
//Path to a library.
pkgcfg_lib_PC_LIBFABRIC_ugni:FILEPATH=/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64/libugni.a
libfabric.pc looks like:
prefix=/global/homes/p/pdavis/cori/sw/libfabric/gnu
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include
Name: libfabric
Description: OFI-WG libfabric
URL: https://github.com/ofiwg/libfabric.git
Version: 1.7.2
Requires:
Cflags: -I${includedir}
Libs: -L${libdir} -lfabric -L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64 -lxpmem -L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64 -ludreg -L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64 -lalpsutil -lalpslli -L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64 -lugni -latomic -lpthread -ldl
Libs.private: -L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64 -lxpmem -L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64 -ludreg -L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64 -lalpsutil -lalpslli -L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64 -lugni -latomic -lpthread -ldl
Requires.private:
The absolute path of libatomic is /opt/gcc/8.2.0/snos/lib64/libatomic.a
, so there's not enough information in libfabric.pc to find it, however, /opt/gcc/8.2.0/snos/lib64
is in the LIBRARY_PATH of the cc wrapper, and I can link against libatomic:
pdavis@cori02:~/tests> cc -v atomic.c -latomic
Using built-in specs.
COLLECT_GCC=/opt/gcc/8.2.0/bin/../snos/bin/gcc
COLLECT_LTO_WRAPPER=/opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/lto-wrapper
Target: x86_64-suse-linux
Configured with: ../cray-gcc-8.2.0-201811010913.df0113f60eb17/configure --prefix=/opt/gcc/8.2.0/snos --disable-nls --libdir=/opt/gcc/8.2.0/snos/lib --enable-languages=c,c++,fortran --with-gxx-include-dir=/opt/gcc/8.2.0/snos/include/g++ --with-slibdir=/opt/gcc/8.2.0/snos/lib --with-system-zlib --enable-shared --enable-__cxa_atexit --build=x86_64-suse-linux --with-ppl --with-cloog --disable-multilib
Thread model: posix
gcc version 8.2.0 20180726 (Cray Inc.) (GCC)
COLLECT_GCC_OPTIONS='-march=core-avx2' '-static' '-D' '__CRAYXC' '-D' '__CRAY_HASWELL' '-D' '__CRAYXT_COMPUTE_LINUX_TARGET' '-u' 'pthread_mutex_destroy' '-D' '__TARGET_LINUX__' '-v' '-I' '/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include' '-I' '/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include' '-I' '/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include' '-I' '/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include' '-I' '/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include' '-I' '/opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include' '-I' '/opt/cray/pe/pmi/5.0.14/include' '-I' '/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include' '-I' '/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include' '-I' '/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include' '-I' '/opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include' '-I' '/opt/cray-hss-devel/9.0.0/include' '-L/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/lib' '-L/opt/cray/dmapp/default/lib64' '-L/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/lib' '-L/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/lib64' '-L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64' '-L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64' '-L/opt/cray/pe/pmi/5.0.14/lib64' '-L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64' '-L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64' '-L/opt/cray/pe/atp/2.1.3/libApp' '-L/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/lib64'
/opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/cc1 -quiet -v -I /opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include -I /opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include -I /opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include -I /opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include -I /opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include -I /opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include -I /opt/cray/pe/pmi/5.0.14/include -I /opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include -I /opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include -I /opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include -I /opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include -I /opt/cray-hss-devel/9.0.0/include -D __CRAYXC -D __CRAY_HASWELL -D __CRAYXT_COMPUTE_LINUX_TARGET -D __TARGET_LINUX__ atomic.c -quiet -dumpbase atomic.c -march=core-avx2 -auxbase atomic -version -o /tmp/ccwl9GA7.s
GNU C17 (GCC) version 8.2.0 20180726 (Cray Inc.) (x86_64-suse-linux)
compiled by GNU C version 8.2.0 20180726 (Cray Inc.), GMP version 6.0.0, MPFR version 3.1.3, MPC version 1.0.3, isl version isl-0.15-GMP
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring nonexistent directory "/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/../../../../x86_64-suse-linux/include"
#include "..." search starts here:
#include <...> search starts here:
/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include
/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include
/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include
/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include
/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include
/opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include
/opt/cray/pe/pmi/5.0.14/include
/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include
/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include
/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include
/opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include
/opt/cray-hss-devel/9.0.0/include
/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/include
/usr/local/include
/opt/gcc/8.2.0/snos/include
/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/include-fixed
/usr/include
End of search list.
GNU C17 (GCC) version 8.2.0 20180726 (Cray Inc.) (x86_64-suse-linux)
compiled by GNU C version 8.2.0 20180726 (Cray Inc.), GMP version 6.0.0, MPFR version 3.1.3, MPC version 1.0.3, isl version isl-0.15-GMP
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: af69ce7fa3d5dbba875fe0994824dbac
COLLECT_GCC_OPTIONS='-march=core-avx2' '-static' '-D' '__CRAYXC' '-D' '__CRAY_HASWELL' '-D' '__CRAYXT_COMPUTE_LINUX_TARGET' '-u' 'pthread_mutex_destroy' '-D' '__TARGET_LINUX__' '-v' '-I' '/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include' '-I' '/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include' '-I' '/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include' '-I' '/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include' '-I' '/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include' '-I' '/opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include' '-I' '/opt/cray/pe/pmi/5.0.14/include' '-I' '/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include' '-I' '/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include' '-I' '/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include' '-I' '/opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include' '-I' '/opt/cray-hss-devel/9.0.0/include' '-L/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/lib' '-L/opt/cray/dmapp/default/lib64' '-L/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/lib' '-L/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/lib64' '-L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64' '-L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64' '-L/opt/cray/pe/pmi/5.0.14/lib64' '-L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64' '-L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64' '-L/opt/cray/pe/atp/2.1.3/libApp' '-L/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/lib64'
as -v -I /opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include -I /opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include -I /opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include -I /opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include -I /opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include -I /opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include -I /opt/cray/pe/pmi/5.0.14/include -I /opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include -I /opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include -I /opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include -I /opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include -I /opt/cray-hss-devel/9.0.0/include --64 -o /tmp/ccg3Klvj.o /tmp/ccwl9GA7.s
GNU assembler version 2.31.1 (x86_64-suse-linux) using BFD version (GNU Binutils; SUSE Linux Enterprise 15) 2.31.1.20180828-6.3
COMPILER_PATH=/opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/:/opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/:/opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/:/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/:/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/
LIBRARY_PATH=/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/:/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-march=core-avx2' '-static' '-D' '__CRAYXC' '-D' '__CRAY_HASWELL' '-D' '__CRAYXT_COMPUTE_LINUX_TARGET' '-u' 'pthread_mutex_destroy' '-D' '__TARGET_LINUX__' '-v' '-I' '/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include' '-I' '/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include' '-I' '/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include' '-I' '/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include' '-I' '/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include' '-I' '/opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include' '-I' '/opt/cray/pe/pmi/5.0.14/include' '-I' '/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include' '-I' '/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include' '-I' '/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include' '-I' '/opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include' '-I' '/opt/cray-hss-devel/9.0.0/include' '-L/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/lib' '-L/opt/cray/dmapp/default/lib64' '-L/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/lib' '-L/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/lib64' '-L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64' '-L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64' '-L/opt/cray/pe/pmi/5.0.14/lib64' '-L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64' '-L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64' '-L/opt/cray/pe/atp/2.1.3/libApp' '-L/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/lib64'
/opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/collect2 -plugin /opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/liblto_plugin.so -plugin-opt=/opt/gcc/8.2.0/snos/libexec/gcc/x86_64-suse-linux/8.2.0/lto-wrapper -plugin-opt=-fresolution=/tmp/ccCWTErv.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lc -m elf_x86_64 -static -u pthread_mutex_destroy /usr/lib/../lib64/crt1.o /usr/lib/../lib64/crti.o /opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/crtbeginT.o -L/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/lib -L/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/lib64 -L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64 -L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64 -L/opt/cray/pe/pmi/5.0.14/lib64 -L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64 -L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64 -L/opt/cray/pe/atp/2.1.3/libApp -L/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/lib64 -L/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0 -L/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/../../.. /tmp/ccg3Klvj.o -latomic -rpath=/opt/cray/pe/gcc-libs --no-as-needed -lAtpSigHandler -lAtpSigHCommData --undefined=_ATP_Data_Globals --undefined=__atpHandlerInstall -lpthread -Ttext-segment=0x20000000 -zmax-page-size=0x20000000 --whole-archive -lhugetlbfs --no-whole-archive -lsci_gnu_82_mpi -lsci_gnu_82 -lpthread -lhugetlbfs -lmpich_gnu_82 -lrt -lugni -lpthread -lpmi -lpthread -lalpslli -lpthread -lwlm_detect -lalpsutil -lpthread -lrca -lxpmem -lugni -lpthread -ludreg --as-needed -lgfortran -lquadmath --no-as-needed --as-needed -lmvec --no-as-needed --as-needed -lm --no-as-needed --as-needed -lpthread --no-as-needed --start-group -lgcc -lgcc_eh -lc --end-group /opt/gcc/8.2.0/snos/lib/gcc/x86_64-suse-linux/8.2.0/crtend.o /usr/lib/../lib64/crtn.o
COLLECT_GCC_OPTIONS='-march=core-avx2' '-static' '-D' '__CRAYXC' '-D' '__CRAY_HASWELL' '-D' '__CRAYXT_COMPUTE_LINUX_TARGET' '-u' 'pthread_mutex_destroy' '-D' '__TARGET_LINUX__' '-v' '-I' '/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/include' '-I' '/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/include' '-I' '/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/include' '-I' '/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/include' '-I' '/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/include' '-I' '/opt/cray/gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari/include' '-I' '/opt/cray/pe/pmi/5.0.14/include' '-I' '/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/include' '-I' '/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/include' '-I' '/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/include' '-I' '/opt/cray/krca/2.2.5-7.0.0.1_4.34__g92086a7.ari/include' '-I' '/opt/cray-hss-devel/9.0.0/include' '-L/opt/cray/pe/libsci/19.02.1/GNU/8.1/x86_64/lib' '-L/opt/cray/dmapp/default/lib64' '-L/opt/cray/pe/mpt/7.7.6/gni/mpich-gnu/8.2/lib' '-L/opt/cray/rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari/lib64' '-L/opt/cray/alps/6.6.50-7.0.0.1_3.35__g962f7108.ari/lib64' '-L/opt/cray/xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari/lib64' '-L/opt/cray/pe/pmi/5.0.14/lib64' '-L/opt/cray/ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari/lib64' '-L/opt/cray/udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari/lib64' '-L/opt/cray/pe/atp/2.1.3/libApp' '-L/opt/cray/wlm_detect/1.3.3-7.0.0.1_3.20__g7109084.ari/lib64'
pdavis@cori02:~/tests>
My cmake line is cmake .. -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn -DCMAKE_CXX_COMPILER=CC
, and my modules are:
1) modules/3.2.11.1 13) pmi/5.0.14
2) nsg/1.2.0 14) dmapp/7.1.1-7.0.0.1_5.20__g25e5077.ari
3) gcc/8.2.0 15) gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari
4) craype-haswell 16) xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari
5) craype-hugepages2M 17) job/2.2.4-7.0.0.1_3.30__g36b56f4.ari
6) craype-network-aries 18) dvs/2.11_2.2.137-7.0.0.1_10.1__g04d35b33
7) craype/2.5.18 19) alps/6.6.50-7.0.0.1_3.35__g962f7108.ari
8) cray-mpich/7.7.6 20) rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari
9) altd/2.0 21) atp/2.1.3
10) cray-libsci/19.02.1 22) PrgEnv-gnu/6.0.5
11) udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari 23) cmake/3.14.4
12) ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari
As a possibly unrelated aside, I had a terrible time getting libtool to statically link libatomic on Cori when building libfabric. I had to do make LDFLAGS=-all-static
to force static linking, despite configuring libfabric with --enable-static --disable-shared
. It seems like this is a known feature of libtool in certain circumstances.
I can work around this by editing CMakeCache.txt to insert the absolute path of libatomic, and I can successfully build. Something is clearly wrong somewhere, but I'm not sure what I've done wrong.