Segmentation fault with helloBPWriter.py on cori
Created by: rkube
I'm trying to run the helloBPWriter.py on cori and get a segmentation fault. This is a self-compiled version of adios2 using icc.
Command used: $ mpirun -n 1 python -m mpi4py helloBPWriter.py
libhugetlbfs [cori10:3626]: WARNING: Hugepage size 2097152 unavailablelibhugetlbfs [cori10:3626]: WARNING: New heap segment map at 0x10000000000 failed: Cannot allocate memory [cori10:03626] *** Process received signal *** [cori10:03626] Signal: Segmentation fault (11) [cori10:03626] Signal code: Address not mapped (1) [cori10:03626] Failing at address: 0xac2473a8 [cori10:03626] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aaaaace5360] [cori10:03626] [ 1] /global/common/software/m3169/openmpi/3.1.4/intel/lib/libmpi.so.40(MPI_Comm_dup+0x19e)[0x2aaaabfa528e] [cori10:03626] [ 2] /global/homes/r/rkube/local/lib64/libadios2.so.2(_ZN6adios26helper4Comm9DuplicateEi+0x13)[0x2aaaee350d23] [cori10:03626] [ 3] /global/homes/r/rkube/local/lib64/libadios2.so.2(ZN6adios24core5ADIOSC1ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEibS7+0xbe)[0x2aaaee18ea1e] [cori10:03626] [ 4] /global/homes/r/rkube/local/lib64/python3.7/site-packages/adios2.cpython-37m-x86_64-linux-gnu.so(+0x20065f0d)[0x2aaacdf0ff0d] [cori10:03626] [ 5] /global/homes/r/rkube/local/lib64/python3.7/site-packages/adios2.cpython-37m-x86_64-linux-gnu.so(+0x20059fad)[0x2aaacdf03fad] [cori10:03626] [ 6] /global/homes/r/rkube/local/lib64/python3.7/site-packages/adios2.cpython-37m-x86_64-linux-gnu.so(+0x20066c7d)[0x2aaacdf10c7d] [cori10:03626] [ 7] python(_PyMethodDef_RawFastCallDict+0x24d)[0x55555568963d] [cori10:03626] [ 8] python(_PyCFunction_FastCallDict+0x21)[0x5555556897c1] [cori10:03626] [ 9] python(_PyObject_Call_Prepend+0x63)[0x555555687e53] [cori10:03626] [10] python(PyObject_Call+0x6e)[0x55555567adbe] [cori10:03626] [11] python(+0x9e214)[0x5555555f2214] [cori10:03626] [12] python(_PyObject_FastCallKeywords+0x128)[0x5555556c0588] [cori10:03626] [13] python(_PyEval_EvalFrameDefault+0x52f8)[0x5555557246e8] [cori10:03626] [14] python(_PyEval_EvalCodeWithName+0x2f9)[0x555555668539] [cori10:03626] [15] python(PyEval_EvalCodeEx+0x44)[0x555555669424] [cori10:03626] [16] python(PyEval_EvalCode+0x1c)[0x55555566944c] [cori10:03626] [17] python(+0x1dae81)[0x55555572ee81] [cori10:03626] [18] python(_PyMethodDef_RawFastCallKeywords+0xe9)[0x5555556b85d9] [cori10:03626] [19] python(_PyCFunction_FastCallKeywords+0x21)[0x5555556b8861] [cori10:03626] [20] python(_PyEval_EvalFrameDefault+0x47a4)[0x555555723b94] [cori10:03626] [21] python(_PyEval_EvalCodeWithName+0x2f9)[0x555555668539] [cori10:03626] [22] python(_PyFunction_FastCallKeywords+0x325)[0x5555556b7ef5] [cori10:03626] [23] python(_PyEval_EvalFrameDefault+0x416)[0x55555571f806] [cori10:03626] [24] python(_PyEval_EvalCodeWithName+0x2f9)[0x555555668539] [cori10:03626] [25] python(_PyFunction_FastCallKeywords+0x387)[0x5555556b7f57] [cori10:03626] [26] python(_PyEval_EvalFrameDefault+0x14dc)[0x5555557208cc] [cori10:03626] [27] python(_PyEval_EvalCodeWithName+0x2f9)[0x555555668539] [cori10:03626] [28] python(_PyFunction_FastCallKeywords+0x387)[0x5555556b7f57] [cori10:03626] [29] python(_PyEval_EvalFrameDefault+0x14dc)[0x5555557208cc] [cori10:03626] *** End of error message ***
(delta) rkube@cori10:~/repos/ADIOS2/examples/hello/bpWriter> python --version Python 3.7.4
(delta) rkube@cori10:~/repos/ADIOS2/examples/hello/bpWriter> module list Currently Loaded Modulefiles:
- modules/3.2.11.1 13) job/2.2.4-7.0.0.1_3.30__g36b56f4.ari
- nsg/1.2.0 14) dvs/2.11_2.2.137-7.0.0.1_10.1__g04d35b33
- intel/19.0.3.199 15) alps/6.6.50-7.0.0.1_3.35__g962f7108.ari
- craype-network-aries 16) rca/2.2.20-7.0.0.1_4.34__g8e3fb5b.ari
- craype/2.5.18 17) atp/2.1.3
- cray-libsci/19.02.1 18) PrgEnv-intel/6.0.5
- udreg/2.3.2-7.0.0.1_4.27__g8175d3d.ari 19) craype-haswell
- ugni/6.0.14.0-7.0.0.1_7.30__ge78e5b0.ari 20) cray-mpich/7.7.6
- pmi/5.0.14 21) craype-hugepages2M
- dmapp/7.1.1-7.0.0.1_5.20__g25e5077.ari 22) cmake/3.14.4
- gni-headers/5.0.12.0-7.0.0.1_7.39__g3b1768f.ari 23) python/3.7-anaconda-2019.07
- xpmem/2.2.17-7.0.0.1_3.24__g7acee3a.ari 24) openmpi/3.1.4