ADIOS2 issueshttps://code.ornl.gov/pnb/ADIOS2/-/issues2020-01-21T19:49:28Zhttps://code.ornl.gov/pnb/ADIOS2/-/issues/1912io.AvailableAttributes()2020-01-21T19:49:28ZPodhorszki, Norbertio.AvailableAttributes()*Created by: guj*
There is a test in TestNativeHDF5WriteRead.cpp
EXPECT_EQ(1, io.AvailableAttributes(var4Name).size())
That I am currently removing b/c var4 only shows at ts=1
and this test is called after. browsing all time steps.
...*Created by: guj*
There is a test in TestNativeHDF5WriteRead.cpp
EXPECT_EQ(1, io.AvailableAttributes(var4Name).size())
That I am currently removing b/c var4 only shows at ts=1
and this test is called after. browsing all time steps.
Thus it thinks var4 is not a valid var.
So would like to. confirm that io.AvailableAttribtues() applies to only the current engine ts?
If so this test above makes is not right. It should be 0 instead of 1.
https://code.ornl.gov/pnb/ADIOS2/-/issues/1906Reading a file with many subfiles fails2020-03-02T12:01:51ZPodhorszki, NorbertReading a file with many subfiles fails*Created by: pnorbert*
I thought this was addressed before but I don't find an issue for it.
I have an XGC output with 1024 subfiles on a KNL machine. Trying to bpls dump a variable that is written from all processes fails when openin...*Created by: pnorbert*
I thought this was addressed before but I don't find an issue for it.
I have an XGC output with 1024 subfiles on a KNL machine. Trying to bpls dump a variable that is written from all processes fails when opening too many subfiles.
```
$ bpls -l xgc.restart.00006.bp/ -d inum_arr -n 16
int32_t inum_arr {1024} = 2293 / 4061532
bpls caught an exception
ERROR: couldn't open file xgc.restart.00006.bp/data.1019, check permissions or path existence, in call to POSIX open
$ ls -l xgc.restart.00006.bp/data.1019
-rw-r----- 1 n55002 gn55 321625842 Dec 18 11:50 xgc.restart.00006.bp/data.1019
```
`$ bp4dbg.py xgc.restart.00006.bp/data.1019` dumps a perfectly fine data file.
https://code.ornl.gov/pnb/ADIOS2/-/issues/1891Large memory consumption in BufferSTL when using PerformPuts and Flush instea...2019-12-06T15:19:22ZPodhorszki, NorbertLarge memory consumption in BufferSTL when using PerformPuts and Flush instead of Begin/EndStep (BP4)*Created by: franzpoeschel*
During performance evaluations of the ADIOS2 backend in the [openPMD API](https://github.com/openPMD/openPMD-api), I noticed an unusually large heap memory consumption in non-streaming workflows (in this case...*Created by: franzpoeschel*
During performance evaluations of the ADIOS2 backend in the [openPMD API](https://github.com/openPMD/openPMD-api), I noticed an unusually large heap memory consumption in non-streaming workflows (in this case the BP4 engine) and traced it back to memory not being freed from the marshalling buffer (`BufferSTL`).
Since openPMD's iterations cannot be easily modeled using the ADIOS2 step concept, this backend only uses steps for streaming engines. For disk-based engines, we use `Engine::PerformPuts` and `Engine::Flush` instead. The documentation for the latter method says:
> Manually flush to underlying transport to guarantee data is moved
This suggests that data should not be present in ADIOS after this call. (?) The figure below shows the memory trace from a small example, writing 30 openPMD iterations from [PIConGPU](https://github.com/ComputationalRadiationPhysics/picongpu) to disk with several flushes per iteration. This memory buildup is not visible when using ADIOS steps.
![Bildschirmfoto vom 2019-11-27 12-01-26](https://user-images.githubusercontent.com/14241876/69718079-af35b100-110d-11ea-85fb-405520e37d41.png)
Did I understand the functionality of `Engine::Flush` correctly? In that case, calling it should free the buffer. If not, is there a suggested alternative to avoid using ADIOS steps without building up heap memory usage in the described way?https://code.ornl.gov/pnb/ADIOS2/-/issues/1886Make exceptions more concise.2019-11-22T16:50:12ZPodhorszki, NorbertMake exceptions more concise.*Created by: rkube*
Hi,
working with adios I came along a confusing exception. Basically, I tried to access a variable without having a file open.
```python
import adios2
adios = adios2.ADIOS()
ioread = adios.DeclareIO("reader")
...*Created by: rkube*
Hi,
working with adios I came along a confusing exception. Basically, I tried to access a variable without having a file open.
```python
import adios2
adios = adios2.ADIOS()
ioread = adios.DeclareIO("reader")
res = ioread.InquireVariable("ECEI_L2408")
In [9]: type(res)
Out[9]: adios2.Variable
In [11]: res.Shape()
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-11-a8250f87005c> in <module>
----> 1 res.Shape()
ValueError: ERROR: found null pointer in call to Variable::Shape
In [15]: res.Steps()
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-15-a39dc5e97e8d> in <module>
----> 1 res.Steps()
ValueError: ERROR: found null pointer in call to Variable::Steps
```
Of course there is no file open to inquire from. But I think the error message could be made
so that it gives a hint as to what is wrong.
Another thing is accessing BP4 files. I tried this code to open the BP4 file "KSTAR.bp":
```python
import adios2
adios = adios2.ADIOS()
ioread = adios.DeclareIO("reader")
bpStream = ioread.Open("KSTAR.bp", adios2.Mode.Read)
Traceback (most recent call last):
File "verify_kstarbp.py", line 6, in <module>
bpStream = ioread.Open("KSTAR.bp", adios2.Mode.Read)
RuntimeError: ERROR: couldn't read from file KSTAR.bp, in call to POSIX IO read
: iostream error
```
The error is that I try to open a BP4 file, probably with the wrong engine. It would be great if the error message would tell me what is wrong instead of giving just an iostream error.
https://code.ornl.gov/pnb/ADIOS2/-/issues/1885CMake: Enabled Fortran Bindings Infect Downstream C++2020-02-05T14:38:53ZPodhorszki, NorbertCMake: Enabled Fortran Bindings Infect Downstream C++*Created by: ax3l*
This issue was originally raised in https://github.com/conda-forge/openpmd-api-feedstock/pull/39#issuecomment-554280078 .
Why do I need as a downstream, C++11-only project a Fortran compiler to be able to parse the...*Created by: ax3l*
This issue was originally raised in https://github.com/conda-forge/openpmd-api-feedstock/pull/39#issuecomment-554280078 .
Why do I need as a downstream, C++11-only project a Fortran compiler to be able to parse the `ADIOS2Config.cmake` file when reading it in as a dependency? (Note: ADIOS2 was build with Fortran support but I do not require those bindings downstream.)
This seams unnecessary to me.
```
-- The Fortran compiler identification is unknown
CMake Error at /home/conda/feedstock_root/build_artifacts/openpmd-api_1573808717385/<...>/lib/cmake/adios2/adios2-config-common.cmake:16 (enable_language):
No CMAKE_Fortran_COMPILER could be found.
Tell CMake where to find the compiler by setting either the environment
variable "FC" or the CMake cache entry CMAKE_Fortran_COMPILER to the full
path to the compiler, or to the compiler name if it is in the PATH.
Call Stack (most recent call first):
/home/conda/feedstock_root/build_artifacts/openpmd-api_1573808717385/<...>/lib/cmake/adios2/adios2-config.cmake:29 (include)
CMakeLists.txt:232 (find_package)
-- Configuring incomplete, errors occurred!
```
The problem with this is that it commutes to downstream dependencies, even transitively, due to `find_dependencies(ADIOS2)` in downstream `Config.cmake` packages...https://code.ornl.gov/pnb/ADIOS2/-/issues/1862Python SST BeginStep segmentation fault / hang with timeout other than -1 or 02019-11-09T14:28:56ZPodhorszki, NorbertPython SST BeginStep segmentation fault / hang with timeout other than -1 or 0*Created by: suchyta1*
I've been running into some errors when I try to read using SST from Python and the timeout parameter is set to something other than -1 or 0. To reproduce the behavior, I wrote a simple writer/reader pair, include...*Created by: suchyta1*
I've been running into some errors when I try to read using SST from Python and the timeout parameter is set to something other than -1 or 0. To reproduce the behavior, I wrote a simple writer/reader pair, included below.
With the timeout of 0.5 and ADIOS 2.4.0, ADIOS 2.5.0, or newer, I get a segmentation fault or a reader hang when I run on my Mac or Ubuntu virtual machine. Based on the print statements, the reader is hanging or segmentation faulting inside the BeginStep() call. (Sometimes, the job will run fine, but not reliably.) As far as I can tell, if I change the timeout to -1 or 0.0, it runs fine.
I'm not sure if there's an issue from any other of the bindings. I haven't noticed it if there is, but I haven't exhaustively tested.
## writer.py
```
#!/usr/bin/env python
from mpi4py import MPI
import adios2
import numpy as np
import time
import sys
if __name__ == "__main__":
tmp = np.ones(10)
comm = MPI.COMM_WORLD
adios = adios2.ADIOS(comm, adios2.DebugON)
io = adios.DeclareIO("test")
io.SetEngine('SST')
io.SetParameter('RendezvousReaderCount', '1')
io.SetParameter("QueueLimit", "1")
io.SetParameter("QueueFullPolicy", "Discard")
io.DefineVariable("t", tmp, [10], [10], [10])
time.sleep(1)
engine = io.Open("test.bp", adios2.Mode.Write, comm)
for i in range(10):
engine.BeginStep()
v = io.InquireVariable("t")
engine.Put(v, tmp)
engine.EndStep()
print("Wrote Step:", i); sys.stdout.flush()
time.sleep(1)
engine.Close()
```
## reader.py
```
#!/usr/bin/env python
from mpi4py import MPI
import adios2
import numpy as np
import time
import sys
if __name__ == "__main__":
comm = MPI.COMM_WORLD
adios = adios2.ADIOS(comm, adios2.DebugON)
io = adios.DeclareIO("test")
io.SetEngine('SST')
"""
io.SetParameter('RendezvousReaderCount', '0')
io.SetParameter("QueueLimit", "1")
io.SetParameter("QueueFullPolicy", "Discard")
"""
engine = io.Open("test.bp", adios2.Mode.Read, comm)
while True:
print("Read begin step"); sys.stdout.flush()
StepStatus = engine.BeginStep(adios2.StepMode.Read, 0.5)
print("Done read begin step"); sys.stdout.flush()
if StepStatus == adios2.StepStatus.EndOfStream:
print("Done looping"); sys.stdout.flush()
break
elif StepStatus == adios2.StepStatus.OK:
print("Read found step:", engine.CurrentStep(), "\n"); sys.stdout.flush()
engine.EndStep(); sys.stdout.flush()
elif StepStatus == adios2.StepStatus.NotReady:
print("Not ready\n"); sys.stdout.flush()
continue
elif StepStatus == adios2.StepStatus.OtherError:
print("Other"); sys.stdout.flush()
break
engine.Close()
```
## job.sh
```
mpirun -n 1 ./writer.py &
mpirun -n 1 ./reader.py &
wait
```
## Example of the error output, from the virtual machine
```
Read begin step
Wrote Step: 0
Done read begin step
Read found step: 0
Read begin step
Done read begin step
Not ready
Read begin step
Wrote Step: 1
Done read begin step
Read found step: 1
Read begin step
Done read begin step
Not ready
Read begin step
Wrote Step: 2
[adiosVM:03069] *** Process received signal ***
[adiosVM:03069] Signal: Segmentation fault (11)
[adiosVM:03069] Signal code: Address not mapped (1)
[adiosVM:03069] Failing at address: 0x38
[adiosVM:03069] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fdb2ac50390]
[adiosVM:03069] [ 1] /opt/adios2/lib/libadios2_cmselect.so(+0x1db0)[0x7fdb0cac3db0]
[adiosVM:03069] [ 2] /opt/adios2/lib/libadios2_cmselect.so(libcmselect_LTX_remove_periodic+0x19)[0x7fdb0cac3e59]
[adiosVM:03069] [ 3] /opt/adios2/lib/libadios2_evpath.so(INT_CMremove_task+0x20)[0x7fdb19009400]
[adiosVM:03069] [ 4] /opt/adios2/lib/libadios2_evpath.so(CMremove_task+0x28)[0x7fdb1901d248]
[adiosVM:03069] [ 5] /opt/adios2/lib/libadios2_sst.so.2(+0xe3da)[0x7fdb1a1f13da]
[adiosVM:03069] [ 6] /opt/adios2/lib/libadios2_sst.so.2(SstAdvanceStep+0x3a6)[0x7fdb1a1f3406]
[adiosVM:03069] [ 7] /opt/adios2/lib/libadios2.so.2(_ZN6adios24core6engine9SstReader9BeginStepENS_8StepModeEf+0x12f)[0x7fdb1ac3d56f]
[adiosVM:03069] [ 8] /opt/adios2/lib/python3.5/site-packages/adios2.cpython-35m-x86_64-linux-gnu.so(+0x29903)[0x7fdb1cc2b903]
[adiosVM:03069] [ 9] /opt/adios2/lib/python3.5/site-packages/adios2.cpython-35m-x86_64-linux-gnu.so(+0x3fe9b)[0x7fdb1cc41e9b]
[adiosVM:03069] [10] /opt/adios2/lib/python3.5/site-packages/adios2.cpython-35m-x86_64-linux-gnu.so(+0x16ca3)[0x7fdb1cc18ca3]
[adiosVM:03069] [11] python(PyCFunction_Call+0x77)[0x4ea117]
[adiosVM:03069] [12] python(PyEval_EvalFrameEx+0x5a36)[0x529816]
[adiosVM:03069] [13] python[0x52d2b9]
[adiosVM:03069] [14] python(PyEval_EvalCode+0x1f)[0x52dfbf]
[adiosVM:03069] [15] python[0x5fc652]
[adiosVM:03069] [16] python(PyRun_FileExFlags+0x9a)[0x5feafa]
[adiosVM:03069] [17] python(PyRun_SimpleFileExFlags+0x1bc)[0x5fecec]
[adiosVM:03069] [18] python(Py_Main+0x456)[0x63ec96]
[adiosVM:03069] [19] python(main+0xe1)[0x4d02e1]
[adiosVM:03069] [20] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fdb2a895830]
[adiosVM:03069] [21] python(_start+0x29)[0x5d4b99]
[adiosVM:03069] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 3069 on node adiosVM exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
Wrote Step: 3
Wrote Step: 4
Wrote Step: 5
Wrote Step: 6
Wrote Step: 7
Wrote Step: 8
Wrote Step: 9
```
## Hanging looks similar
```
Read begin step
Wrote Step: 0
Done read begin step
Read found step: 0
Read begin step
Done read begin step
Not ready
Read begin step
Wrote Step: 1
Done read begin step
Read found step: 1
Read begin step
Done read begin step
Not ready
Read begin step
Wrote Step: 2
Done read begin step
Read found step: 2
Read begin step
Done read begin step
Not ready
Read begin step
Wrote Step: 3
Done read begin step
Read found step: 3
Read begin step
Done read begin step
Not ready
Read begin step
Done read begin step
Not ready
Read begin step
Wrote Step: 4
Done read begin step
Read found step: 4
Read begin step
Done read begin step
Not ready
Read begin step
Wrote Step: 5
Done read begin step
Read found step: 5
Read begin step
Done read begin step
Not ready
Read begin step
Wrote Step: 6
Done read begin step
Read found step: 6
Read begin step
Done read begin step
Not ready
Read begin step
Wrote Step: 7
Done read begin step
Read found step: 7
Read begin step
Done read begin step
Not ready
Read begin step
Wrote Step: 8
Wrote Step: 9
```
@eisenhauer Eisenhauer, GregEisenhauer, Greghttps://code.ornl.gov/pnb/ADIOS2/-/issues/1848MGARD Decompression with s parameter is incorrect2019-10-31T15:19:52ZPodhorszki, NorbertMGARD Decompression with s parameter is incorrect*Created by: NAThompson*
**Describe the bug**
When MGARD is compressed with a certain value of `s` for the Besov s-norm, it must be decompressed with the exact same value of `s`, or else the decompressed values are completely wrong.
...*Created by: NAThompson*
**Describe the bug**
When MGARD is compressed with a certain value of `s` for the Besov s-norm, it must be decompressed with the exact same value of `s`, or else the decompressed values are completely wrong.
**To Reproduce**
The ADIOS2 wrapper always uses the `s`-parameter to compress; see [here](https://github.com/ornladios/ADIOS2/blob/master/source/adios2/operator/compress/CompressMGARD.cpp#L103
).
However, it never uses the `s`-parameter to decompress, see [here](https://github.com/ornladios/ADIOS2/blob/master/source/adios2/operator/compress/CompressMGARD.cpp#L146).
I regard this as very low-priority, since MGARD is considering storing `s` in the compressed stream; see [here](https://github.com/CODARcode/MGARD/issues/37).
https://code.ornl.gov/pnb/ADIOS2/-/issues/1837BP3 vs BP4 behavior changes.2020-03-03T17:55:55ZPodhorszki, NorbertBP3 vs BP4 behavior changes.*Created by: NAThompson*
**Describe the bug**
If we uncomment the `bpIO.SetEngine("bp3")` line, the following program aborts with an exception:
```cpp
adios2::ADIOS adios(adios2::DebugON);
adios2::IO bpIO = adios.Declare...*Created by: NAThompson*
**Describe the bug**
If we uncomment the `bpIO.SetEngine("bp3")` line, the following program aborts with an exception:
```cpp
adios2::ADIOS adios(adios2::DebugON);
adios2::IO bpIO = adios.DeclareIO("BPFile_N2N");
//bpIO.SetEngine("bp3");
adios2::Variable<Real> random_data = bpIO.DefineVariable<Real>("random_data", {}, {}, {v.size()}, adios2::ConstantDims);
std::string filename = "random_vector_" + std::to_string(state.range(0)) + ".bp";
adios2::Engine bpWriter = bpIO.Open(filename, adios2::Mode::Write);
bpWriter.Put<Real>(random_data, v.data());
bpWriter.Close();
adios2::Engine bpReader = bpIO.Open(filename, adios2::Mode::Read);
```
**Desktop (please complete the following information):**
- OS/Platform: Ubuntu 18.0.4
- Build: gcc 7.4.0, cmake 15
**Additional context**
This is found in [ADIOS2Benchmarks](https://github.com/NAThompson/ADIOS2Benchmarks)
https://code.ornl.gov/pnb/ADIOS2/-/issues/1820Python API for associated attributes2019-10-15T20:46:00ZPodhorszki, NorbertPython API for associated attributes*Created by: pnorbert*
Is there a Python equivalent of the C++ AvailableAttributes(string varname)?
Needed by @isosc
This does not work:
```
$ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linu...*Created by: pnorbert*
Is there a Python equivalent of the C++ AvailableAttributes(string varname)?
Needed by @isosc
This does not work:
```
$ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import adios2
>>> import numpy
>>> fr = adios2.open("heat2d/fortran/a.bp", "r")
>>> vars_info = fr.available_variables()
>>> vars_info
{'T': {'AvailableStepsCount': '4', 'Type': 'double', 'SingleValue': 'false', 'Max': '200', 'Shape': '768, 1024', 'Min': '0'}, 'dT': {'AvailableStepsCount': '4', 'Type': 'double', 'SingleValue': 'false', 'Max': '74.5893', 'Shape': '768, 1024', 'Min': '-77.3378'}}
>>> attrs_info = fr.available_attributes()
>>> attrs_info
{'dT/description': {'Value': '"Temperature difference between two steps calculated in analysis"', 'Type': 'string', 'Elements': '1'}, 'T/unit': {'Value': '"C"', 'Type': 'string', 'Elements': '1'}, 'T/description': {'Value': '"Temperature from simulation"', 'Type': 'string', 'Elements': '1'}}
>>> desc = fr.available_attributes('dT')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: available_attributes(): incompatible function arguments. The following argument types are supported:
1. (self: adios2.File) -> Dict[str, Dict[str, str]]
Invoked with: <adios2.file named '../heat2d/fortran/a.bp' and mode 'r'>, 'dT'
```https://code.ornl.gov/pnb/ADIOS2/-/issues/1815Mutable attributes, single values, association of variables to variables2019-10-07T22:25:09ZPodhorszki, NorbertMutable attributes, single values, association of variables to variables*Created by: pnorbert*
We have a gap between the convenience of attributes and the single value variables, hence @ax3l is asking for the ability to change an attribute's value over time. Instead, they could use single values for such in...*Created by: pnorbert*
We have a gap between the convenience of attributes and the single value variables, hence @ax3l is asking for the ability to change an attribute's value over time. Instead, they could use single values for such information but only if variables can be associated to variables (like attributes to variables currently).
https://code.ornl.gov/pnb/ADIOS2/-/issues/1816Auto-generated info for directory hierarchy in ADIOS output2020-03-02T12:12:35ZPodhorszki, NorbertAuto-generated info for directory hierarchy in ADIOS output*Created by: pnorbert*
Challenge: reconstruct the hierarchy at read time fast. OpenPMD spends 30 seconds to do that currently.
Idea: ADIOS could save the hierarchy as a string variable as a JSON/YAML document at write time, that eith...*Created by: pnorbert*
Challenge: reconstruct the hierarchy at read time fast. OpenPMD spends 30 seconds to do that currently.
Idea: ADIOS could save the hierarchy as a string variable as a JSON/YAML document at write time, that either the application can use to generate the hierarchy or ADIOS could use it to generate a tree instead of the plain map of variables and attributes.
This could speed up the variable/variable and attribute/variable association functions at read time as well.
@ax3l @psychocoderHPC @franzpoeschel @BeyondEspressohttps://code.ornl.gov/pnb/ADIOS2/-/issues/1814BP-X: manage multiple pre-sized buffers to avoid realloc of huge single buffer2019-10-07T20:32:52ZPodhorszki, NorbertBP-X: manage multiple pre-sized buffers to avoid realloc of huge single buffer*Created by: pnorbert*
PIConGPU may use all memory so that ADIOS buffer just fits. If a realloc would happen, it could fail due to lack of memory. Still, a some memory would be available to create another buffer to store more data.
S...*Created by: pnorbert*
PIConGPU may use all memory so that ADIOS buffer just fits. If a realloc would happen, it could fail due to lack of memory. Still, a some memory would be available to create another buffer to store more data.
Suggestion: BP-X should manage multiple smaller buffers (we are talking about 500MB, 1GB buffers here) and let (sophisticated) user create these buffers on the fly.
https://code.ornl.gov/pnb/ADIOS2/-/issues/1766Make SST fail better2019-10-15T23:45:23ZEisenhauer, GregMake SST fail betterSST needs to detect underlying network failure events and fail more cleanly / deliver better diagnostics.
The most important is likely when we're running out of file descriptors, but we should also notice bind failures and other things ...SST needs to detect underlying network failure events and fail more cleanly / deliver better diagnostics.
The most important is likely when we're running out of file descriptors, but we should also notice bind failures and other things happening deep inside the communication system.Eisenhauer, GregEisenhauer, Greghttps://code.ornl.gov/pnb/ADIOS2/-/issues/1738Error build ADIOS2 2.4.1, unknown type name 'INT4'2019-09-28T12:45:09ZPodhorszki, NorbertError build ADIOS2 2.4.1, unknown type name 'INT4'*Created by: ChrocheMisawa*
Hi guys, I'm new to ADIOS2,
When I tried to install ADIOS2, I got error making the package, I wonder why it is happening(2nd picture) and how do I solve it?
the system is WSL2
Thanks very much!
Chen
...*Created by: ChrocheMisawa*
Hi guys, I'm new to ADIOS2,
When I tried to install ADIOS2, I got error making the package, I wonder why it is happening(2nd picture) and how do I solve it?
the system is WSL2
Thanks very much!
Chen
![) {62B$N{4 AY_$Q(4 3$H](https://user-images.githubusercontent.com/47645670/64903737-48ffcd00-d68c-11e9-8e0a-144a490a99a6.png)
![image](https://user-images.githubusercontent.com/47645670/64903745-574de900-d68c-11e9-9a03-fdcdfd0ac481.png)
2.5.0 Production ReleaseEisenhauer, GregEisenhauer, Greghttps://code.ornl.gov/pnb/ADIOS2/-/issues/1731Quadratic increasing costs for Engine::Put() during one step when using the S...2019-10-10T21:52:27ZPodhorszki, NorbertQuadratic increasing costs for Engine::Put() during one step when using the SST Engine*Created by: franzpoeschel*
While using ADIOS2 in PIConGPU for streaming IO, I noticed that during one step, each call to `Engine::Put()` took at least as long as the previous call. While investigating this, I found out the following:
...*Created by: franzpoeschel*
While using ADIOS2 in PIConGPU for streaming IO, I noticed that during one step, each call to `Engine::Put()` took at least as long as the previous call. While investigating this, I found out the following:
- The SST engine does not distinguish between sync and deferred mode (see [→](https://github.com/ornladios/ADIOS2/blob/f1c6435d87e501de95b7e3f0d1b75e2b00b0f85b/source/adios2/engine/sst/SstWriter.cpp#L293) and [→](https://github.com/ornladios/ADIOS2/issues/1315)), so each written dataset is instantly marshalled and written to an internal buffer.
The total incoming data is not known up front, so reallocation becomes necessary.
- Reallocation in the BP3 serializer (which is used by SST) intentionally overrides the STL GNU default power of 2 reallocation and instead enforces a linear behavior (see [→](https://github.com/ornladios/ADIOS2/blob/f1c6435d87e501de95b7e3f0d1b75e2b00b0f85b/source/adios2/toolkit/format/buffer/heap/BufferSTL.cpp#L28))
This means that data written during one step in the SST engine will be reallocated as many times as subsequent chunks are written. This is probably fine for BP3 where deferred workflows are encouraged that avoid this behavior, but this will not work for SST.
By uncommenting [this line](https://github.com/ornladios/ADIOS2/blob/f1c6435d87e501de95b7e3f0d1b75e2b00b0f85b/source/adios2/toolkit/format/buffer/heap/BufferSTL.cpp#L30), I was able to avoid this issue for now. https://code.ornl.gov/pnb/ADIOS2/-/issues/1728SST crashes with LibFabric2019-09-28T18:21:38ZPodhorszki, NorbertSST crashes with LibFabric*Created by: khuck*
ADIOS2 was configured on a 36-core Linux workstation with the following cmake output:
```
Currently Loaded Modules:
1) gcc/8.1 2) mpi/openmpi-4.0.1_gcc-8.1 3) cmake/3.15.1 4) python/3.6.8
++ which m...*Created by: khuck*
ADIOS2 was configured on a 36-core Linux workstation with the following cmake output:
```
Currently Loaded Modules:
1) gcc/8.1 2) mpi/openmpi-4.0.1_gcc-8.1 3) cmake/3.15.1 4) python/3.6.8
++ which mpicc
++ which mpic++
++ which mpif90
+ cmake -DCMAKE_C_COMPILER=/packages/openmpi/4.0.1-gcc8.1/bin/mpicc -DCMAKE_CXX_COMPILER=/packages/openmpi/4.0.1-gcc8.1/bin/mpic++ -DCMAKE_Fortran_COMPILER=/packages/openmpi/4.0.1-gcc8.1/bin/mpif90 -DADIOS2_USE_Python=ON -DCMAKE_INSTALL_PREFIX=/home/users/khuck/src/ADIOS2/install_mpi -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
-- The C compiler identification is GNU 8.1.0
-- The CXX compiler identification is GNU 8.1.0
-- Check for working C compiler: /packages/openmpi/4.0.1-gcc8.1/bin/mpicc
-- Check for working C compiler: /packages/openmpi/4.0.1-gcc8.1/bin/mpicc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /packages/openmpi/4.0.1-gcc8.1/bin/mpic++
-- Check for working CXX compiler: /packages/openmpi/4.0.1-gcc8.1/bin/mpic++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find Blosc (missing: BLOSC_LIBRARY BLOSC_INCLUDE_DIR)
-- Could NOT find BZip2 (missing: BZIP2_LIBRARIES BZIP2_INCLUDE_DIR)
-- Could NOT find ZFP (missing: ZFP_LIBRARY ZFP_INCLUDE_DIR)
-- Could NOT find SZ (missing: SZ_LIBRARY ZLIB_LIBRARY ZSTD_LIBRARY SZ_INCLUDE_DIR)
-- Could NOT find MGARD (missing: MGARD_LIBRARY ZLIB_LIBRARY MGARD_INCLUDE_DIR)
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7")
-- Could NOT find PNG: Found unsuitable version "1.5.13", but required is at least "1.6.0" (found /usr/lib64/libpng.so)
-- The Fortran compiler identification is GNU 8.1.0
-- Check for working Fortran compiler: /packages/openmpi/4.0.1-gcc8.1/bin/mpif90
-- Check for working Fortran compiler: /packages/openmpi/4.0.1-gcc8.1/bin/mpif90 -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /packages/openmpi/4.0.1-gcc8.1/bin/mpif90 supports Fortran 90
-- Checking whether /packages/openmpi/4.0.1-gcc8.1/bin/mpif90 supports Fortran 90 -- yes
-- Found MPI_C: /packages/openmpi/4.0.1-gcc8.1/bin/mpicc (found version "3.1")
-- Found MPI_CXX: /packages/openmpi/4.0.1-gcc8.1/bin/mpic++ (found version "3.1")
-- Found MPI_Fortran: /packages/openmpi/4.0.1-gcc8.1/bin/mpif90 (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: C Fortran CXX
-- Found ZeroMQ: /usr/lib64/libzmq.so (found suitable version "4.1.4", minimum required is "4.1")
-- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS C) (found version "")
-- Found PythonInterp: /packages/python/3.6.8/bin/python3 (found version "3.6.8")
-- Found PythonLibs: /packages/python/3.6.8/lib/libpython3.6m.so (found version "3.6.8")
-- Found PythonModule_numpy: /packages/python/3.6.8/lib/python3.6/site-packages/numpy
-- Found PythonModule_mpi4py: /home/users/khuck/.local/lib/python3.6/site-packages/mpi4py
-- Found PythonFull: /packages/python/3.6.8/bin/python3 found components: Interp Libs numpy mpi4py
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.27.1")
-- Checking for module 'libfabric'
-- Found libfabric, version 1.6.1
-- Found LIBFABRIC: /usr/lib64/libfabric.so (Required is at least version "1.6")
-- Checking for module 'cray-drc'
-- No package 'cray-drc' found
-- Could NOT find CrayDRC (missing: CrayDRC_LIBRARIES)
-- Looking for shmget
-- Looking for shmget - found
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- ADIOS2 ThirdParty: Configuring KWSys
-- Checking whether header cstdio is available
-- Checking whether header cstdio is available - yes
-- Checking for Large File Support
-- Checking for Large File Support - yes
-- Checking whether C++ compiler has 'long long'
-- Checking whether C++ compiler has 'long long' - yes
-- Checking whether C++ compiler has '__int64'
-- Checking whether C++ compiler has '__int64' - no
-- Checking whether wstring is available
-- Checking whether wstring is available - yes
-- Checking whether C compiler has ptrdiff_t in stddef.h
-- Checking whether C compiler has ptrdiff_t in stddef.h - yes
-- Checking whether C compiler has ssize_t in unistd.h
-- Checking whether C compiler has ssize_t in unistd.h - yes
-- Checking whether CXX compiler has setenv
-- Checking whether CXX compiler has setenv - yes
-- Checking whether CXX compiler has unsetenv
-- Checking whether CXX compiler has unsetenv - yes
-- Checking whether CXX compiler has environ in stdlib.h
-- Checking whether CXX compiler has environ in stdlib.h - no
-- Checking whether CXX compiler has utimes
-- Checking whether CXX compiler has utimes - yes
-- Checking whether CXX compiler has utimensat
-- Checking whether CXX compiler has utimensat - yes
-- Checking whether CXX compiler struct stat has st_mtim member
-- Checking whether CXX compiler struct stat has st_mtim member - yes
-- Checking whether CXX compiler struct stat has st_mtimespec member
-- Checking whether CXX compiler struct stat has st_mtimespec member - no
-- Checking whether <ext/stdio_filebuf.h> is available
-- Checking whether <ext/stdio_filebuf.h> is available - yes
-- ADIOS2 ThirdParty: Configuring GTest
-- ADIOS2 ThirdParty: Configuring pybind11
-- Found PythonLibs: /packages/python/3.6.8/lib/libpython3.6m.so
-- pybind11 v2.2.4
-- ADIOS2 ThirdParty: Configuring pugixml
-- ADIOS2 ThirdParty: Configuring nlohmann_json
-- ADIOS2 ThirdParty: Configuring atl
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of double
-- Check size of double - done
-- Check size of float
-- Check size of float - done
-- Check size of int
-- Check size of int - done
-- Check size of short
-- Check size of short - done
-- Looking for include file malloc.h
-- Looking for include file malloc.h - found
-- Looking for include file unistd.h
-- Looking for include file unistd.h - found
-- Looking for include file stdlib.h
-- Looking for include file stdlib.h - found
-- Looking for include file string.h
-- Looking for include file string.h - found
-- Looking for include file sys/time.h
-- Looking for include file sys/time.h - found
-- Looking for include file windows.h
-- Looking for include file windows.h - not found
-- Looking for fork
-- Looking for fork - found
-- Found atl: /home/users/khuck/src/ADIOS2/build/thirdparty/atl/atl/atl-config.cmake (found version "2.2.1")
-- ADIOS2 ThirdParty: Configuring dill
-- Check size of void*
-- Check size of void* - done
-- Check size of long
-- Check size of long - done
-- Check if the system is big endian
-- Searching 16 bit integer
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Checking for module 'libffi'
-- Found libffi, version 3.0.13
-- Found LibFFI: -lffi
-- Enabling emulation
-- Looking for include file stdarg.h
-- Looking for include file stdarg.h - found
-- Looking for include file memory.h
-- Looking for include file memory.h - found
-- Found dill: /home/users/khuck/src/ADIOS2/build/thirdparty/dill/dill/dill-config.cmake (found version "2.4.0")
-- ADIOS2 ThirdParty: Configuring ffs
-- Check size of off_t
-- Check size of off_t - done
-- Check size of long double
-- Check size of long double - done
-- Check size of long long
-- Check size of long long - done
-- Check size of size_t
-- Check size of size_t - done
-- Looking for socket
-- Looking for socket - found
-- Found BISON: /usr/bin/bison (found version "3.0.4")
-- Found FLEX: /usr/bin/flex (found version "2.5.37")
-- Found dill: /home/users/khuck/src/ADIOS2/build/thirdparty/dill/dill/dill-config.cmake (found suitable version "2.4.0", minimum required is "2.3.1")
-- Found atl: /home/users/khuck/src/ADIOS2/build/thirdparty/atl/atl/atl-config.cmake (found suitable version "2.2.1", minimum required is "2.2.1")
-- Looking for netdb.h
-- Looking for netdb.h - found
-- Looking for sockLib.h
-- Looking for sockLib.h - not found
-- Looking for sys/select.h
-- Looking for sys/select.h - found
-- Looking for sys/socket.h
-- Looking for sys/socket.h - found
-- Looking for sys/times.h
-- Looking for sys/times.h - found
-- Looking for sys/uio.h
-- Looking for sys/uio.h - found
-- Looking for sys/un.h
-- Looking for sys/un.h - found
-- Looking for winsock.h
-- Looking for winsock.h - not found
-- Looking for strtof
-- Looking for strtof - found
-- Looking for strtod
-- Looking for strtod - found
-- Looking for strtold
-- Looking for strtold - found
-- Looking for getdomainname
-- Looking for getdomainname - found
-- Check size of struct iovec
-- Check size of struct iovec - done
-- Performing Test HAS_IOV_BASE_IOVEC
-- Performing Test HAS_IOV_BASE_IOVEC - Success
-- Found atl: /home/users/khuck/src/ADIOS2/build/thirdparty/atl/atl/atl-config.cmake (found version "2.2.1")
-- Found ffs: /home/users/khuck/src/ADIOS2/build/thirdparty/ffs/ffs/ffs-config.cmake (found version "1.6.0")
-- ADIOS2 ThirdParty: Configuring enet
-- Looking for getaddrinfo
-- Looking for getaddrinfo - found
-- Looking for getnameinfo
-- Looking for getnameinfo - found
-- Looking for gethostbyaddr_r
-- Looking for gethostbyaddr_r - found
-- Looking for gethostbyname_r
-- Looking for gethostbyname_r - found
-- Looking for poll
-- Looking for poll - found
-- Looking for fcntl
-- Looking for fcntl - found
-- Looking for inet_pton
-- Looking for inet_pton - found
-- Looking for inet_ntop
-- Looking for inet_ntop - found
-- Performing Test HAS_MSGHDR_FLAGS
-- Performing Test HAS_MSGHDR_FLAGS - Success
-- Performing Test HAS_SOCKLEN_T
-- Performing Test HAS_SOCKLEN_T - Success
-- Found enet: /home/users/khuck/src/ADIOS2/build/thirdparty/enet/enet/enet-config.cmake (found version "1.3.14")
-- ADIOS2 ThirdParty: Configuring EVPath
-- Performing Test HAVE_MATH
-- Performing Test HAVE_MATH - Failed
-- Performing Test HAVE_LIBM_MATH
-- Performing Test HAVE_LIBM_MATH - Success
-- Found atl: /home/users/khuck/src/ADIOS2/build/thirdparty/atl/atl/atl-config.cmake (found suitable version "2.2.1", minimum required is "2.2.1")
-- Found atl: /home/users/khuck/src/ADIOS2/build/thirdparty/atl/atl/atl-config.cmake (found version "2.2.1")
-- Found ffs: /home/users/khuck/src/ADIOS2/build/thirdparty/ffs/ffs/ffs-config.cmake (found suitable version "1.6.0", minimum required is "1.5.1")
-- Could NOT find nvml (missing: NVML_INCLUDE_DIR)
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- Found enet: /home/users/khuck/src/ADIOS2/build/thirdparty/enet/enet/enet-config.cmake (found suitable version "1.3.14", minimum required is "1.3.13")
-- - Udt4 library was not found. This is not a fatal error, just that the Udt4 transport will not be built.
-- Found LIBFABRIC: /usr/lib64/libfabric.so
-- Looking for ibv_create_qp
-- Looking for ibv_create_qp - not found
-- Looking for ibv_create_qp in ibverbs
-- Looking for ibv_create_qp in ibverbs - found
-- Found IBVERBS: ibverbs
-- Could NOT find nnti (missing: NNTI_INCLUDE_DIR NNTI_trios_nnti_LIBRARY NNTI_trios_support_LIBRARY)
-- Looking for hostlib.h
-- Looking for hostlib.h - not found
-- Looking for sys/sockio.h
-- Looking for sys/sockio.h - not found
-- Performing Test HAVE_FDS_BITS
-- Performing Test HAVE_FDS_BITS - Failed
-- Looking for writev
-- Looking for writev - found
-- Looking for uname
-- Looking for uname - found
-- Looking for getloadavg
-- Looking for getloadavg - found
-- Looking for gettimeofday
-- Looking for gettimeofday - found
-- Looking for getifaddrs
-- Looking for getifaddrs - found
-- Found atl: /home/users/khuck/src/ADIOS2/build/thirdparty/atl/atl/atl-config.cmake (found suitable version "2.2.1", minimum required is "2.2.1")
-- Found atl: /home/users/khuck/src/ADIOS2/build/thirdparty/atl/atl/atl-config.cmake (found version "2.2.1")
-- Found ffs: /home/users/khuck/src/ADIOS2/build/thirdparty/ffs/ffs/ffs-config.cmake (found suitable version "1.6.0", minimum required is "1.6.0")
-- Found EVPath: /home/users/khuck/src/ADIOS2/build/thirdparty/EVPath/EVPath/EVPathConfigCommon.cmake (found version "4.4.0")
-- Found atl: /home/users/khuck/src/ADIOS2/build/thirdparty/atl/atl/atl-config.cmake (found suitable version "2.2.1", minimum required is "2.2.1")
-- Found atl: /home/users/khuck/src/ADIOS2/build/thirdparty/atl/atl/atl-config.cmake (found version "2.2.1")
-- Looking for rdma/fi_ext_gni.h
-- Looking for rdma/fi_ext_gni.h - not found
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Verifying Fortran/CXX Compiler Compatibility
-- Verifying Fortran/CXX Compiler Compatibility - Success
-- Found MPI: TRUE (found version "3.1") found components: C
ADIOS2 build configuration:
ADIOS Version: 2.4.0
C++ Compiler : GNU 8.1.0
/packages/openmpi/4.0.1-gcc8.1/bin/mpic++
Fortran Compiler : GNU 8.1.0
/packages/openmpi/4.0.1-gcc8.1/bin/mpif90
Installation prefix: /home/users/khuck/src/ADIOS2/install_mpi
bin: bin
lib: lib64
include: include
cmake: lib64/cmake/adios2
python: lib64/python3.6/site-packages
Features:
Library Type: shared
Build Type: RelWithDebInfo
Testing: ON
Build Options:
Blosc : OFF
BZip2 : OFF
ZFP : OFF
SZ : OFF
MGARD : OFF
PNG : OFF
MPI : ON
DataMan : ON
SSC : ON
SST : ON
ZeroMQ : ON
HDF5 : OFF
Python : ON
Fortran : ON
SysVShMem: ON
Profiling: ON
Endian_Reverse: OFF
RDMA Transport for Staging: Available
-- Configuring done
-- Generating done
-- Build files have been written to: /home/users/khuck/src/ADIOS2/build
```
When trying to run the heatTransfer example on this workstation with SST, the following crash happened (similar/same crash happens without the --mca arguments):
```
mpirun --mca btl_openib_allow_ib true --mca btl_openib_warn_default_gid_prefix 0 -n 16 ./heatSimulation sim.bp 4 4 64 64 100 10 : -n 4 ./heatAnalysis sim.bp analysis.bp 2 2
Process decomposition : 4 x 4
Array size per process : 64 x 64
Number of output steps : 100
Iterations per step : 10
Using SST engine for input
Using BP4 engine for output
[delphi:53491] *** Process received signal ***
[delphi:53491] Signal: Segmentation fault (11)
[delphi:53491] Signal code: Address not mapped (1)
[delphi:53491] Failing at address: 0x10
[delphi:53491] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x7f2fc755c5d0]
[delphi:53491] [ 1] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x17080)[0x7f2fc6d00080]
[delphi:53491] [ 2] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x171ec)[0x7f2fc6d001ec]
[delphi:53491] [ 3] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(SstWriterOpen+0xc6)[0x7f2fc6cf6ba6]
[delphi:53491] [ 4] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core6engine9SstWriterC2ERNS0_2IOERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x167)[0x7f2fc8a14827]
[delphi:53491] [ 5] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core2IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x5f3)[0x7f2fc871b003]
[delphi:53491] [ 6] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios22IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0xe3)[0x7f2fc8a7c3d3]
[delphi:53491] [ 7] ./heatSimulation[0x40fb36]
[delphi:53491] [ 8] ./heatSimulation[0x40b4ef]
[delphi:53491] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f2fc71a23d5]
[delphi:53491] [10] ./heatSimulation[0x40b71f]
[delphi:53491] *** End of error message ***
[delphi:53489] *** Process received signal ***
[delphi:53489] Signal: Segmentation fault (11)
[delphi:53489] Signal code: Address not mapped (1)
[delphi:53489] Failing at address: 0x10
[delphi:53489] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x7f115d8045d0]
[delphi:53489] [ 1] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x17080)[0x7f115cfa8080]
[delphi:53489] [ 2] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x171ec)[0x7f115cfa81ec]
[delphi:53489] [ 3] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(SstWriterOpen+0xc6)[0x7f115cf9eba6]
[delphi:53489] [ 4] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core6engine9SstWriterC2ERNS0_2IOERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x167)[0x7f115ecbc827]
[delphi:53489] [ 5] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core2IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x5f3)[0x7f115e9c3003]
[delphi:53489] [ 6] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios22IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0xe3)[0x7f115ed243d3]
[delphi:53489] [ 7] ./heatSimulation[0x40fb36]
[delphi:53489] [ 8] ./heatSimulation[0x40b4ef]
[delphi:53489] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f115d44a3d5]
[delphi:53489] [10] ./heatSimulation[0x40b71f]
[delphi:53489] *** End of error message ***
[delphi:53482] *** Process received signal ***
[delphi:53482] Signal: Segmentation fault (11)
[delphi:53482] Signal code: Address not mapped (1)
[delphi:53482] Failing at address: 0x10
[delphi:53482] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x7f63ceddf5d0]
[delphi:53482] [ 1] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x17080)[0x7f63ce583080]
[delphi:53482] [ 2] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x171ec)[0x7f63ce5831ec]
[delphi:53482] [ 3] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(SstWriterOpen+0xc6)[0x7f63ce579ba6]
[delphi:53482] [ 4] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core6engine9SstWriterC2ERNS0_2IOERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x167)[0x7f63d0297827]
[delphi:53482] [ 5] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core2IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x5f3)[0x7f63cff9e003]
[delphi:53482] [ 6] [delphi:53486] *** Process received signal ***
[delphi:53486] Signal: Segmentation fault (11)
[delphi:53486] Signal code: Address not mapped (1)
[delphi:53486] Failing at address: 0x10
/storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios22IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0xe3)[0x7f63d02ff3d3]
[delphi:53482] [ 7] ./heatSimulation[0x40fb36]
[delphi:53482] [ 8] ./heatSimulation[0x40b4ef]
[delphi:53482] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f63cea253d5]
[delphi:53482] [10] ./heatSimulation[0x40b71f]
[delphi:53482] *** End of error message ***
[delphi:53484] *** Process received signal ***
[delphi:53484] Signal: Segmentation fault (11)
[delphi:53484] Signal code: Address not mapped (1)
[delphi:53484] Failing at address: 0x10
[delphi:53486] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x7fe2f7e8b5d0]
[delphi:53486] [ 1] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x17080)[0x7fe2f762f080]
[delphi:53486] [ 2] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x171ec)[0x7fe2f762f1ec]
[delphi:53486] [ 3] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(SstWriterOpen+0xc6)[0x7fe2f7625ba6]
[delphi:53486] [ 4] [delphi:53487] *** Process received signal ***
[delphi:53487] Signal: Segmentation fault (11)
[delphi:53487] Signal code: Address not mapped (1)
[delphi:53487] Failing at address: 0x10
[delphi:53487] [ 0] [delphi:53496] *** Process received signal ***
[delphi:53496] Signal: Segmentation fault (11)
[delphi:53496] Signal code: Address not mapped (1)
[delphi:53496] Failing at address: 0x10
[delphi:53496] [ 0] [delphi:53498] *** Process received signal ***
[delphi:53498] Signal: Segmentation fault (11)
[delphi:53498] Signal code: Address not mapped (1)
[delphi:53498] Failing at address: 0x10
[delphi:53484] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x7fa4407215d0]
[delphi:53484] [ 1] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x17080)[0x7fa43fec5080]
[delphi:53484] [ 2] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x171ec)[0x7fa43fec51ec]
[delphi:53484] [ 3] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(SstWriterOpen+0xc6)[0x7fa43febbba6]
[delphi:53484] [ 4] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core6engine9SstWriterC2ERNS0_2IOERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x167)[0x7fa441bd9827]
[delphi:53484] [ 5] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core2IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x5f3)[0x7fa4418e0003]
[delphi:53484] [ 6] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios22IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0xe3)[0x7fa441c413d3]
[delphi:53484] [ 7] ./heatSimulation[0x40fb36]
[delphi:53484] [ 8] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core6engine9SstWriterC2ERNS0_2IOERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x167)[0x7fe2f9343827]
[delphi:53486] [ 5] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core2IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x5f3)[0x7fe2f904a003]
[delphi:53486] [ 6] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios22IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0xe3)[0x7fe2f93ab3d3]
[delphi:53486] [ 7] ./heatSimulation[0x40fb36]
[delphi:53486] [ 8] ./heatSimulation[0x40b4ef]
[delphi:53486] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fe2f7ad13d5]
[delphi:53486] [10] ./heatSimulation[0x40b71f]
[delphi:53486] *** End of error message ***
/lib64/libpthread.so.0(+0xf5d0)[0x7efcb0ff15d0]
[delphi:53487] [ 1] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x17080)[0x7efcb0795080]
[delphi:53487] [ 2] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x171ec)[0x7efcb07951ec]
[delphi:53487] [ 3] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(SstWriterOpen+0xc6)[0x7efcb078bba6]
[delphi:53487] [ 4] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core6engine9SstWriterC2ERNS0_2IOERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x167)[0x7efcb24a9827]
[delphi:53487] [ 5] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core2IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x5f3)[0x7efcb21b0003]
[delphi:53487] [ 6] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios22IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0xe3)[0x7efcb25113d3]
[delphi:53487] [ 7] ./heatSimulation[0x40fb36]
[delphi:53487] [ 8] ./heatSimulation[0x40b4ef]
[delphi:53487] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7efcb0c373d5]
[delphi:53487] [10] ./heatSimulation[0x40b71f]
[delphi:53487] *** End of error message ***
/lib64/libpthread.so.0(+0xf5d0)[0x7fa144f785d0]
[delphi:53496] [ 1] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x17080)[0x7fa14471c080]
[delphi:53496] [ 2] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x171ec)[0x7fa14471c1ec]
[delphi:53496] [ 3] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(SstWriterOpen+0xc6)[0x7fa144712ba6]
[delphi:53496] [ 4] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core6engine9SstWriterC2ERNS0_2IOERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x167)[0x7fa146430827]
[delphi:53496] [ 5] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core2IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x5f3)[0x7fa146137003]
[delphi:53496] [ 6] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios22IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0xe3)[0x7fa1464983d3]
[delphi:53496] [ 7] ./heatSimulation[0x40fb36]
[delphi:53496] [ 8] ./heatSimulation[0x40b4ef]
[delphi:53496] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fa144bbe3d5]
[delphi:53496] [10] [delphi:53498] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x7f124180d5d0]
[delphi:53498] [ 1] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x17080)[0x7f1240fb1080]
[delphi:53498] [ 2] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x172ff)[0x7f1240fb12ff]
[delphi:53498] [ 3] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(SstReaderOpen+0xaa)[0x7f1240fa49fa]
[delphi:53498] [ 4] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core6engine9SstReaderC1ERNS0_2IOERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x161)[0x7f1242cb2be1]
[delphi:53498] [ 5] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core2IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x71e)[0x7f12429cc12e]
[delphi:53498] [ 6] ./heatSimulation[0x40b71f]
[delphi:53496] *** End of error message ***
./heatSimulation[0x40b4ef]
[delphi:53484] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fa4403673d5]
[delphi:53484] [10] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios22IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0xe3)[0x7f1242d2d3d3]
[delphi:53498] [ 7] ./heatAnalysis[0x40a827]
[delphi:53498] [ 8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f12414533d5]
[delphi:53498] [ 9] ./heatAnalysis[0x40ba5f]
[delphi:53498] *** End of error message ***
./heatSimulation[0x40b71f]
[delphi:53484] *** End of error message ***
[delphi:53499] *** Process received signal ***
[delphi:53499] Signal: Segmentation fault (11)
[delphi:53499] Signal code: Address not mapped (1)
[delphi:53499] Failing at address: 0x10
[delphi:53499] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x7f1b289265d0]
[delphi:53499] [ 1] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x17080)[0x7f1b280ca080]
[delphi:53499] [ 2] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x172ff)[0x7f1b280ca2ff]
[delphi:53499] [ 3] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(SstReaderOpen+0xaa)[0x7f1b280bd9fa]
[delphi:53499] [ 4] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core6engine9SstReaderC1ERNS0_2IOERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x161)[0x7f1b29dcbbe1]
[delphi:53499] [ 5] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core2IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x71e)[0x7f1b29ae512e]
[delphi:53499] [ 6] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios22IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0xe3)[0x7f1b29e463d3]
[delphi:53499] [ 7] ./heatAnalysis[0x40a827]
[delphi:53499] [ 8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1b2856c3d5]
[delphi:53499] [ 9] ./heatAnalysis[0x40ba5f]
[delphi:53499] *** End of error message ***
[delphi:53485] *** Process received signal ***
[delphi:53485] Signal: Segmentation fault (11)
[delphi:53485] Signal code: Address not mapped (1)
[delphi:53485] Failing at address: 0x10
[delphi:53485] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x7fc7d7b995d0]
[delphi:53485] [ 1] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x17080)[0x7fc7d733d080]
[delphi:53485] [ 2] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(+0x171ec)[0x7fc7d733d1ec]
[delphi:53485] [ 3] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/../lib64/libadios2_sst.so.2(SstWriterOpen+0xc6)[0x7fc7d7333ba6]
[delphi:53485] [ 4] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core6engine9SstWriterC2ERNS0_2IOERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x167)[0x7fc7d9051827]
[delphi:53485] [ 5] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios24core2IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0x5f3)[0x7fc7d8d58003]
[delphi:53485] [ 6] /storage/users/khuck/src/ADIOS2/install_mpi/lib64/libadios2.so.2(_ZN6adios22IO4OpenERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4ModeEP19ompi_communicator_t+0xe3)[0x7fc7d90b93d3]
[delphi:53485] [ 7] ./heatSimulation[0x40fb36]
[delphi:53485] [ 8] ./heatSimulation[0x40b4ef]
[delphi:53485] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fc7d77df3d5]
[delphi:53485] [10] ./heatSimulation[0x40b71f]
[delphi:53485] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 7 with PID 0 on node delphi exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
```
And the backtrace from one of the ranks:
```
[khuck@delphi cpp]$ gdb ./heatSimulation core.56907
ImportError: No module named site
[khuck@delphi cpp]$ module unload python
[khuck@delphi cpp]$ gdb ./heatSimulation core.56907
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /storage/users/khuck/src/adiosvm/Tutorial/heat2d/cpp/heatSimulation...done.
[New LWP 56907]
[New LWP 56929]
[New LWP 57462]
[New LWP 56941]
[New LWP 57851]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
btCore was generated by `./heatSimulation sim.bp 4 4 64 64 100 10'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fef071c3080 in fi_ep_bind (flags=0, bfid=0x157b120, ep=0x0) at /usr/include/rdma/fi_endpoint.h:168
168 return ep->fid.ops->bind(&ep->fid, bfid, flags);
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.172-2.el7.x86_64 elfutils-libs-0.172-2.el7.x86_64 infinipath-psm-3.3-26_g604758e_open.2.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_6.x86_64 libattr-2.4.46-13.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-13.el7.x86_64 libfabric-1.6.1-2.el7.x86_64 libffi-3.0.13-18.el7.x86_64 libibverbs-17.2-3.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 libpciaccess-0.14-1.el7.x86_64 libpsm2-10.3.58-1.el7.x86_64 librdmacm-17.2-3.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libsodium13-1.0.5-1.el7.x86_64 libuuid-2.23.2-59.el7_6.1.x86_64 numactl-libs-2.0.9-7.el7.x86_64 openpgm-5.2.122-2.el7.x86_64 pcre-8.32-17.el7.x86_64 systemd-libs-219-62.el7.x86_64 zeromq-4.1.4-5.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt
#0 0x00007fef071c3080 in fi_ep_bind (flags=0, bfid=0x157b120, ep=0x0) at /usr/include/rdma/fi_endpoint.h:168
#1 init_fabric (fabric=0x1578c40, Params=<optimized out>)
at /home/users/khuck/src/ADIOS2/source/adios2/toolkit/sst/dp/rdma_dp.c:198
#2 0x00007fef071c31ec in RdmaInitWriter (Svcs=0x7fef073cbac0 <Svcs>, CP_Stream=0x155c420, Params=0x155c2f8)
at /home/users/khuck/src/ADIOS2/source/adios2/toolkit/sst/dp/rdma_dp.c:558
#3 0x00007fef071b9ba6 in SstWriterOpen (Name=Name@entry=0x155c3e0 "sim.bp", Params=Params@entry=0x155c2f8,
comm=comm@entry=0x155a710) at /home/users/khuck/src/ADIOS2/source/adios2/toolkit/sst/cp/cp_writer.c:1124
#4 0x00007fef08ed7827 in adios2::core::engine::SstWriter::SstWriter(adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, ompi_communicator_t*) ()
at /home/users/khuck/src/ADIOS2/source/adios2/engine/sst/SstWriter.cpp:35
#5 0x00007fef08bde003 in construct<adios2::core::engine::SstWriter, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, ompi_communicator_t*&>
(this=<optimized out>, __p=0x155c230) at /storage/packages/gcc/8.1/include/c++/8.1.0/new:169
#6 construct<adios2::core::engine::SstWriter, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, ompi_communicator_t*&> (__a=...,
__p=0x155c230) at /storage/packages/gcc/8.1/include/c++/8.1.0/bits/alloc_traits.h:475
#7 _Sp_counted_ptr_inplace<adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, ompi_communicator_t*&> (__a=..., this=0x155c220)
at /storage/packages/gcc/8.1/include/c++/8.1.0/bits/shared_ptr_base.h:549
#8 __shared_count<adios2::core::engine::SstWriter, std::allocator<adios2::core::engine::SstWriter>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, ompi_communicator_t*&> (__a=..., this=<optimized out>)
at /storage/packages/gcc/8.1/include/c++/8.1.0/bits/shared_ptr_base.h:662
#9 __shared_ptr<std::allocator<adios2::core::engine::SstWriter>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, ompi_communicator_t*&> (
__a=..., __tag=..., this=<optimized out>)
at /storage/packages/gcc/8.1/include/c++/8.1.0/bits/shared_ptr_base.h:1328
#10 shared_ptr<std::allocator<adios2::core::engine::SstWriter>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, ompi_communicator_t*&> (
__a=..., __tag=..., this=<optimized out>)
at /storage/packages/gcc/8.1/include/c++/8.1.0/bits/shared_ptr.h:360
---Type <return> to continue, or q <return> to quit---
#11 allocate_shared<adios2::core::engine::SstWriter, std::allocator<adios2::core::engine::SstWriter>, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, ompi_communicator_t*&> (__a=...)
at /storage/packages/gcc/8.1/include/c++/8.1.0/bits/shared_ptr.h:707
#12 make_shared<adios2::core::engine::SstWriter, adios2::core::IO&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode const&, ompi_communicator_t*&> ()
at /storage/packages/gcc/8.1/include/c++/8.1.0/bits/shared_ptr.h:723
#13 adios2::core::IO::Open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, ompi_communicator_t*) () at /home/users/khuck/src/ADIOS2/source/adios2/core/IO.cpp:567
#14 0x00007fef08f3f3d3 in adios2::IO::Open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, adios2::Mode, ompi_communicator_t*) ()
at /home/users/khuck/src/ADIOS2/bindings/CXX11/adios2/cxx11/IO.cpp:112
#15 0x000000000040fb36 in IO::IO(Settings const&, ompi_communicator_t*) () at simulation/IO_adios2.cpp:79
#16 0x000000000040b4ef in main () at simulation/heatSimulation.cpp:78
#17 0x00007fef076653d5 in __libc_start_main (main=0x40b320 <main>, argc=8, argv=0x7ffe942a12e8,
init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe942a12d8)
at ../csu/libc-start.c:266
#18 0x000000000040b71f in _start () at simulation/IO_adios2.cpp:118
```Eisenhauer, GregEisenhauer, Greghttps://code.ornl.gov/pnb/ADIOS2/-/issues/1710difficulties building rpm package for adios22019-10-01T11:48:04ZPodhorszki, Norbertdifficulties building rpm package for adios2*Created by: olesenm*
I was attempting to get an [ADIOS2 science package](https://build.opensuse.org/package/show/science/adios2) for openSUSE. I thought it could just be as simple as swapping out autoconfig for cmake, but ran into some...*Created by: olesenm*
I was attempting to get an [ADIOS2 science package](https://build.opensuse.org/package/show/science/adios2) for openSUSE. I thought it could just be as simple as swapping out autoconfig for cmake, but ran into some issues.
It is still a rough work-in-progress, but I've attached a tar file with specs etc.
The main issue seems to be with the different ideas of the file-system or build root.
The RPM build is within a BUILDROOT/ structure, which means that stuff like /usr/lib64/cmake/... are actually found below that.
I figured there must be a kitware guy (eg, @chuckatkins) who can resolve this in an instant.
[adios2-rpm-build.tar.gz](https://github.com/ornladios/ADIOS2/files/3560422/adios2-rpm-build.tar.gz)
https://code.ornl.gov/pnb/ADIOS2/-/issues/1705Python write api does not check for non-contiguous or non-c-layout numpy arrays2019-10-16T17:30:59ZPodhorszki, NorbertPython write api does not check for non-contiguous or non-c-layout numpy arrays*Created by: isosc*
Issue: Numpy arrays which are not contiguous C-ordered arrays are not properly understood by the write API, leading to missing or out-of-order data in ADIOS2. Currently can work around by making a c-ordered copy (whi...*Created by: isosc*
Issue: Numpy arrays which are not contiguous C-ordered arrays are not properly understood by the write API, leading to missing or out-of-order data in ADIOS2. Currently can work around by making a c-ordered copy (which is expensive for large arrays), but generally it seems like the python bindings should check order and handle appropriately.
Code example:
---
```python
from mpi4py import MPI
import adios2 as ad
import numpy as np
import sys
dd = np.array([[111,112,113,114],[121,122,123,124],[131,132,133,134]])
global_shape = dd.shape
local_shape = dd.shape
offset = [0,0]
print ("Original order:")
print (dd)
#Write using c order
with ad.open('c.bp','w', MPI.COMM_WORLD) as fh:
fh.write('myvar',np.copy(dd,'C'),global_shape,offset,local_shape)
with ad.open('c.bp','r', MPI.COMM_WORLD) as fh:
aa = fh.read('myvar')
print ("Written with c order:")
print (aa)
#write using fortran order
with ad.open('f.bp','w', MPI.COMM_WORLD) as fh:
fh.write('myvar',np.copy(dd,'F'),global_shape,offset,local_shape)
with ad.open('f.bp','r', MPI.COMM_WORLD) as fh:
aa = fh.read('myvar')
print ("Written with fortran order:")
print (aa)
```
---
Result:
Original order:
[[111 112 113 114]
[121 122 123 124]
[131 132 133 134]]
Written with c order:
[[111 112 113 114]
[121 122 123 124]
[131 132 133 134]]
Written with fortran order:
[[111 121 131 112]
[122 132 113 123]
[133 114 124 134]]
https://code.ornl.gov/pnb/ADIOS2/-/issues/1688Block selection issue when one step does not have any block2019-10-15T23:48:38ZPodhorszki, NorbertBlock selection issue when one step does not have any block*Created by: pnorbert*
`ERROR: invalid blockID 0 from steps start 2 in variable v3, check argument to Variable<T>::SetBlockID, in call to Get`
The issue shows up in the next step where the variable appears again.
The examples/bas...*Created by: pnorbert*
`ERROR: invalid blockID 0 from steps start 2 in variable v3, check argument to Variable<T>::SetBlockID, in call to Get`
The issue shows up in the next step where the variable appears again.
The examples/basics/localArray example shows this bug if the writer code is forced to omit writing a variable (v3) in step 2 in `localArray_write.cpp:166-169`
```
if (step == 2)
{
chance = 0;
}
```
This will result in v3 missing in step 2:
```
$ bpls -l localArray.bp/ -D v3
double v3 4*[__]*{__} = 0 / 3.4
step 0:
block 0: [0:8] = 0 / 0
step 1:
block 0: [0:5] = 0.1 / 0.1
block 1: [0:6] = 1.1 / 1.1
step 3:
block 0: [0:9] = 1.3 / 1.3
step 4:
block 0: [0:5] = 0.4 / 0.4
block 1: [0:5] = 1.4 / 1.4
block 2: [0:5] = 3.4 / 3.4
```
Then the read example fails when getting variable v3 in step 3.
```
$ ./bin/localArray_read
Process step 0:
...
Process step 2:
v0 has 4 blocks in step 2
block 0 size = "6" offset = "" : 0.2 0.2 0.2 0.2 0.2 0.2
block 1 size = "6" offset = "" : 1.2 1.2 1.2 1.2 1.2 1.2
block 2 size = "6" offset = "" : 2.2 2.2 2.2 2.2 2.2 2.2
block 3 size = "6" offset = "" : 3.2 3.2 3.2 3.2 3.2 3.2
v1 has 4 blocks in step 2
block 0 size = "6" offset = "" : 0.2 0.2 0.2 0.2 0.2 0.2
block 1 size = "10" offset = "" : 1.2 1.2 1.2 1.2 1.2 1.2 1.2 1.2 1.2 1.2
block 2 size = "5" offset = "" : 2.2 2.2 2.2 2.2 2.2
block 3 size = "10" offset = "" : 3.2 3.2 3.2 3.2 3.2 3.2 3.2 3.2 3.2 3.2
v2 has 4 blocks in step 2
block 0 size = "6" offset = "" : 0.2 0.2 0.2 0.2 0.2 0.2
block 1 size = "7" offset = "" : 1.2 1.2 1.2 1.2 1.2 1.2 1.2
block 2 size = "9" offset = "" : 2.2 2.2 2.2 2.2 2.2 2.2 2.2 2.2 2.2
block 3 size = "10" offset = "" : 3.2 3.2 3.2 3.2 3.2 3.2 3.2 3.2 3.2 3.2
Variable v3 not found in step 2
Process step 3:
v0 has 4 blocks in step 3
block 0 size = "6" offset = "" : 0.3 0.3 0.3 0.3 0.3 0.3
block 1 size = "6" offset = "" : 1.3 1.3 1.3 1.3 1.3 1.3
block 2 size = "6" offset = "" : 2.3 2.3 2.3 2.3 2.3 2.3
block 3 size = "6" offset = "" : 3.3 3.3 3.3 3.3 3.3 3.3
v1 has 4 blocks in step 3
block 0 size = "6" offset = "" : 0.3 0.3 0.3 0.3 0.3 0.3
block 1 size = "10" offset = "" : 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3
block 2 size = "5" offset = "" : 2.3 2.3 2.3 2.3 2.3
block 3 size = "10" offset = "" : 3.3 3.3 3.3 3.3 3.3 3.3 3.3 3.3 3.3 3.3
v2 has 4 blocks in step 3
block 0 size = "5" offset = "" : 0.3 0.3 0.3 0.3 0.3
block 1 size = "10" offset = "" : 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3
block 2 size = "7" offset = "" : 2.3 2.3 2.3 2.3 2.3 2.3 2.3
block 3 size = "9" offset = "" : 3.3 3.3 3.3 3.3 3.3 3.3 3.3 3.3 3.3
v3 has 1 blocks in step 3
Invalid argument exception, STOPPING PROGRAM
ERROR: invalid blockID 0 from steps start 2 in variable v3, check argument to Variable<T>::SetBlockID, in call to Get
```
Note: This reading example uses BeginStep/EndStep to step through the steps, so no SetStepSelection() is used here.
Note: The bug appears with both BP3 and BP4 engines.
2.6.0 Production Releasehttps://code.ornl.gov/pnb/ADIOS2/-/issues/1675Fail to build ADIOS2 in Bebop2019-09-03T18:30:40ZPodhorszki, NorbertFail to build ADIOS2 in Bebop*Created by: tongshu83*
$ module unload intel-mkl/2017.3.196-v7uuj6z
$ module load intel/17.0.4-74uvhji
$ module load intel-mpi/2017.3-dfphq6k
$ module load libpsm2/10.3-17
$ module load cmake/3.13.4-354d6wl
$ module load jdk/8u141...*Created by: tongshu83*
$ module unload intel-mkl/2017.3.196-v7uuj6z
$ module load intel/17.0.4-74uvhji
$ module load intel-mpi/2017.3-dfphq6k
$ module load libpsm2/10.3-17
$ module load cmake/3.13.4-354d6wl
$ module load jdk/8u141-b15-mopj6qr
$ module load tcl/8.6.6-x4wnbsg
$ module load bzip2/1.0.6-hcvyuh5
$ module load zlib/1.2.11-6632jqd
$ module load anaconda3/5.2.0
$ cd ADIOS2
$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_PREFIX=../../install/adios2 -DADIOS2_USE_Fortran=OFF ..
...
ADIOS2 build configuration:
ADIOS Version: 2.4.0.239
C++ Compiler : GNU 4.8.5
/usr/bin/c++
Installation prefix: /lcrc/project/PACC/tong/HPC_Autotuner/install/adios2
bin: bin
lib: lib64
include: include
cmake: lib64/cmake/adios2
python: lib64/python3.6/site-packages
Features:
Library Type: shared
Build Type: Release
Testing: ON
Build Options:
Blosc : OFF
BZip2 : ON
ZFP : OFF
SZ : OFF
MGARD : OFF
PNG : ON
MPI : ON
DataMan : OFF
Table : OFF
SSC : OFF
SST : ON
DataSpaces: OFF
ZeroMQ : OFF
HDF5 : OFF
Python : ON
Fortran : OFF
SysVShMem: ON
Profiling: ON
Endian_Reverse: OFF
RDMA Transport for Staging: Available
-- Configuring done
-- Generating done
$ make -j8
...
[ 48%] Linking CXX shared library ../../lib64/libadios2.so
[ 48%] Built target adios2
Scanning dependencies of target sst_conn_tool
Scanning dependencies of target TestADIOSInterfaceWrite
Scanning dependencies of target TestADIOSInterface
Scanning dependencies of target TestADIOSNoMpi
Scanning dependencies of target bpls
Scanning dependencies of target adios_reorganize
[ 48%] Building CXX object testing/adios2/interface/CMakeFiles/TestADIOSInterface.dir/TestADIOSInterface.cpp.o
[ 48%] Building C object source/adios2/toolkit/sst/util/CMakeFiles/sst_conn_tool.dir/sst_conn_tool.c.o
[ 48%] Building CXX object testing/adios2/interface/CMakeFiles/TestADIOSInterfaceWrite.dir/TestADIOSInterfaceWrite.cpp.o
[ 48%] Building CXX object testing/adios2/interface/CMakeFiles/TestADIOSNoMpi.dir/TestADIOSNoMpi.cpp.o
[ 49%] Building CXX object source/utils/CMakeFiles/bpls.dir/bpls/bpls.cpp.o
[ 50%] Building CXX object source/utils/CMakeFiles/adios_reorganize.dir/adios_reorganize/main.cpp.o
Scanning dependencies of target adios_iotest
[ 50%] Building CXX object source/utils/adios_iotest/CMakeFiles/adios_iotest.dir/settings.cpp.o
[ 51%] Linking C executable ../../../../../bin/sst_conn_tool
[ 51%] Building CXX object source/utils/adios_iotest/CMakeFiles/adios_iotest.dir/decomp.cpp.o
/usr/bin/ld: warning: libpsm_infinipath.so.1, needed by //usr/lib64/libfabric.so.1, not found (try using -rpath or -rpath-link)
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_fp_msg@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_get_parameters'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_isend'
../../../../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_begin_read_from_memory'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_reply_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_getctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_num_devunits'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_send'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open_opts_get_defaults'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_request_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_connect'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_cancel'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_am_register_handlers_2@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_close'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_ipeek'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_setctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_test'
../../../../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_finish_read'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_irecv'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_iprobe'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_epid_lookup'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_register_handler'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_register_handlers'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open'
//usr/lib64/libfabric.so.1: undefined reference to `psm_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_get_string'
collect2: error: ld returned 1 exit status
make[2]: *** [bin/sst_conn_tool] Error 1
make[1]: *** [source/adios2/toolkit/sst/util/CMakeFiles/sst_conn_tool.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 51%] Building CXX object source/utils/CMakeFiles/adios_reorganize.dir/adios_reorganize/Reorganize.cpp.o
[ 51%] Building CXX object source/utils/adios_iotest/CMakeFiles/adios_iotest.dir/processConfig.cpp.o
[ 51%] Building CXX object source/utils/CMakeFiles/adios_reorganize.dir/Utils.cpp.o
[ 51%] Linking CXX executable ../../../bin/TestADIOSNoMpi
/usr/bin/ld: warning: libpsm_infinipath.so.1, needed by //usr/lib64/libfabric.so.1, not found (try using -rpath or -rpath-link)
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_fp_msg@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_get_parameters'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_isend'
../../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_begin_read_from_memory'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_reply_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_getctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_num_devunits'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_send'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open_opts_get_defaults'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_request_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_connect'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_cancel'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_am_register_handlers_2@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_close'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_ipeek'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_setctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_test'
../../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_finish_read'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_irecv'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_iprobe'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_epid_lookup'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_register_handler'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_register_handlers'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open'
//usr/lib64/libfabric.so.1: undefined reference to `psm_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_get_string'
collect2: error: ld returned 1 exit status
make[2]: *** [bin/TestADIOSNoMpi] Error 1
make[1]: *** [testing/adios2/interface/CMakeFiles/TestADIOSNoMpi.dir/all] Error 2
[ 51%] Building CXX object source/utils/adios_iotest/CMakeFiles/adios_iotest.dir/ioGroup.cpp.o
Scanning dependencies of target adios2py
[ 51%] Building CXX object source/utils/CMakeFiles/bpls.dir/__/__/thirdparty/pugixml/pugixml/src/pugixml.cpp.o
[ 51%] Building CXX object bindings/Python/CMakeFiles/adios2py.dir/py11ADIOS.cpp.o
[ 51%] Building CXX object bindings/Python/CMakeFiles/adios2py.dir/py11IO.cpp.o
[ 52%] Linking CXX executable ../../bin/adios_reorganize
[ 52%] Linking CXX executable ../../../bin/TestADIOSInterfaceWrite
/usr/bin/ld: warning: libpsm_infinipath.so.1, needed by //usr/lib64/libfabric.so.1, not found (try using -rpath or -rpath-link)
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_fp_msg@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_get_parameters'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_isend'
../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_begin_read_from_memory'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_reply_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_getctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_num_devunits'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_send'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open_opts_get_defaults'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_request_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_connect'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_cancel'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_am_register_handlers_2@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_close'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_ipeek'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_setctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_test'
../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_finish_read'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_irecv'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_iprobe'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_epid_lookup'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_register_handler'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_register_handlers'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open'
//usr/lib64/libfabric.so.1: undefined reference to `psm_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_get_string'
collect2: error: ld returned 1 exit status
make[2]: *** [bin/adios_reorganize] Error 1
make[1]: *** [source/utils/CMakeFiles/adios_reorganize.dir/all] Error 2
[ 53%] Building CXX object source/utils/adios_iotest/CMakeFiles/adios_iotest.dir/stream.cpp.o
/usr/bin/ld: warning: libpsm_infinipath.so.1, needed by //usr/lib64/libfabric.so.1, not found (try using -rpath or -rpath-link)
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_fp_msg@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_get_parameters'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_isend'
../../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_begin_read_from_memory'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_reply_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_getctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_num_devunits'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_send'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open_opts_get_defaults'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_request_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_connect'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_cancel'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_am_register_handlers_2@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_close'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_ipeek'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_setctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_test'
../../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_finish_read'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_irecv'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_iprobe'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_epid_lookup'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_register_handler'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_register_handlers'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open'
//usr/lib64/libfabric.so.1: undefined reference to `psm_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_get_string'
collect2: error: ld returned 1 exit status
make[2]: *** [bin/TestADIOSInterfaceWrite] Error 1
make[1]: *** [testing/adios2/interface/CMakeFiles/TestADIOSInterfaceWrite.dir/all] Error 2
[ 53%] Building CXX object source/utils/adios_iotest/CMakeFiles/adios_iotest.dir/adiosStream.cpp.o
[ 53%] Building CXX object source/utils/adios_iotest/CMakeFiles/adios_iotest.dir/adios_iotest.cpp.o
[ 53%] Building CXX object bindings/Python/CMakeFiles/adios2py.dir/py11Variable.cpp.o
[ 53%] Linking CXX executable ../../../bin/TestADIOSInterface
[ 54%] Building CXX object bindings/Python/CMakeFiles/adios2py.dir/py11Attribute.cpp.o
/usr/bin/ld: warning: libpsm_infinipath.so.1, needed by //usr/lib64/libfabric.so.1, not found (try using -rpath or -rpath-link)
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_fp_msg@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_get_parameters'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_isend'
../../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_begin_read_from_memory'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_reply_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_getctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_num_devunits'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_send'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open_opts_get_defaults'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_request_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_connect'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_cancel'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_am_register_handlers_2@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_close'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_ipeek'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_setctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_test'
../../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_finish_read'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_irecv'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_iprobe'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_epid_lookup'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_register_handler'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_register_handlers'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open'
//usr/lib64/libfabric.so.1: undefined reference to `psm_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_get_string'
collect2: error: ld returned 1 exit status
make[2]: *** [bin/TestADIOSInterface] Error 1
make[1]: *** [testing/adios2/interface/CMakeFiles/TestADIOSInterface.dir/all] Error 2
[ 54%] Building CXX object bindings/Python/CMakeFiles/adios2py.dir/py11Engine.cpp.o
[ 54%] Building CXX object bindings/Python/CMakeFiles/adios2py.dir/py11Operator.cpp.o
[ 54%] Building CXX object bindings/Python/CMakeFiles/adios2py.dir/py11File.cpp.o
[ 54%] Linking CXX executable ../../../bin/adios_iotest
/usr/bin/ld: warning: libpsm_infinipath.so.1, needed by //usr/lib64/libfabric.so.1, not found (try using -rpath or -rpath-link)
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_fp_msg@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_get_parameters'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_isend'
../../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_begin_read_from_memory'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_reply_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_getctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_num_devunits'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_send'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open_opts_get_defaults'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_request_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_connect'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_cancel'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_am_register_handlers_2@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_close'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_ipeek'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_setctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_test'
../../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_finish_read'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_irecv'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_iprobe'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_epid_lookup'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_register_handler'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_register_handlers'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open'
//usr/lib64/libfabric.so.1: undefined reference to `psm_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_get_string'
collect2: error: ld returned 1 exit status
make[2]: *** [bin/adios_iotest] Error 1
make[1]: *** [source/utils/adios_iotest/CMakeFiles/adios_iotest.dir/all] Error 2
[ 54%] Building CXX object bindings/Python/CMakeFiles/adios2py.dir/py11glue.cpp.o
[ 54%] Linking CXX executable ../../bin/bpls
/usr/bin/ld: warning: libpsm_infinipath.so.1, needed by //usr/lib64/libfabric.so.1, not found (try using -rpath or -rpath-link)
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_fp_msg@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_get_parameters'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_isend'
../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_begin_read_from_memory'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_reply_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_info_query@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_getctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_num_devunits'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_send'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open_opts_get_defaults'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_mq_ipeek_dequeue_multi@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_request_short'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_connect'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_cancel'
//usr/lib64/libfabric.so.1: undefined reference to `psm2_am_register_handlers_2@PSM2_1.0'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_close'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_ipeek'
//usr/lib64/libfabric.so.1: undefined reference to `psm_epaddr_setctxt'
//usr/lib64/libfabric.so.1: undefined reference to `psm_finalize'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_test'
../../lib64/libadios2.so.2.4.0: undefined reference to `png_image_finish_read'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_irecv'
//usr/lib64/libfabric.so.1: undefined reference to `psm_mq_iprobe'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_epid_lookup'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_register_handler'
//usr/lib64/libfabric.so.1: undefined reference to `psm_am_register_handlers'
//usr/lib64/libfabric.so.1: undefined reference to `psm_ep_open'
//usr/lib64/libfabric.so.1: undefined reference to `psm_init'
//usr/lib64/libfabric.so.1: undefined reference to `psm_error_get_string'
collect2: error: ld returned 1 exit status
make[2]: *** [bin/bpls] Error 1
make[1]: *** [source/utils/CMakeFiles/bpls.dir/all] Error 2
[ 55%] Linking CXX shared module ../../lib64/python3.6/site-packages/adios2.cpython-36m-x86_64-linux-gnu.so
[ 55%] Built target adios2py
make: *** [all] Error 2
[tshu@beboplogin2 HPC_Autotuner]$ vi build-adios2.sh
E575: viminfo: Illegal starting char in line: 0/time.txt
E575: viminfo: Illegal starting char in line: ^I"^I1^I0
Could you please let me know how to fix this issue? Thank you very much!
Eisenhauer, GregEisenhauer, Greg