ADIOS2 issueshttps://code.ornl.gov/pnb/ADIOS2/-/issues2019-10-15T19:58:14Zhttps://code.ornl.gov/pnb/ADIOS2/-/issues/1825Downstream RPATH overwritten2019-10-15T19:58:14ZPodhorszki, NorbertDownstream RPATH overwritten*Created by: ax3l*
I am using the `ADIOS2Config.cmake` module for v2.5.0 downstream in openPMD-api.
Somehow, the ADIOS2 `CMakeLists.txt` lines overwrite my *downstream* [`-DCMAKE_INSTALL_RPATH=$ORIGIN`](https://github.com/openPMD/ope...*Created by: ax3l*
I am using the `ADIOS2Config.cmake` module for v2.5.0 downstream in openPMD-api.
Somehow, the ADIOS2 `CMakeLists.txt` lines overwrite my *downstream* [`-DCMAKE_INSTALL_RPATH=$ORIGIN`](https://github.com/openPMD/openPMD-api/blob/130f7d3ceef25a1412515987728209ccd0ecb31e/setup.py#L60) setting to read `$ORIGIN/.libsopenpmd_api` instead of `$ORIGIN`.
https://github.com/ornladios/ADIOS2/blob/v2.5.0/CMakeLists.txt#L156-L179
As soon as I am not linking to ADIOS2 anymore, the problem is gone.
Probably we should only set the rpath as CMake property on targets or append to already set, user-provided `CMAKE_INSTALL_RPATH` locations?https://code.ornl.gov/pnb/ADIOS2/-/issues/1823Dims of size_t potentially problematic in 32-bit systems2019-10-15T01:13:45ZPodhorszki, NorbertDims of size_t potentially problematic in 32-bit systems*Created by: ax3l*
As we discussed last Monday, I think that the [definition of `adios2::Dims`](https://github.com/ornladios/ADIOS2/blob/v2.5.0/source/adios2/common/ADIOSTypes.h#L188) as of
```C++
using Dims = std::vector<size_t>;
``...*Created by: ax3l*
As we discussed last Monday, I think that the [definition of `adios2::Dims`](https://github.com/ornladios/ADIOS2/blob/v2.5.0/source/adios2/common/ADIOSTypes.h#L188) as of
```C++
using Dims = std::vector<size_t>;
```
might be potentially a bit problematic. The problem I am describing here applies to clusters of 32bit systems, such as Raspberry PI clusters, older Nvidia Tegra systems, among others. (Been there with PIConGPU, done that.)
`adios2::Dims` are used, inter alia, for (global) `shapes`, `counts` and `start`s (/offsets) in user-control. Since these metadata attributes are somewhat loose in ADIOS, user-level concepts like a [moving simulation window](https://github.com/ComputationalRadiationPhysics/picongpu/wiki/PIConGPU-domain-definitions#moving-window) can at some point overflow the range of `size_t` for e.g. the `start`. Similar problems can occur when writing the 1D shape of dataframes/tabular data, such as global particle arrays.
It is therefore probably safer to define this to a [fixed integer type, such as](https://github.com/openPMD/openPMD-api/blob/0.9.0-alpha/include/openPMD/Dataset.hpp#L33-L34)
```C++
#include <vector>
#include <cstdint>
// ...
using Dims = std::vector< std::uint64_t >;
```https://code.ornl.gov/pnb/ADIOS2/-/issues/1822Docs: endian_reverse unclear2019-10-15T22:35:34ZPodhorszki, NorbertDocs: endian_reverse unclear*Created by: ax3l*
Migrated from [here](https://github.com/spack/spack/pull/13107#discussion_r333224411) so it does not get lost.
The CMake flag `ADIOS2_USE_ENDIAN_REVERSE=ON` looks scary to me for the following reason: if a file was...*Created by: ax3l*
Migrated from [here](https://github.com/spack/spack/pull/13107#discussion_r333224411) so it does not get lost.
The CMake flag `ADIOS2_USE_ENDIAN_REVERSE=ON` looks scary to me for the following reason: if a file was written in one endinaness and read again in another, the BP file format should encode that and our BPEngine could automatically take care of it. (E.g. HDF5 does it that way).
I was working/interning a few years ago in Juelich where serveral big and little endian machines were connected to the same file system and workflows going back and forth were common. Using various adios modules, depending on the origin of the data, would be very complicated.
If it's not possible to already express this in BP itself, which I think it should be for portability, then we should at least make this a runtime option for now.https://code.ornl.gov/pnb/ADIOS2/-/issues/1820Python API for associated attributes2019-10-15T20:46:00ZPodhorszki, NorbertPython API for associated attributes*Created by: pnorbert*
Is there a Python equivalent of the C++ AvailableAttributes(string varname)?
Needed by @isosc
This does not work:
```
$ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linu...*Created by: pnorbert*
Is there a Python equivalent of the C++ AvailableAttributes(string varname)?
Needed by @isosc
This does not work:
```
$ python3
Python 3.5.2 (default, Jul 10 2019, 11:58:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import adios2
>>> import numpy
>>> fr = adios2.open("heat2d/fortran/a.bp", "r")
>>> vars_info = fr.available_variables()
>>> vars_info
{'T': {'AvailableStepsCount': '4', 'Type': 'double', 'SingleValue': 'false', 'Max': '200', 'Shape': '768, 1024', 'Min': '0'}, 'dT': {'AvailableStepsCount': '4', 'Type': 'double', 'SingleValue': 'false', 'Max': '74.5893', 'Shape': '768, 1024', 'Min': '-77.3378'}}
>>> attrs_info = fr.available_attributes()
>>> attrs_info
{'dT/description': {'Value': '"Temperature difference between two steps calculated in analysis"', 'Type': 'string', 'Elements': '1'}, 'T/unit': {'Value': '"C"', 'Type': 'string', 'Elements': '1'}, 'T/description': {'Value': '"Temperature from simulation"', 'Type': 'string', 'Elements': '1'}}
>>> desc = fr.available_attributes('dT')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: available_attributes(): incompatible function arguments. The following argument types are supported:
1. (self: adios2.File) -> Dict[str, Dict[str, str]]
Invoked with: <adios2.file named '../heat2d/fortran/a.bp' and mode 'r'>, 'dT'
```https://code.ornl.gov/pnb/ADIOS2/-/issues/1816Auto-generated info for directory hierarchy in ADIOS output2020-03-02T12:12:35ZPodhorszki, NorbertAuto-generated info for directory hierarchy in ADIOS output*Created by: pnorbert*
Challenge: reconstruct the hierarchy at read time fast. OpenPMD spends 30 seconds to do that currently.
Idea: ADIOS could save the hierarchy as a string variable as a JSON/YAML document at write time, that eith...*Created by: pnorbert*
Challenge: reconstruct the hierarchy at read time fast. OpenPMD spends 30 seconds to do that currently.
Idea: ADIOS could save the hierarchy as a string variable as a JSON/YAML document at write time, that either the application can use to generate the hierarchy or ADIOS could use it to generate a tree instead of the plain map of variables and attributes.
This could speed up the variable/variable and attribute/variable association functions at read time as well.
@ax3l @psychocoderHPC @franzpoeschel @BeyondEspressohttps://code.ornl.gov/pnb/ADIOS2/-/issues/1815Mutable attributes, single values, association of variables to variables2019-10-07T22:25:09ZPodhorszki, NorbertMutable attributes, single values, association of variables to variables*Created by: pnorbert*
We have a gap between the convenience of attributes and the single value variables, hence @ax3l is asking for the ability to change an attribute's value over time. Instead, they could use single values for such in...*Created by: pnorbert*
We have a gap between the convenience of attributes and the single value variables, hence @ax3l is asking for the ability to change an attribute's value over time. Instead, they could use single values for such information but only if variables can be associated to variables (like attributes to variables currently).
https://code.ornl.gov/pnb/ADIOS2/-/issues/1814BP-X: manage multiple pre-sized buffers to avoid realloc of huge single buffer2019-10-07T20:32:52ZPodhorszki, NorbertBP-X: manage multiple pre-sized buffers to avoid realloc of huge single buffer*Created by: pnorbert*
PIConGPU may use all memory so that ADIOS buffer just fits. If a realloc would happen, it could fail due to lack of memory. Still, a some memory would be available to create another buffer to store more data.
S...*Created by: pnorbert*
PIConGPU may use all memory so that ADIOS buffer just fits. If a realloc would happen, it could fail due to lack of memory. Still, a some memory would be available to create another buffer to store more data.
Suggestion: BP-X should manage multiple smaller buffers (we are talking about 500MB, 1GB buffers here) and let (sophisticated) user create these buffers on the fly.
https://code.ornl.gov/pnb/ADIOS2/-/issues/1813Installation Error2019-12-02T19:36:53ZPodhorszki, NorbertInstallation Error*Created by: AaronV77*
Hey everyone,
I'm getting the following error when trying to build 2.4 on Onyx at ERDC. I don't know why I'm getting this error message, because in cmake I am specifiying the gcc compiler and all should be well...*Created by: AaronV77*
Hey everyone,
I'm getting the following error when trying to build 2.4 on Onyx at ERDC. I don't know why I'm getting this error message, because in cmake I am specifiying the gcc compiler and all should be well. Is this an ADIOS bug or something that I haven't loaded properly on Onyx? Lastly, this isn't a lot of information to go off of, so let me know what other files might be helpful for tracking down this issue.
```
/usr/bin/ld: cannot find -lgcc_s
/usr/bin/ld: ../../../../../lib64/libadios2_enet.a(unix.c.o): in function `enet_address_set_host':
unix.c:(.text+0x15f): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: ../../../../../lib64/libadios2_evpath.a(ip_config.c.o): in function `get_qual_hostname':
ip_config.c:(.text+0xdba): warning: Using 'gethostbyaddr' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: ../../../../../lib64/libadios2_evpath.a(ip_config.c.o): in function `get_self_ip_iface':
ip_config.c:(.text+0x400): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
```
Thanks!https://code.ornl.gov/pnb/ADIOS2/-/issues/1811ADIOS 2.5.0: Build issue with ZFP2020-03-03T17:40:05ZPodhorszki, NorbertADIOS 2.5.0: Build issue with ZFP*Created by: pnorbert*
cmake configuration does not find my ZFP installation on my VM.
This did work with v2.4.0:
```
adios@adiosVM:~/Software/ADIOS2/build$
$ cmake -DCMAKE_INSTALL_PREFIX=/opt/adios2 -DADIOS2_USE_MPI=ON \
...*Created by: pnorbert*
cmake configuration does not find my ZFP installation on my VM.
This did work with v2.4.0:
```
adios@adiosVM:~/Software/ADIOS2/build$
$ cmake -DCMAKE_INSTALL_PREFIX=/opt/adios2 -DADIOS2_USE_MPI=ON \
-DADIOS2_USE_Fortran=ON -DADIOS2_USE_Python=ON \
-DADIOS2_USE_HDF5=ON -DHDF5_ROOT=/opt/hdf5-parallel \
-DBLOSC_ROOT=/opt/blosc \
-DADIOS2_USE_MGARD=ON -DMGARD_ROOT=/opt/MGARD \
-DSZ_ROOT=/opt/SZ/2.0.2.1 \
-DZFP_ROOT=/opt/zfp/0.5.5 \
-DADIOS2_BUILD_TESTING=OFF \
-DADIOS2_BUILD_EXAMPLES_EXPERIMENTAL=OFF \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DPYTHON_EXECUTABLE=/usr/bin/python3 ..
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
...
-- Found Blosc: /opt/blosc/lib/libblosc.so (found version "..")
-- Found BZip2: /usr/lib/x86_64-linux-gnu/libbz2.so (found version "1.0.6")
-- Looking for BZ2_bzCompressInit
-- Looking for BZ2_bzCompressInit - found
-- Found ZFP: /opt/zfp/0.5.5/lib/libzfp.a
-- Found SZ: /opt/SZ/2.0.2.1/lib/libSZ.a
-- Found MGARD: /opt/MGARD/lib/libmgard.a
-- Found ZLIB: /opt/SZ/2.0.2.1/lib/libzlib.a (found version "1.2.8")
...
```
The same command with v2.5.0 does not find ZFP:
```
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Blosc: /opt/blosc/lib/libblosc.so (found suitable version "1.17.1", minimum required is "1.7")
-- Found BZip2: /usr/lib/x86_64-linux-gnu/libbz2.so (found version "1.0.6")
-- Looking for BZ2_bzCompressInit
-- Looking for BZ2_bzCompressInit - found
-- Could NOT find zfp (missing: zfp_DIR)
-- Found SZ: /opt/SZ/2.0.2.1/lib/libSZ.a
-- Found MGARD: /opt/MGARD/lib/libmgard.a
```
If I set **zfp_DIR**, then it works fine.
`$ export zfp_DIR=/opt/zfp/0.5.5/lib/cmake`
https://code.ornl.gov/pnb/ADIOS2/-/issues/1810ADIOS 2.5.0: Static Build Issues Linking Blosc2019-10-16T17:56:23ZPodhorszki, NorbertADIOS 2.5.0: Static Build Issues Linking Blosc*Created by: ax3l*
I am building a static version of ADIOS2 2.5.0 against a static version of C-Blosc 1.15.0 for a PyPI release of openPMD-api with the following CMake flags:
```
-DBUILD_SHARED_LIBS=OFF
-DCMAKE_POSITION_INDEPENDENT...*Created by: ax3l*
I am building a static version of ADIOS2 2.5.0 against a static version of C-Blosc 1.15.0 for a PyPI release of openPMD-api with the following CMake flags:
```
-DBUILD_SHARED_LIBS=OFF
-DCMAKE_POSITION_INDEPENDENT_CODE=ON
-DADIOS2_BUILD_EXAMPLES=OFF
-DADIOS2_BUILD_TESTING=OFF
```
with results in the config
```
ADIOS2 build configuration:
ADIOS Version: 2.5.0
C++ Compiler : GNU 8.3.1
/opt/rh/devtoolset-8/root/usr/bin/c++
Fortran Compiler : GNU 8.3.1
/opt/rh/devtoolset-8/root/usr/bin/gfortran
Installation prefix: /usr
bin: bin
lib: lib64
include: include
cmake: lib64/cmake/adios2
Features:
Library Type: static (with PIC)
Build Type: Release
Testing: OFF
Examples: OFF
Build Options:
Blosc : ON
BZip2 : OFF
ZFP : OFF
SZ : OFF
MGARD : OFF
PNG : OFF
MPI : OFF
DataMan : OFF
Table : OFF
SSC : OFF
SST : ON
DataSpaces: OFF
ZeroMQ : OFF
HDF5 : ON
Python : OFF
Fortran : ON
SysVShMem: ON
Profiling: OFF
Endian_Reverse: OFF
RDMA Transport for Staging: Unconfigured
-- Configuring done
-- Generating done
```
Just or the record, this is the [Docker environment](https://github.com/openPMD/openPMD-api/blob/dev/Dockerfile) I am building in.
ADIOS2 builds the library, but fails in the `adios-config` step with:
```
Configuring adios2-config dummy CMake project
-- The C compiler identification is GNU 8.3.1
-- The CXX compiler identification is GNU 8.3.1
-- Check for working C compiler: /opt/rh/devtoolset-8/root/usr/bin/cc
-- Check for working C compiler: /opt/rh/devtoolset-8/root/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /opt/rh/devtoolset-8/root/usr/bin/c++
-- Check for working CXX compiler: /opt/rh/devtoolset-8/root/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The Fortran compiler identification is GNU 8.3.1
-- Check for working Fortran compiler: /opt/rh/devtoolset-8/root/usr/bin/gfortran
-- Check for working Fortran compiler: /opt/rh/devtoolset-8/root/usr/bin/gfortran -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /opt/rh/devtoolset-8/root/usr/bin/gfortran supports Fortran 90
-- Checking whether /opt/rh/devtoolset-8/root/usr/bin/gfortran supports Fortran 90 -- yes
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /usr/lib/libhdf5.a;/usr/lib64/librt.so;/usr/lib64/libdl.so;/usr/lib64/libm.so (found version "1.10.5") found components: C
-- Found atl: /usr/lib64/cmake/adios2/thirdparty/atl-config.cmake (found suitable version "2.2.1", minimum required is "2.2.1")
-- Found atl: /usr/lib64/cmake/adios2/thirdparty/atl-config.cmake (found version "2.2.1")
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.23")
-- Checking for module 'libffi'
-- Found libffi, version 3.0.5
-- Found LibFFI: -lffi
-- Found dill: /usr/lib64/cmake/adios2/thirdparty/dill-config.cmake (found version "2.4.0")
-- Found ffs: /usr/lib64/cmake/adios2/thirdparty/ffs-config.cmake (found suitable version "1.6.0", minimum required is "1.6.0")
-- Found dill: /usr/lib64/cmake/adios2/thirdparty/dill-config.cmake (found suitable version "2.4.0", minimum required is "2.4.0")
-- Found enet: /usr/lib64/cmake/adios2/thirdparty/enet-config.cmake (found suitable version "1.3.14", minimum required is "1.3.14")
-- Found EVPath: /usr/lib64/cmake/adios2/thirdparty/EVPathConfigCommon.cmake (found version "4.4.0")
-- Found adios2: /usr/lib64/cmake/adios2/adios2-config.cmake (found version "2.5.0")
-- Configuring done
CMake Error at CMakeLists.txt:12 (add_executable):
Target "with_C" links to target "Blosc::Blosc" but the target was not
found. Perhaps a find_package() call is missing for an IMPORTED target, or
an ALIAS target is missing?
CMake Error at CMakeLists.txt:24 (add_executable):
Target "with_Fortran" links to target "Blosc::Blosc" but the target was not
found. Perhaps a find_package() call is missing for an IMPORTED target, or
an ALIAS target is missing?
CMake Error at CMakeLists.txt:14 (add_executable):
Target "with_CXX" links to target "Blosc::Blosc" but the target was not
found. Perhaps a find_package() call is missing for an IMPORTED target, or
an ALIAS target is missing?
-- Generating done
CMake Generate step failed. Build files cannot be regenerated correctly.
Failed
CMake Error at cmake/install/post/cmake_install.cmake:44 (message):
generate-adios2-config.sh returned 1, should be 0
Call Stack (most recent call first):
cmake_install.cmake:51 (include)
make: *** [install] Error 1
```
I can skip this step with `-DADIOS2_INSTALL_GENERATE_CONFIG=OFF` but this only transitions the problem downstream to my build of openPMD-api.
It it possible some dependencies or really just an alias are missing from the CMake targets?https://code.ornl.gov/pnb/ADIOS2/-/issues/1808ADIOS 2.5.0 Warning: BPSerialiser Instantiation-after-Specialization2019-11-07T21:33:23ZPodhorszki, NorbertADIOS 2.5.0 Warning: BPSerialiser Instantiation-after-Specialization*Created by: ax3l*
With ADIOS 2.5.0 warning with `clang@5.0.0` and `-Winstantiation-after-specialization`:
```
source/adios2/toolkit/format/bp/BPSerializer.cpp:875:39:
error: explicit instantiation of
'PutAttributeCharacteristicVa...*Created by: ax3l*
With ADIOS 2.5.0 warning with `clang@5.0.0` and `-Winstantiation-after-specialization`:
```
source/adios2/toolkit/format/bp/BPSerializer.cpp:875:39:
error: explicit instantiation of
'PutAttributeCharacteristicValueInIndex<std::__cxx11::basic_string<char> >'
that occurs after an explicit specialization has no effect [-Winstantiation-after-specialization]
ADIOS2_FOREACH_ATTRIBUTE_STDTYPE_1ARG(declare_template_instantiation)
source/adios2/toolkit/format/bp/BPSerializer.inl:23:27:
note: previous template specialization is here
inline void BPSerializer::PutAttributeCharacteristicValueInIndex(
source/adios2/toolkit/format/bp/BPSerializer.cpp:875:39:
error: explicit instantiation of
'GetAttributeSizeInData<std::__cxx11::basic_string<char> >'
that occurs after an explicit specialization has no effect [-Winstantiation-after-specialization]
ADIOS2_FOREACH_ATTRIBUTE_STDTYPE_1ARG(declare_template_instantiation)
source/adios2/toolkit/format/bp/BPSerializer.inl:68:29:
note: previous template specialization is here
inline size_t BPSerializer::GetAttributeSizeInData(source/adios2/toolkit/format/bp/BPSerializer.cpp:898:29: error: explicit instantiation of
'PutPayloadInBuffer<std::__cxx11::basic_string<char> >'
that occurs after an explicit specialization has no effect
[-Winstantiation-after-specialization]
ADIOS2_FOREACH_STDTYPE_1ARG(declare_template_instantiation)
source/adios2/toolkit/format/bp/BPSerializer.inl:58:27:
note: previous template specialization is here
inline void BPSerializer::PutPayloadInBuffer(
```https://code.ornl.gov/pnb/ADIOS2/-/issues/1806ADIOS 2.5.0 Warning: unsigned <0 is always false2019-10-07T14:09:10ZPodhorszki, NorbertADIOS 2.5.0 Warning: unsigned <0 is always false*Created by: ax3l*
In ADIOS 2.5.0:
```
source/adios2/toolkit/format/bp/BPBase.cpp:135:42:
error: comparison of unsigned expression < 0 is always false
[-Wtautological-compare]
(m_Parameters.StatsLevel < 0 || m_Parameters.StatsLevel...*Created by: ax3l*
In ADIOS 2.5.0:
```
source/adios2/toolkit/format/bp/BPBase.cpp:135:42:
error: comparison of unsigned expression < 0 is always false
[-Wtautological-compare]
(m_Parameters.StatsLevel < 0 || m_Parameters.StatsLevel > 5))
```https://code.ornl.gov/pnb/ADIOS2/-/issues/1804Build Error in ADIOS 2.5.0 on OSX: JsonWorker2019-10-04T03:27:14ZPodhorszki, NorbertBuild Error in ADIOS 2.5.0 on OSX: JsonWorker*Created by: ax3l*
Reference: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=76704 in https://github.com/conda-forge/adios2-feedstock/pull/8
Compiler: Clang 4.0.1 on Apple-Darwin13.4.0
CMake feature matr...*Created by: ax3l*
Reference: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=76704 in https://github.com/conda-forge/adios2-feedstock/pull/8
Compiler: Clang 4.0.1 on Apple-Darwin13.4.0
CMake feature matrix:
```
ADIOS2 build configuration:
ADIOS Version: 2.5.0
C++ Compiler : Clang 4.0.1
/usr/local/miniconda/conda-bld/adios2_1569893413423/_build_env/bin/x86_64-apple-darwin13.4.0-clang++
Installation prefix: /usr/local/miniconda/conda-bld/adios2_1569893413423/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh
bin: bin
lib: lib
include: include
cmake: lib/cmake/adios2
python: lib/python3.6/site-packages
Features:
Library Type: shared
Build Type: Release
Testing: OFF
Examples: OFF
Build Options:
Blosc : ON
BZip2 : OFF
ZFP : OFF
SZ : OFF
MGARD : OFF
PNG : ON
MPI : ON
DataMan : ON
Table : ON
SSC : ON
SST : ON
DataSpaces: OFF
ZeroMQ : ON
HDF5 : ON
Python : ON
Fortran : OFF
SysVShMem: ON
Profiling: ON
Endian_Reverse: OFF
RDMA Transport for Staging: Unconfigured
```
Compile error:
```
/usr/local/miniconda/conda-bld/adios2_1569893413423/work/source/adios2/toolkit/query/JsonWorker.cpp:128:37: error: call to member function 'compare' is ambiguous
if (m_SourceReader->m_IO.m_Name.compare(ioName) != 0)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
/usr/local/miniconda/conda-bld/adios2_1569893413423/_build_env/bin/../include/c++/v1/string:1189:9: note: candidate function
int compare(const basic_string& __str) const _NOEXCEPT;
^
/usr/local/miniconda/conda-bld/adios2_1569893413423/_build_env/bin/../include/c++/v1/string:1191:9: note: candidate function
int compare(__self_view __sv) const _NOEXCEPT;
^
/usr/local/miniconda/conda-bld/adios2_1569893413423/_build_env/bin/../include/c++/v1/string:1205:9: note: candidate function not viable: no known conversion from 'nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer>' to 'const value_type *' (aka 'const char *') for 1st argument
int compare(const value_type* __s) const _NOEXCEPT;
^
/usr/local/miniconda/conda-bld/adios2_1569893413423/_build_env/bin/../include/c++/v1/string:1204:9: note: candidate function template not viable: requires at least 4 arguments, but 1 was provided
compare(size_type __pos1, size_type __n1, const _Tp& __t, size_type __pos2, size_type __n2=npos) const;
^
/usr/local/miniconda/conda-bld/adios2_1569893413423/_build_env/bin/../include/c++/v1/string:1193:9: note: candidate function not viable: requires 3 arguments, but 1 was provided
int compare(size_type __pos1, size_type __n1, __self_view __sv) const;
^
/usr/local/miniconda/conda-bld/adios2_1569893413423/_build_env/bin/../include/c++/v1/string:1195:9: note: candidate function not viable: requires 3 arguments, but 1 was provided
int compare(size_type __pos1, size_type __n1, const basic_string& __str) const;
^
/usr/local/miniconda/conda-bld/adios2_1569893413423/_build_env/bin/../include/c++/v1/string:1206:9: note: candidate function not viable: requires 3 arguments, but 1 was provided
int compare(size_type __pos1, size_type __n1, const value_type* __s) const;
^
/usr/local/miniconda/conda-bld/adios2_1569893413423/_build_env/bin/../include/c++/v1/string:1207:9: note: candidate function not viable: requires 4 arguments, but 1 was provided
int compare(size_type __pos1, size_type __n1, const value_type* __s, size_type __n2) const;
^
/usr/local/miniconda/conda-bld/adios2_1569893413423/_build_env/bin/../include/c++/v1/string:1196:9: note: candidate function not viable: requires at least 4 arguments, but 1 was provided
int compare(size_type __pos1, size_type __n1, const basic_string& __str, size_type __pos2, size_type __n2=npos) const;
^
1 error generated.
```https://code.ornl.gov/pnb/ADIOS2/-/issues/1802SZ compression of 3D single precision data2020-03-03T20:59:11ZPodhorszki, NorbertSZ compression of 3D single precision data*Created by: tleger26*
I’m trying to use SZ through ADIOS2 to compress some single precision data and have run into an odd issue. When passing the data in 3D matrix form, the decompressed data for some portions of the domain are signif...*Created by: tleger26*
I’m trying to use SZ through ADIOS2 to compress some single precision data and have run into an odd issue. When passing the data in 3D matrix form, the decompressed data for some portions of the domain are significantly (orders of magnitude) different from the original data being compressed, regardless of the error bound mode or level specified. If I convert the data into a vector (1D array) first and then pass it into ADIOS2, then the SZ compression works as expected. I’ve also tried using ZFP 0.5.5 and it works as expected for data in both 3D matrix and vector form. I’ve tried version 2.4.0 and 2.5.0 of ADIOS2, along with SZ 2.1.5.3, 2.1.6.0, and 2.1.7.0, all with the same results. Has anyone else tried and/or experienced this behaviour?https://code.ornl.gov/pnb/ADIOS2/-/issues/1800ZFP not found in 2.5.02019-10-02T06:29:44ZPodhorszki, NorbertZFP not found in 2.5.0*Created by: ax3l*
Hi,
within the release 2.4.0 to 2.5.0, the install of ADIOS2 cannot find zfp anymore.
Related to https://github.com/spack/spack/pull/12987
Probably caused by #1784
With Spack, ZFP 0.5.1-0.5.4 provided and AD...*Created by: ax3l*
Hi,
within the release 2.4.0 to 2.5.0, the install of ADIOS2 cannot find zfp anymore.
Related to https://github.com/spack/spack/pull/12987
Probably caused by #1784
With Spack, ZFP 0.5.1-0.5.4 provided and ADIOS 2.5.0:
```
1 error found in build log:
15 -- Detecting CXX compile features
16 -- Detecting CXX compile features - done
17 -- Found Blosc: /home/axel/src/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-8.3.0/c-blos
c-1.17.0-q5lqzezea6b7b5v4lt2aogurt3k6y3w3/lib/libblosc.so (found suitable version "1.17.0
", minimum required is "1.7")
18 -- Found BZip2: /home/axel/src/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-8.3.0/bzip2-
1.0.8-y3jwkzygasvmix7wsxsp7ughqon54dl5/lib/libbz2.so (found version "1.0.8")
19 -- Looking for BZ2_bzCompressInit
20 -- Looking for BZ2_bzCompressInit - found
>> 21 CMake Error at cmake/DetectOptions.cmake:36 (find_package):
22 Could not find a package configuration file provided by "zfp" (requested
23 version 0.5.1) with any of the following names:
24
25 zfpConfig.cmake
26 zfp-config.cmake
27
```https://code.ornl.gov/pnb/ADIOS2/-/issues/1794cmake not checking for python3-dev dependency2019-09-30T18:09:55ZPodhorszki, Norbertcmake not checking for python3-dev dependency*Created by: williamfgc*
@chuckatkins not sure if you can reproduce, I am not doing anything fancy.
System: Ubuntu 18
CMake version: 3.15.3
```
cmake ../ADIOS2
-- The C compiler identification is GNU 7.4.0
-- The CXX compiler id...*Created by: williamfgc*
@chuckatkins not sure if you can reproduce, I am not doing anything fancy.
System: Ubuntu 18
CMake version: 3.15.3
```
cmake ../ADIOS2
-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find Blosc (missing: BLOSC_LIBRARY BLOSC_INCLUDE_DIR) (Required is at least version "1.7")
-- Could NOT find BZip2 (missing: BZIP2_LIBRARIES BZIP2_INCLUDE_DIR)
-- Could NOT find zfp (missing: zfp_DIR)
-- Could NOT find SZ (missing: SZ_LIBRARY ZSTD_LIBRARY SZ_INCLUDE_DIR)
-- Could NOT find MGARD (missing: MGARD_LIBRARY BLOSC_LIBRARY MGARD_INCLUDE_DIR)
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
-- Could NOT find PNG (missing: PNG_LIBRARY PNG_PNG_INCLUDE_DIR) (Required is at least version "1.6.0")
-- Looking for a Fortran compiler
-- Looking for a Fortran compiler - /usr/bin/f95
-- The Fortran compiler identification is GNU 7.4.0
-- Check for working Fortran compiler: /usr/bin/f95
-- Check for working Fortran compiler: /usr/bin/f95 -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /usr/bin/f95 supports Fortran 90
-- Checking whether /usr/bin/f95 supports Fortran 90 -- yes
-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found version "3.1")
-- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so (found version "3.1")
-- Found MPI_Fortran: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_usempif08.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: C Fortran CXX
-- Found ZeroMQ: /usr/lib/x86_64-linux-gnu/libzmq.so (found suitable version "4.2.5", minimum required is "4.1")
-- Could NOT find DataSpaces (missing: DATASPACES_VERSION DATASPACES_INCLUDE_DIR DATASPACES_LIBRARIES DSPACES_LIBRARY) (Required is at least version "1.8")
-- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS C) (found version "")
-- Found PythonInterp: /usr/bin/python3 (found version "3.6.8")
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so (found version "3.6.8")
-- Found PythonModule_numpy: /usr/lib/python3/dist-packages/numpy
-- Found PythonModule_mpi4py: /usr/lib/python3/dist-packages/mpi4py
-- Found PythonFull: /usr/bin/python3 found components: Interp Libs numpy mpi4py
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Checking for module 'libfabric'
-- No package 'libfabric' found
-- Could NOT find LIBFABRIC (missing: LIBFABRIC_LIBRARIES) (Required is at least version "1.6")
-- Could NOT find NVStream (missing: NVSTREAM_LIBRARY NVSTREAM_INCLUDE_DIR)
-- NVSTREAM_FOUND is FALSE, LIB is NVSTREAM_LIBRARY-NOTFOUND
-- Looking for shmget
-- Looking for shmget - found
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- ADIOS2 ThirdParty: Configuring KWSys
-- Checking whether header cstdio is available
-- Checking whether header cstdio is available - yes
-- Checking for Large File Support
-- Checking for Large File Support - yes
-- Checking whether C++ compiler has 'long long'
-- Checking whether C++ compiler has 'long long' - yes
-- Checking whether C++ compiler has '__int64'
-- Checking whether C++ compiler has '__int64' - no
-- Checking whether wstring is available
-- Checking whether wstring is available - yes
-- Checking whether C compiler has ptrdiff_t in stddef.h
-- Checking whether C compiler has ptrdiff_t in stddef.h - yes
-- Checking whether C compiler has ssize_t in unistd.h
-- Checking whether C compiler has ssize_t in unistd.h - yes
-- Checking whether CXX compiler has setenv
-- Checking whether CXX compiler has setenv - yes
-- Checking whether CXX compiler has unsetenv
-- Checking whether CXX compiler has unsetenv - yes
-- Checking whether CXX compiler has environ in stdlib.h
-- Checking whether CXX compiler has environ in stdlib.h - no
-- Checking whether CXX compiler has utimes
-- Checking whether CXX compiler has utimes - yes
-- Checking whether CXX compiler has utimensat
-- Checking whether CXX compiler has utimensat - yes
-- Checking whether CXX compiler struct stat has st_mtim member
-- Checking whether CXX compiler struct stat has st_mtim member - yes
-- Checking whether CXX compiler struct stat has st_mtimespec member
-- Checking whether CXX compiler struct stat has st_mtimespec member - no
-- Checking whether <ext/stdio_filebuf.h> is available
-- Checking whether <ext/stdio_filebuf.h> is available - yes
-- ADIOS2 ThirdParty: Configuring GTest
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- ADIOS2 ThirdParty: Configuring pybind11
CMake Error at thirdparty/pybind11/pybind11/tools/FindPythonLibsNew.cmake:95 (message):
Python config failure:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: cannot import name 'sysconfig'
Call Stack (most recent call first):
thirdparty/pybind11/pybind11/tools/pybind11Tools.cmake:16 (find_package)
thirdparty/pybind11/pybind11/CMakeLists.txt:33 (include)
-- Configuring incomplete, errors occurred!
See also "/home/wgodoy/workspace/ADIOS2-build/CMakeFiles/CMakeOutput.log".
See also "/home/wgodoy/workspace/ADIOS2-build/CMakeFiles/CMakeError.log".
```
https://code.ornl.gov/pnb/ADIOS2/-/issues/1780Installation on CORI2020-03-03T21:08:42ZPodhorszki, NorbertInstallation on CORI*Created by: abidmalikwaterloo*
I am trying to install ADIOS2 on Cori. I have all the dependencies for the ADIOS2 loaded and PATH and LD_LIBRARY_PATH updated accordingly
```
***************************
module load cuda
module load ...*Created by: abidmalikwaterloo*
I am trying to install ADIOS2 on Cori. I have all the dependencies for the ADIOS2 loaded and PATH and LD_LIBRARY_PATH updated accordingly
```
***************************
module load cuda
module load gcc/8.1.1-openacc-gcc-8-branch-20190215
module load zlib/1.2.11
module load bzip2/1.0.6
module load cmake/3.14.4
module load module load openmpi/3.1.4
module unload darshan
export PATH=$PATH:/global/homes/a/amalik/CODAR/libzmq/install/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/global/homes/a/amalik/CODAR/libzmq/install/lib
export PATH=$PATH:/global/homes/a/amalik/CODAR/libfabric-1.7.0/install/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/global/homes/a/amalik/CODAR/libfabric-1.7.0/install/lib
export PATH=$PATH:/global/homes/a/amalik/CODAR/adios2/sz-1.4.13.0/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/global/homes/a/amalik/CODAR/adios2/sz-1.4.13.0/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/global/homes/a/amalik/CODAR/c-blosc/lib
****************************
```
I tried to build ADIOS2
```
cmake -DCMAKE_INSTALL_PREFIX=/global/homes/a/amalik/CODAR/adios2/ADIOS2/ -DCMAKE_C_COMPILER=/usr/common/software/gcc/8.1.1-openacc-gcc-8-branch-20190215/bin/gcc -DCMAKE_CXX_COMPILER=/usr/common/software/gcc/8.1.1-openacc-gcc-8-branch-20190215/bin/g++ -DCMAKE_FORTRAN_COMPILER=/usr/common/software/gcc/8.1.1-openacc-gcc-8-branch-20190215/bin/gfortran -DADIOS2_USE_MPI=ON -DADIOS2_USE_HDF5=OFF -DADIOS2_USE_ZeroMQ=ON -DADIOS2_USE_Fortran=ON -DADIOS2_USE_Python=OFF -DADIOS2_USE_SST=ON -DADIOS2_USE_BZip2=ON -DADIOS2_USE_ZFP=OFF -DADIOS2_USE_DataMan=ON -DADIOS2_USE_Profiling=OFF -DADIOS2_USE_Blosc=ON -DBUILD_SHARED_LIBS=ON -DLIBFABRIC_ROOT=/global/homes/a/amalik/CODAR/libfabric-1.7.0 -DSZ_ROOT=/global/homes/a/amalik/CODAR/adios2/sz-1.4.13.0 -DSZ_INCLUDE_DIR=/global/homes/a/amalik/CODAR/adios2/sz-1.4.13.0/include -DSZ_LIBRARY=/global/homes/a/amalik/CODAR/adios2/sz-1.4.13.0/lib/libSZ.a -DBLOSC_INCLUDE_DIR=/global/homes/a/amalik/CODAR/c-blosc/include -DBLOSC_LIBRARY=/global/homes/a/amalik/CODAR/c-blosc/lib/libblosc.a -DADIOS2_BUILD_TESTING=ON -DCMAKE_BUILD_TYPE=Release -DMPIEXEC_MAX_NUMPROCS=16 ..
``
but I am getting the following:
``
-- Could NOT find SZ (missing: ZSTD_LIBRARY)
-- Could NOT find MGARD (missing: MGARD_LIBRARY MGARD_INCLUDE_DIR)
CMake Error at /global/common/sw/cray/cnl7/haswell/cmake/3.14.4/gcc/8.2.0/2hef55n/share/cmake-3.14/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
Could NOT find ZeroMQ (missing: ZeroMQ_LIBRARY ZeroMQ_INCLUDE_DIR)
(Required is at least version "4.1")
Call Stack (most recent call first):
/global/common/sw/cray/cnl7/haswell/cmake/3.14.4/gcc/8.2.0/2hef55n/share/cmake-3.14/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
```
I am taking care of SZ library and ZeroMQ library. But still getting the error. I am following the steps for the Summit for ADIOS2. Do you have anything/installation script for Cori?https://code.ornl.gov/pnb/ADIOS2/-/issues/1766Make SST fail better2019-10-15T23:45:23ZEisenhauer, GregMake SST fail betterSST needs to detect underlying network failure events and fail more cleanly / deliver better diagnostics.
The most important is likely when we're running out of file descriptors, but we should also notice bind failures and other things ...SST needs to detect underlying network failure events and fail more cleanly / deliver better diagnostics.
The most important is likely when we're running out of file descriptors, but we should also notice bind failures and other things happening deep inside the communication system.Eisenhauer, GregEisenhauer, Greghttps://code.ornl.gov/pnb/ADIOS2/-/issues/1765POSIX Transport creates file on Open() without Flush() 2019-09-19T20:20:49ZPodhorszki, NorbertPOSIX Transport creates file on Open() without Flush() *Created by: philip-davis*
The open() system call can create the file on disk immediately. To demonstrate this, log into two different login nodes on, for example, Cori and run this code on one of them:
```
#include <sys/types.h>
#...*Created by: philip-davis*
The open() system call can create the file on disk immediately. To demonstrate this, log into two different login nodes on, for example, Cori and run this code on one of them:
```
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
int main()
{
int fd;
struct stat st;
open("abc.txt", O_WRONLY | O_CREAT | O_TRUNC, 0666);
sleep(100);
return(0);
}
```
abc.txt will be visible on the other login node immediately, both for NFS and Lustre file systems.
The reason I bring this up is that this open() call[ is the same as the one used by FilePOSIX::Open() when openMode is Mode::Write.](https://github.com/ornladios/ADIOS2/blob/7da36fbec480d531119fadd7e655eb730e913878/source/adios2/toolkit/transport/file/FilePOSIX.cpp#L51) If I'm reading the code correctly, FilePOSIX is the default Transport library, including for BP4. This matters for the File Metadata Index. Even though [we wait until after writing the header to do a flush](https://github.com/ornladios/ADIOS2/blob/7da36fbec480d531119fadd7e655eb730e913878/source/adios2/engine/bp4/BP4Writer.cpp#L215-L230), that flush is a no-op for POSIX, and the zero-length file has already been created.
This is, in turn, a problem because the reader could see the zero-length metadata index file, and then interpreting the header as a new timestep. I expect ParseMetadata would fail noisily in this case, but I haven't looked closely enough at the code to know that for sure.https://code.ornl.gov/pnb/ADIOS2/-/issues/1760Current master is broken when configuring with SZ2019-09-19T02:30:50ZPodhorszki, NorbertCurrent master is broken when configuring with SZ*Created by: williamfgc*
the `fortran_add_test_helper` function doesn't handle the BPWriteReadSZ* tests in the operation subdirectory. The following message appears:
```
Features:
Library Type: shared
Build Type: Debug
...*Created by: williamfgc*
the `fortran_add_test_helper` function doesn't handle the BPWriteReadSZ* tests in the operation subdirectory. The following message appears:
```
Features:
Library Type: shared
Build Type: Debug
Testing: ON
Build Options:
Blosc : ON
BZip2 : ON
ZFP : OFF
SZ : ON
MGARD : ON
PNG : ON
MPI : ON
DataMan : OFF
Table : OFF
SSC : OFF
SST : ON
DataSpaces: OFF
ZeroMQ : ON
HDF5 : ON
Python : ON
Fortran : ON
SysVShMem: ON
Profiling: ON
Endian_Reverse: OFF
RDMA Transport for Staging: Unconfigured
-- Configuring done
CMake Error at testing/adios2/bindings/fortran/CMakeLists.txt:9 (add_executable):
Cannot find source file:
TestBPWriteReadSZ3D.F90
Tried extensions .c .C .c++ .cc .cpp .cxx .cu .m .M .mm .h .hh .h++ .hm
.hpp .hxx .in .txx
Call Stack (most recent call first):
testing/adios2/bindings/fortran/CMakeLists.txt:76 (fortran_add_test_helper)
CMake Error at testing/adios2/bindings/fortran/CMakeLists.txt:9 (add_executable):
Cannot find source file:
TestBPWriteReadSZ2D.F90
Tried extensions .c .C .c++ .cc .cpp .cxx .cu .m .M .mm .h .hh .h++ .hm
.hpp .hxx .in .txx
Call Stack (most recent call first):
testing/adios2/bindings/fortran/CMakeLists.txt:75 (fortran_add_test_helper)
CMake Error at testing/adios2/bindings/fortran/CMakeLists.txt:9 (add_executable):
No SOURCES given to target: Test.Bindings.Fortran.BPWriteReadSZ3D
Call Stack (most recent call first):
testing/adios2/bindings/fortran/CMakeLists.txt:76 (fortran_add_test_helper)
CMake Error at testing/adios2/bindings/fortran/CMakeLists.txt:9 (add_executable):
No SOURCES given to target: Test.Bindings.Fortran.BPWriteReadSZ2D
Call Stack (most recent call first):
testing/adios2/bindings/fortran/CMakeLists.txt:75 (fortran_add_test_helper)
CMake Generate step failed. Build files cannot be regenerated correctly.
```
2.5.0 Production Release