ADIOS2 issueshttps://code.ornl.gov/pnb/ADIOS2/-/issues2020-03-13T18:38:38Zhttps://code.ornl.gov/pnb/ADIOS2/-/issues/2039Add the building of Docker images to CI2020-03-13T18:38:38ZPodhorszki, NorbertAdd the building of Docker images to CI*Created by: HannoSpreeuw*
I could add the building of Docker images to your CI system, if you would like that.
This is in relation to issue #2038, so to prevent this from occurring again. *Created by: HannoSpreeuw*
I could add the building of Docker images to your CI system, if you would like that.
This is in relation to issue #2038, so to prevent this from occurring again. https://code.ornl.gov/pnb/ADIOS2/-/issues/2038Dockerfile not working2020-03-13T18:38:21ZPodhorszki, NorbertDockerfile not working*Created by: HannoSpreeuw*
**Describe the bug**
These commands from bash:
`wget https://raw.githubusercontent.com/ornladios/ADIOS2/master/scripts/docker/images/ubuntu18.04/Dockerfile`
`docker build -t adios2_im .`
output
```
Clo...*Created by: HannoSpreeuw*
**Describe the bug**
These commands from bash:
`wget https://raw.githubusercontent.com/ornladios/ADIOS2/master/scripts/docker/images/ubuntu18.04/Dockerfile`
`docker build -t adios2_im .`
output
```
Cloning into 'ADIOS2'...
CMake Error at CMakeLists.txt:6 (cmake_minimum_required):
CMake 3.12 or higher is required. You are running version 3.10.2
```
**Expected behavior**
A succesful build of the Ubuntu Bionic Docker image with Adios2 installed.
**Desktop (please complete the following information):**
This is on Manjaro Linux 19.0.2.
Docker version 19.03.6-ce, build 369ce74a3c
**Additional context**
Can fix this by using
```
# FROM ubuntu:18.04
FROM ubuntu:19.10
```
Also, the paths need to be fixed or very few Unix commands can be issued from within a shell in the container:
```
ENV PYTHONPATH="/usr/local/lib/python3.6/site-packages/:/opt/adios2/lib/python3.6/site-packages:${PYTHONPATH}"
ENV PATH="/opt/adios2/bin:${PATH}"
ENV LD_LIBRARY_PATH="/usr/local/lib:/opt/zfp/lib:/opt/sz/lib:/opt/adios2/lib:${LD_LIBRARY_PATH}"
```
E.g. `ls` will not work without the addition of `:${PATH}`.
Please let me know if you will accept a PR for these changes.
https://code.ornl.gov/pnb/ADIOS2/-/issues/2033segfault running adios_reorganize to convert to hdf52020-03-11T17:02:51ZPodhorszki, Norbertsegfault running adios_reorganize to convert to hdf5*Created by: jhidding*
**Describe the bug**
Running the example line: `adios_reorganize sim.bp sim.h5 BPFile "" HDF5 ""` from the documentation creates a segmentation fault. Both with and without the `mpirun` bit.
**To Reproduce**
...*Created by: jhidding*
**Describe the bug**
Running the example line: `adios_reorganize sim.bp sim.h5 BPFile "" HDF5 ""` from the documentation creates a segmentation fault. Both with and without the `mpirun` bit.
**To Reproduce**
1. have a .bp file (in this case created with the OpenFOAM Adios writer)
2. run `adios_reorganize`, doesn't matter which output back-end is used
**Expected behavior**
no segfault
**Desktop (please complete the following information):**
- OS/Platform: Fedora 31
- Build: gcc 9.2.1
**Additional context**
The same bp file seems to read fine using the Python module.
printed backtrace:
```
[ESLT0107:12052:0:12052] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
0 /lib64/libucs.so.0(+0x1b25f) [0x7f40e808525f]
1 /lib64/libucs.so.0(+0x1b42a) [0x7f40e808542a]
2 /lib64/libc.so.6(cfree+0x52) [0x7f40ea63b7e2]
3 /home/johannes/.local/lib/libadios2.so.2(_ZN6adios26format15BP3Deserializer12PostDataReadIdEEvRNS_4core8VariableIT_EERNS6_4InfoERKNS_6helper16SubStreamBoxInfoEbm+0x265) [0x7f40eb02e945]
4 /home/johannes/.local/lib/libadios2.so.2(_ZN6adios24core6engine9BP3Reader18ReadVariableBlocksIdEEvRNS0_8VariableIT_EE+0x19e) [0x7f40eaf3e84e]
5 /home/johannes/.local/lib/libadios2.so.2(_ZN6adios24core6engine9BP3Reader9DoGetSyncERNS0_8VariableIdEEPd+0x86) [0x7f40eaf378a6]
6 /home/johannes/.local/lib/libadios2.so.2(_ZN6adios24core6Engine3GetIdEEvRNS0_8VariableIT_EEPS4_NS_4ModeE+0x19b) [0x7f40eae9215b]
7 /home/johannes/.local/lib/libadios2.so.2(_ZN6adios24core6Engine3GetIdEEvRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPT_NS_4ModeE+0x64) [0x7f40eae922e4]
8 adios_reorganize() [0x409232]
9 adios_reorganize() [0x40cfd7]
10 adios_reorganize() [0x405f2a]
11 /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f40ea5d61a3]
12 adios_reorganize() [0x4061be]
===================
```
**follow up**
Let me know if you need me to repeat this with ADIOS2 built with build-type set to Debughttps://code.ornl.gov/pnb/ADIOS2/-/issues/2040Cannot run bp4dbg2020-03-16T11:54:05ZPodhorszki, NorbertCannot run bp4dbg*Created by: pnorbert*
**Describe the bug**
After installing adios2, I cannot run the bp4dbg utility because it fails with this error:
```
$ /opt/adios2/bin/bp4dbg localArray.bp/md.0
=================================================...*Created by: pnorbert*
**Describe the bug**
After installing adios2, I cannot run the bp4dbg utility because it fails with this error:
```
$ /opt/adios2/bin/bp4dbg localArray.bp/md.0
========================================================
Metadata File: localArray.bp/md.0
========================================================
Traceback (most recent call last):
File "/opt/adios2/bin/bp4dbg", line 105, in <module>
DumpMetadataFiles(args)
File "/opt/adios2/bin/bp4dbg", line 81, in DumpMetadataFiles
DumpMetaData(fname)
File "/opt/adios2/lib/python3/dist-packages/adios2/bp4dbg/metadata.py", line 607, in DumpMetaData
status = bp4dbg_utils.ReadHeader(f, fileSize, "Metadata")
NameError: name 'bp4dbg_utils' is not defined
```
**Additional context**
The bp4dbg scripts use bp4dbg_utils._FunctionName()_ to call the utility functions but they import them as
`from .utils import *`
The function calls cannot just be simply renamed to _FunctionName()_ because there will be clashes with other functions:
```
Traceback (most recent call last):
File "/opt/adios2/bin/bp4dbg", line 105, in <module>
DumpMetadataFiles(args)
File "/opt/adios2/bin/bp4dbg", line 81, in DumpMetadataFiles
DumpMetaData(fname)
File "/opt/adios2/lib/python3/dist-packages/adios2/bp4dbg/metadata.py", line 607, in DumpMetaData
status = ReadHeader(f, fileSize, "Metadata")
TypeError: ReadHeader() takes 1 positional argument but 3 were given
```
https://code.ornl.gov/pnb/ADIOS2/-/issues/2047Python binding issues2020-03-17T22:42:07ZPodhorszki, NorbertPython binding issues*Created by: JasonRuonanWang*
I get the following error when I try to run gray-scott. This was on Mac 10.15.3 with Clang 11.0.0, and Python 3.7.7. Python was installed through homebrew and is the only Python3 installation on the machine...*Created by: JasonRuonanWang*
I get the following error when I try to run gray-scott. This was on Mac 10.15.3 with Clang 11.0.0, and Python 3.7.7. Python was installed through homebrew and is the only Python3 installation on the machine.
```
jason@iMac ~/w/adiosvm/Tutorial/gray-scott master mpirun --oversubscribe -n 4 build/gray-scott simulation/settings-staging.json : -n 1 build/pdf_calc gs.bp pdf.bp 100 : -n 1 python3 plot/pdfplot.py -i pdf.bp
Traceback (most recent call last):
File "plot/pdfplot.py", line 3, in <module>
import adios2
File "/usr/local/lib/python3.7/site-packages/adios2/__init__.py", line 1, in <module>
from .adios2 import *
ImportError: dlopen(/usr/local/lib/python3.7/site-packages/adios2/adios2.so, 2): Symbol not found: __PyThreadState_Current
Referenced from: /usr/local/lib/python3.7/site-packages/adios2/adios2.so
Expected in: flat namespace
in /usr/local/lib/python3.7/site-packages/adios2/adios2.so
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[50203,1],5]
Exit code: 1
--------------------------------------------------------------------------
```
The following error is a compile error I got from compiling ADIOS2 with python3 on Ubuntu 19.10. It was a brand new VM installation, with everything (gcc 9.2.1 and libpython3-dev) installed through apt. Cmake in this case is not able to find the python3 installation automatically. I had to specify DPython_LIBRARIES and DPython_INCLUDE_DIR, but it still didn't work.
```
jason@vm ~/build/adios2 cmake ~/d/a2 -DPython_LIBRARIES=/usr/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7.so -DPython_INCLUDE_DIR=/usr/include/python3.7m
-- Could NOT find Blosc (missing: BLOSC_LIBRARY BLOSC_INCLUDE_DIR) (Required is at least version "1.7")
-- Could NOT find BZip2 (missing: BZIP2_LIBRARIES BZIP2_INCLUDE_DIR)
-- Could NOT find ZFP (missing: ZFP_DIR)
-- Could NOT find SZ (missing: SZ_LIBRARY ZLIB_LIBRARY ZSTD_LIBRARY SZ_INCLUDE_DIR)
-- Could NOT find MGARD (missing: MGARD_LIBRARY ZLIB_LIBRARY BLOSC_LIBRARY MGARD_INCLUDE_DIR)
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
-- Could NOT find PNG (missing: PNG_LIBRARY PNG_PNG_INCLUDE_DIR) (Required is at least version "1.6.0")
-- Found MPI: TRUE (found version "3.1") found components: C Fortran CXX
-- Could NOT find ZeroMQ (missing: ZeroMQ_LIBRARY ZeroMQ_INCLUDE_DIR) (Required is at least version "4.1")
-- Could NOT find DataSpaces (missing: DATASPACES_VERSION DATASPACES_INCLUDE_DIR DATASPACES_LIBRARIES DSPACES_LIBRARY) (Required is at least version "1.8")
-- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS C) (found version "")
CMake Error at cmake/upstream/FindPython/Support.cmake:2068 (if):
if given arguments:
"_Python_INC_VERSION" "VERSION_EQUAL"
Unknown arguments specified
Call Stack (most recent call first):
cmake/upstream/FindPython.cmake:326 (include)
cmake/FindPython.cmake:8 (include)
cmake/DetectOptions.cmake:213 (find_package)
CMakeLists.txt:130 (include)
-- Configuring incomplete, errors occurred!
See also "/home/jason/build/adios2/CMakeFiles/CMakeOutput.log".
See also "/home/jason/build/adios2/CMakeFiles/CMakeError.log".
✘ jason@vm ~/build/adios2
```
https://code.ornl.gov/pnb/ADIOS2/-/issues/2042Append mode error2020-03-16T20:11:14ZPodhorszki, NorbertAppend mode error*Created by: jychoi-hpc*
**Describe the bug**
I am working on Pixie3D to write restart files with appending mode without any success. I wrote a test case to reproduce the error.
In the test code, I am trying to write a file with ap...*Created by: jychoi-hpc*
**Describe the bug**
I am working on Pixie3D to write restart files with appending mode without any success. I wrote a test case to reproduce the error.
In the test code, I am trying to write a file with appending mode but getting an error.
When writing with `adios2_mode_write`, it works. But, a file with `adios2_mode_append` is corrupted and cannot do `bpls` (no error during the runtime though).
**To Reproduce**
Here is my test code (writer.F90)
```
program writer
use mpi
use adios2
implicit none
integer(kind=8), dimension(1) :: shape_dims, start_dims, count_dims
real, dimension(:), allocatable :: myArray
integer :: inx, irank, isize, ierr, i, var1_type
type(adios2_adios) :: adios
type(adios2_io) :: io
type(adios2_variable) :: var, var1
type(adios2_attribute) :: attr
type(adios2_engine) :: engine
character(len=:), allocatable :: var1_name
integer :: comm
integer :: istatus
real*8 :: totalpe;
CHARACTER(len=32) :: arg
integer :: mode
logical :: do_read = .false.
logical :: do_append = .false.
do i = 1, iargc()
call getarg(i, arg)
if (trim(arg)=='-read') do_read = .true.
if (trim(arg)=='-append') do_append = .true.
end do
! Launch MPI
comm = MPI_COMM_WORLD
call MPI_Init(ierr)
call MPI_Comm_rank(comm, irank, ierr)
call MPI_Comm_size(comm, isize, ierr)
! Application variables
inx = 10
allocate( myArray(inx) )
do i=1,inx
myArray(i) = i-1
end do
! Variable dimensions
shape_dims(1) = isize * inx
start_dims(1) = irank * inx
count_dims(1) = inx
if (do_append) then
print *, 'Mode: append'
mode = adios2_mode_append
else
print *, 'Mode: write'
mode = adios2_mode_write
endif
! Create adios handler passing the communicator, debug mode and error flag
call adios2_init(adios, comm, adios2_debug_mode_on, ierr)
! Declare an IO process configuration inside adios
call adios2_declare_io(io, adios, "group1", ierr)
! Defines a variable to be written in bp format
! call adios2_define_variable(var, io, 'totalpe', adios2_type_dp, ierr)
call adios2_define_variable(var1, io, "totalpe", adios2_type_dp, 1, &
(/ 1_8 /), (/ 0_8 /), (/ 1_8 /), &
adios2_constant_dims, ierr)
call adios2_define_variable(var1, io, "arr1", adios2_type_real, 1, &
shape_dims, start_dims, count_dims, &
adios2_constant_dims, ierr)
call adios2_define_attribute(attr, io, "attr", "this is attr", ierr)
! Open myVector_f.bp in write mode, this launches an engine
call adios2_open(engine, io, "out.bp", mode, comm, ierr)
print *, 'adios2_open:ierr', ierr
do i=1,4
call adios2_begin_step(engine, adios2_step_mode_append, ierr)
print *, 'adios2_begin_step:ierr', ierr
! Put myArray contents to bp buffer, based on var1 metadata
call adios2_put(engine, 'totalpe', 777D0, ierr)
call adios2_put(engine, var1, myArray, adios2_mode_sync, ierr)
! Closes engine and deallocates it, becomes unreachable
call adios2_end_step(engine, ierr)
print *, 'adios2_end_step:ierr', ierr
enddo
call adios2_close(engine, ierr)
print *, 'adios2_close:ierr', ierr
! Deallocates adios and calls its destructor
call adios2_finalize(adios, ierr)
if( allocated(myArray) ) deallocate(myArray)
if( allocated(var1_name) ) deallocate(var1_name)
call MPI_Finalize(ierr)
end program writer
```
Makefile
```
FC=mpif90
ADIOS2_DIR=/opt/adios2
ADIOS2_INC=$(shell $(ADIOS2_DIR)/bin/adios2-config --fortran-flags)
ADIOS2_LIB=$(shell $(ADIOS2_DIR)/bin/adios2-config --fortran-libs)
BINS=writer
all: $(BINS)
writer: writer.F90
$(FC) -g -ffree-line-length-0 $(ADIOS2_INC) -o $@ $? $(ADIOS2_LIB)
clean:
rm -rf $(BINS) *.o *.dSYM
```
I am running in the following step. First, I create `out.bp` with `adios2_mode_write` mode
```bash
$ ./writer
$ bpls -lva out.bp
File info:
of variables: 2
of attributes: 1
statistics: Min / Max
float arr1 4*{10} = 0 / 9
string attr attr = "this is attr"
double totalpe 4*{1} = 777 / 777
```
it works fine
Then, I am trying to do with `adios2_mode_append`:
```bash
$ ./writer -append
$ bpls -lva out.bp
$ bpls -lva out.bp/
HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 0:
#000: H5F.c line 509 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1400 in H5F__open(): unable to open file
major: File accessibilty
minor: Unable to open file
#002: H5Fint.c line 1700 in H5F_open(): unable to read superblock
major: File accessibilty
minor: Read failed
```
It should be `BP4` file, not HDF5. It looks like the file corrupted.
Any help and advice will be appreciated.
Thanks.
2.6.0 Production Releasehttps://code.ornl.gov/pnb/ADIOS2/-/issues/2044CI: el7-gnu8-openmpi-ohpc-static builder broken on master2020-03-18T13:30:27ZPodhorszki, NorbertCI: el7-gnu8-openmpi-ohpc-static builder broken on master*Created by: bradking*
Cc: @chuckatkins @eisenhauer @JasonRuonanWang
Several PRs are blocked by failures of the `Install.Make.Fortran` test on the `el7-gnu8-openmpi-ohpc-static` CI build.
One can use [this CDash query](https://o...*Created by: bradking*
Cc: @chuckatkins @eisenhauer @JasonRuonanWang
Several PRs are blocked by failures of the `Install.Make.Fortran` test on the `el7-gnu8-openmpi-ohpc-static` CI build.
One can use [this CDash query](https://open.cdash.org/index.php?project=ADIOS&date=2020-03-17&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=el7-gnu8-openmpi-ohpc-static&field2=buildname&compare2=63&value2=master&field3=buildstarttime&compare3=83&value3=2020-02-15&field4=buildstarttime&compare4=84&value4=2020-03-18) to see that the test passed on that build in `master` consistently until 2020-03-12 and has failed consistently ever since.
The first version that failed in the `master` branch builder was commit bada93f0f5fcdcbacf5fb665981f14450aeb2101. That commit merged #2032 where the failure had been seen during CI, but that PR was able to be merged eventually.
https://code.ornl.gov/pnb/ADIOS2/-/issues/2034Doc images not rendering2020-03-11T13:49:39ZPodhorszki, NorbertDoc images not rendering*Created by: NAThompson*
In the docs, instead of images showing from the url, only the url is displayed. See [here](https://adios2.readthedocs.io/en/latest/components/components.html).
Will submit a PR . . .*Created by: NAThompson*
In the docs, instead of images showing from the url, only the url is displayed. See [here](https://adios2.readthedocs.io/en/latest/components/components.html).
Will submit a PR . . .https://code.ornl.gov/pnb/ADIOS2/-/issues/2023Add Engine::BlocksInfo to the python bindings2020-03-06T16:07:42ZPodhorszki, NorbertAdd Engine::BlocksInfo to the python bindings*Created by: isosc*
Engine::BlocksInfo() seems to be missing from the python bindings. This is essential for extracting metadata information to build skel I/O models.
This seems like it is a minor addition, but requires some knowled...*Created by: isosc*
Engine::BlocksInfo() seems to be missing from the python bindings. This is essential for extracting metadata information to build skel I/O models.
This seems like it is a minor addition, but requires some knowledge of the pybind tool.
https://code.ornl.gov/pnb/ADIOS2/-/issues/2020BP4 does not implement the BP3 "async file open"2020-03-04T19:55:23ZEisenhauer, GregBP4 does not implement the BP3 "async file open"The BP3 engine, like ADIOS1, had a feature where one could "pre-Open" a file, which helped protect applications from long delays on machines where a system-level file open might take a long time. Specifically, in BP3, the actual ADIOS2 ...The BP3 engine, like ADIOS1, had a feature where one could "pre-Open" a file, which helped protect applications from long delays on machines where a system-level file open might take a long time. Specifically, in BP3, the actual ADIOS2 Open() call for writers would spawn an asynchronous task for the sytem level open, and avoid use of the system file for the duration of ADIOS Open to ensure that ADIOS Open could return before system-level open() completed. BP3 writer would join the async task before the first write() that actually used that system-level resource, but between the Open and the first write, the application was free to do other things, essentially hiding the latency of system-level open. However, the addition of index files in BP4 changed the nature of Open, and it no longer refrains from touching the system-level resource. This means that in BP4, it is not possible to pre-Open a ADIOS file and hide the latency of the system-level open() call. Instead, Open blocks internally and waits for the system open to succeed so that it can write to the index files. The pre-Open functionality should be implemented in BP4 as it is in BP3. (See discussion in PR #2007.)https://code.ornl.gov/pnb/ADIOS2/-/issues/2018C-Blosc: Support for Large Vars2020-03-06T13:35:49ZPodhorszki, NorbertC-Blosc: Support for Large Vars*Created by: ax3l*
**Describe the bug**
Writing for individual variables with c-blosc (1) that write very large data fails, since c-blosc has a 2Gbyte limit.
**To Reproduce**
Based on #2010, use this diff
```diff
diff --git a/t...*Created by: ax3l*
**Describe the bug**
Writing for individual variables with c-blosc (1) that write very large data fails, since c-blosc has a 2Gbyte limit.
**To Reproduce**
Based on #2010, use this diff
```diff
diff --git a/testing/adios2/performance/largevar/TestLargeVarWrite.cpp b/testing/adios2/performance/largevar/TestLargeVarWrite.cpp
index 59d4ccad..4c1acef6 100644
--- a/testing/adios2/performance/largevar/TestLargeVarWrite.cpp
+++ b/testing/adios2/performance/largevar/TestLargeVarWrite.cpp
@@ -4,6 +4,7 @@
*/
#include <memory>
+#include <random>
#include <stdexcept>
#include <string>
@@ -29,6 +30,11 @@ TEST_F(Write, Plain1D_1B)
std::unique_ptr<float[]> dataBig(new float[Nx]);
+ auto const seed = std::random_device{}();
+ std::default_random_engine gen{ seed };
+ std::uniform_real_distribution< float > dist( 0.0f, 1.0f );
+ std::generate_n( dataBig.get(), Nx, [&]{ return dist( gen ); } );
+
#ifdef ADIOS2_HAVE_MPI
MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank);
MPI_Comm_size(MPI_COMM_WORLD, &mpiSize);
@@ -53,6 +59,17 @@ TEST_F(Write, Plain1D_1B)
auto varBig = io.DefineVariable<float>("varBig", shape, start, count,
adios2::ConstantDims);
+ // add operations
+ adios2::Operator BloscOp =
+ adios.DefineOperator("BloscCompressor", adios2::ops::LosslessBlosc);
+
+ varBig.AddOperation(BloscOp,
+ {{adios2::ops::blosc::key::compressor, "zstd"}});
+ varBig.AddOperation(BloscOp,
+ {{adios2::ops::blosc::key::clevel, "1"}});
+ varBig.AddOperation(BloscOp,
+ {{adios2::ops::blosc::key::doshuffle, "BLOSC_BITSHUFFLE"}});
+ // FIXME how to set the compressor threads?
adios2::Engine writer = io.Open(fname, adios2::Mode::Write);
```
And execute. Repeat with a smaller array size in `Nx`, e.g. divide by `4`.
**Expected behavior**
The size of the variable should not matter for the operation.
**Desktop (please complete the following information):**
- OS/Platform: Ubuntu 18.04
- Build: gcc 7.4.0, cmake 3.16.2
**Additional context**
Using threaded c-blosc compression with arbitrary variable sizes is our [production ADIOS1 workflow](https://arxiv.org/abs/1706.00522) in PIConGPU.
In ADIOS1, we added a custom-header to split up the user-input into chunks not larger than the maximum supported size in blosc: https://github.com/ornladios/ADIOS/pull/135 (This probably also affects other compressors.)
Probably even better, we can also switch directly to [c-blosc2 that has 64bit buffer support](https://github.com/Blosc/c-blosc2/issues/72).
**Following up**
Fixing this should complete the feature request in #1486.
cc @chuckatkins @pnorbert and @psychocoderHPC @franzpoeschel
https://code.ornl.gov/pnb/ADIOS2/-/issues/2015Update Keys of Variables/Attributes2020-03-03T17:33:07ZPodhorszki, NorbertUpdate Keys of Variables/Attributes*Created by: ax3l*
**Why is this feature important?**
We develop the [openPMD-standard](https://github.com/openPMD/openPMD-standard) as a meta-data schema for particle-mesh codes.
As the standard evolves, we can [update the meta-dat...*Created by: ax3l*
**Why is this feature important?**
We develop the [openPMD-standard](https://github.com/openPMD/openPMD-standard) as a meta-data schema for particle-mesh codes.
As the standard evolves, we can [update the meta-data](https://github.com/openPMD/openPMD-updater/) of existing files to a newer version by:
- renaming paths of variables and attributes
- removing (undefining/hiding) variables/attributes
- adding new attributes
This problem will commonly occur when users open "old" openPMD files in an openPMD reader that supports only a newer version of the standard. A quick, in-place meta-data conversion simplifies reader implementations and provides a convenient user-experience.
**What is the potential impact of this feature in the community?**
This allows faster development of data readers of meta-data schemes (only need to implement the newest version of a schema) and a better user experience with forward-updateable meta-data.
**Is your feature request related to a problem? Please describe.**
Related to openPMD:
- https://www.openpmd.org
- https://github.com/openPMD/openPMD-standard
**Describe the solution you'd like and potential required effort**
I would like to update the "index table" of variables and attributes, e.g. with an update to the file. It's sufficient to "hide" old meta-data, e.g. as revision of a file.
There is no need to remove actual data, maybe a repack command (`h5repack`-like or `adios_reorganize`) that actually copies data could remove old revisions and un-indexed data when its actually needed.
**Describe alternatives you've considered and potential required effort**
An alternative is to copy all data to a new file (series) on a meta-data update.
**Additional context**
cc @pnorbert @chuckatkins @franzpoeschel https://code.ornl.gov/pnb/ADIOS2/-/issues/2014Append: Truncate Selected Steps (e.g larger than N)2020-03-06T13:33:59ZPodhorszki, NorbertAppend: Truncate Selected Steps (e.g larger than N)*Created by: ax3l*
**Why is this feature important?**
A very common workflow in simulations is the following:
- run simulation for a given time
- periodically dump data
- restart from an earlier time step
- overwrite some of the ...*Created by: ax3l*
**Why is this feature important?**
A very common workflow in simulations is the following:
- run simulation for a given time
- periodically dump data
- restart from an earlier time step
- overwrite some of the already existing data
We usually start to truncate existing time steps when writing in a series of files by encoding the iteration/snapshot number in the file name ([convention in the openPMD 1.1.0 schema](https://github.com/openPMD/openPMD-standard/blob/1.1.0/STANDARD.md#iterations-and-time-series)).
If we were to change to `Begin/EndStep` workflows that re-use variables over time, truncating after a seek to a certain position (or even random step access, which is probably way harder).
**What is the potential impact of this feature in the community?**
This is our production workflow in WarpX, PIConGPU, et al. (re: openPMD).
**Is your feature request related to a problem? Please describe.**
Related to file series #1046
**Describe the solution you'd like and potential required effort**
Being able to truncate steps larger than a user-provided step `N` - and actually freeing that memory - would work for us. Also being able to remove selected steps (truncate-on-write) would be even better.
**Describe alternatives you've considered and potential required effort**
We work-around this by creating new files and re-defining variables and attributes every time we write.
**Additional context**
cc @chuckatkins @pnorbert @franzpoeschel https://code.ornl.gov/pnb/ADIOS2/-/issues/2006Function to get the absolute steps of a variable that is not written every step2020-03-02T22:28:50ZPodhorszki, NorbertFunction to get the absolute steps of a variable that is not written every step*Created by: pnorbert*
ADIOS reading works with relative steps, i.e. if a file has 100 output steps, and variable X has 20 steps in it, then one addresses the steps from 0..19. What is missing, the opportunity for a reader to identify w...*Created by: pnorbert*
ADIOS reading works with relative steps, i.e. if a file has 100 output steps, and variable X has 20 steps in it, then one addresses the steps from 0..19. What is missing, the opportunity for a reader to identify which actual steps X was written to.
The absolute step is not required for reading the data. It is desirable for applications/middleware to track back their stepping of that variable.
Asking for a public function for a variable, which returns a vector of the absolute steps the variable was written.
For example, openPMD needs to identify the absolute steps and currently does that by encoding the step in the name of the variable. A proper streaming solution with one-variable-many-steps requires this additional functionality (for openPMD to support its reading model).
@ax3l @franzpoeschel https://code.ornl.gov/pnb/ADIOS2/-/issues/2000Starting with ADIOS22020-03-06T19:02:19ZPodhorszki, NorbertStarting with ADIOS2*Created by: aruhela*
What is the recommended way to start with ADIOS2?
Is there available any Video tutorial to get an overview and basic usage of it ?*Created by: aruhela*
What is the recommended way to start with ADIOS2?
Is there available any Video tutorial to get an overview and basic usage of it ?https://code.ornl.gov/pnb/ADIOS2/-/issues/1972Extended precision floating point support2020-02-18T22:52:03ZPodhorszki, NorbertExtended precision floating point support*Created by: chuckatkins*
This is to track the issues surrounding support for extended precision float, i.e. long double. Some things to consider:
1. The multitude of precision and byte representations to support.
2. Cross-langua...*Created by: chuckatkins*
This is to track the issues surrounding support for extended precision float, i.e. long double. Some things to consider:
1. The multitude of precision and byte representations to support.
2. Cross-language support (Fortran `REAL(16)` may or may not be supported at all and if it is, may not be the same as the corresponding C / C++ `long double`)
3. Padding issues on non-dense representations (i.e. 80-bit long double taking up 128 bits of space)
4. Read support for formats not supported by the current architecture.https://code.ornl.gov/pnb/ADIOS2/-/issues/1965Feature-driven docs and testing for file and staging engines2020-02-13T20:12:35ZEisenhauer, GregFeature-driven docs and testing for file and staging enginesCurrently ADIOS has a lot of features, and not all of them are supported in all engines. It would be good to have a single location which tracks which features are supported by which engines and knows which tests require which features,...Currently ADIOS has a lot of features, and not all of them are supported in all engines. It would be good to have a single location which tracks which features are supported by which engines and knows which tests require which features, that way we could drive testing so that we test that functionality on those engines, and drive the creation of a table in docs, a web page, or something similar that documents what engines actually run with those features.
https://code.ornl.gov/pnb/ADIOS2/-/issues/1966Trouble reading files in python2020-02-14T18:52:53ZPodhorszki, NorbertTrouble reading files in python*Created by: rkube*
I'm trying to read multiple bp files in python. I have filex xgc.3d.000??.c.bp where ??=0..92.
This is the content of the files:
(ml) [rkube@traverse ml_data_case_1_92ts_iteration]$ bpls xgc.3d.00092.c.bp
doub...*Created by: rkube*
I'm trying to read multiple bp files in python. I have filex xgc.3d.000??.c.bp where ??=0..92.
This is the content of the files:
(ml) [rkube@traverse ml_data_case_1_92ts_iteration]$ bpls xgc.3d.00092.c.bp
double a_par {19316, 8}
double apar_del {19316, 8}
double apar_old {19316, 8}
double apar_res {19316, 8}
double apar_try {19316, 8}
double dadt {19316, 8}
double dn_del {19316, 8}
double dpot {19316, 8}
double eden {19316, 8}
double iden {19316, 8}
int32_t iphi scalar
int32_t nnode scalar
int32_t nphi scalar
double pot0 [1]*{19316}
double pot_del {19316, 8}
double pot_mid {19316, 8}
double pot_old {19316, 8}
double pot_res {19316, 8}
double pot_try {19316, 8}
double potm0 [1]*{19316}
double u_e {19316, 8}
double u_i {19316, 8}
When I'm trying to access the file using the example code from the docs I don't see any contents:
```python
print(fname)
with adios2.open(fname, "r") as fh:
for fstep in fh:
step_vars = fstep.available_variables()
for name, info in step_vars.items():
print("variable_name: " + name)
for key, value in info.items():
print("\t" + key + ": " + value)
print("\n")
```
The command above doesn't print anythin
When I try to read data directly from fh it works fine:
```python
data = rh.read("a_par")
```
When I try reading the fields from fh in a loop I get an error:
```
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-61-094e1f12f241> in <module>
2 fname = join(datapath, f"xgc.3d.{nn_idx:05d}.c.bp")
3 with adios2.open(fname, "r") as fh:
----> 4 apar = fh.read("a_par")
5 dpot = fh.read("dpot")
6 fh.close()
ValueError: ERROR: offset 0 from steps start 41 in variable a_par is beyond the largest available step = 41, check Variable SetStepSelection argument stepsCount (random access), or number of BeginStep calls (streaming), in call to Get
```
How can I consistently read the data from my files?https://code.ornl.gov/pnb/ADIOS2/-/issues/1954need a general test for SetMemorySelection in multiple engines2020-02-01T13:45:18ZEisenhauer, Gregneed a general test for SetMemorySelection in multiple enginesJust opening an issue for this as a reminder. When #1866 is resolved (or sooner), we should make sure that there's an actual test for memory selections that can be run on multiple engines. Currently the only place we have for such thin...Just opening an issue for this as a reminder. When #1866 is resolved (or sooner), we should make sure that there's an actual test for memory selections that can be run on multiple engines. Currently the only place we have for such things is staging-common.https://code.ornl.gov/pnb/ADIOS2/-/issues/1953Memory selection and compression leads to corrupt data2020-02-13T15:44:25ZPodhorszki, NorbertMemory selection and compression leads to corrupt data*Created by: pnorbert*
**Describe the bug**
When using SetMemorySelection for a pointer and use SZ compression in the gray-scott example, the data is corrupt.
**To Reproduce**
Use gray-scott, change the settings-files.json.
` ...*Created by: pnorbert*
**Describe the bug**
When using SetMemorySelection for a pointer and use SZ compression in the gray-scott example, the data is corrupt.
**To Reproduce**
Use gray-scott, change the settings-files.json.
` "adios_memory_selection": true,`
Add SZ compression to the SimulationOutput in adios2.xml
```
$ mpirun -n 4 adios2-gray-scott settings-files.json
$ mpirun -n 1 python3 gsplot.py -i gs.bp
```
**Additional context**
The settings-staging.json is using mem selection and if I use compression in staging, the same error shows up.
2.6.0 Production Release