Possibility of having custom organization of groups and datasets with HDF5 engine
Created by: jorblancoa
We have a well defined HDF5 file format that we need to follow.
/report (group)
/report/data (dtype:float, shape: N_time x N_values)
/report/mapping (group)
/report/mapping/element_ids (dtype: uint64, shape: N_values)
What we would like to have on the 2D shape in one DataSet (data) is simply as follows Where the colums are the data and the rows for the timesteps.
However when trying to reproduce the same using ADIOS2 it automatically creates groups for each timestep as shown in the documentation. The output we get when checking the h5 resulting file is:
h5ls -r sonataFstream.h5
The code to achieve this is quite simple
// Simulation
for (size_t step = 0; step < NSteps; ++step) {
oStream.write("report/data", dataToWrite.data(), shape_data, start_data, count_data, adios2::endl);
}
// MAPPING
oStream.write("report/mapping/element_ids", compartmentsToWrite.data(), shape_elements, start_elements, count_elements);
The ideal solution for us would be to have combined the 3 data datasets in one.
/report (group)
/report/data (Dataset {3}{136})
/report/mapping (Group)
/report/mapping/element_ids (Dataset {136})
Where the timesteps would be each of the 3 rows of the data Dataset.
Is there any way of achieving this custom structure of groups and datasets with the current ADIOS2?