Names in API: Feel "More Like XML"
Created by: ax3l
Hi,
one feedback we gave @pnorbert and @sklasky in our last weeks workshop was about the "impression/feeling" that developers and users have when interacting with variable and attribute names in ADIOS files compared to what they know (XML, JSON, filesystem trees or HDF5).
As a matter of personal opinion, HDF5 is still around not because of its performance or painful 80s C API but because people can quickly open and browse it like a filesystem and definitely because of h5py. (For the first part we added ADIOS1 to HDFCompass in the past.) The new ADIOS2 API looks amazing and is super clean, yet as in ADIOS1 one thing is a bit undervalued - hierarchies hidden in "names" (paths).
I am aware that "directory hierarchies" in ADIOS1 and 2 are internally just flat maps of names, which is fine, yet a users doesn't need to know that in the frontend. What we should add to make .bp
file interaction for developers feel like browsing a filesystem is:
- allow to add "empty groups/directories": for example one might just create a path and add some attributes to it
- allow to "list" all kinds of groups, variables or attributes for arbitrary intermediate paths
- add a "group" object to the API with changed "base path": get a local, filtered view while interacting with an
adios::IO
object
For example in h5py
there is a specific Group
class that can be interacted with just as an intermediate object to access deeper Group
s, Attributes
and Variables
without specifying the full path. Such a group basically just changes what is viewed as "root" /
and otherwise would be similar to adios2::IO
.
It would be important to allow "listings" on such a view that are both recursive and non-recursive from the changed "present working dir" in the ADIOS-name (pwd).
Such an intermediate object in the APIs would also be really nice for ADIOS2. It's purely cosmetical, also it's purely optional, yet makes a world of a difference for the developer that expects some kind of "filesystem/XML hierarchy" when writing or reading complex data. With many variables and attributes, we just want to imagine ourselves to put them in little boxes (directories) and we want to create and annotate such directories with attributes even if we do not yet put variables in them.
If I understood the new APIs correctly, we probably just need an adios::Group
object that is a filtered "view" of an adios::IO
object (aka it will prefix all names with the new pwd).
Of course the ADIOS2 "name" of a variable and attribute is more general, yet when used with sane "/" separators we should also expose objects that can reflect fine grained access and listing (with changed pwd
if you will).
cc @C0nsultant who wrote a python wrapper for ADIOS1 that makes it look like h5py in terms of group handling (can you link it?) and @franzpoeschel who is working around this missing abstraction