Skip to content

API Change request: replace Mode:Read in Open with two new modes

Created by: pnorbert

Request: Have two separate and explicit modes for reading step-by-step as a stream and for reading arbitrary step(s) in post-processing. Instead of io.Open(fname, adios2::Mode::Read); have io.Open(fname, adios2::Mode::ReadStepByStep); io.Open(fname, adios2::Mode::ReadAsFile); (names are debatable)

In the high level python API we replace "r" for "rs" and "rf" and an error on "r" will explain the options.

Issue: We want to implement proper streaming through file with the BP4 format. We want to keep in memory the metadata only about a limited number of steps. File-based engines right now must process all metadata and present all steps at Open() time in case the read code uses SetStepSelection() and does not use BeginStep(). Only if the code calls BeginStep(), do we know that it is now doing step-by-step reading.

Reasoning: We always had this conflict to support the special case of reading arbitrary steps which only works in file-based engines. The programmer must decide in the beginning of coding, which way to go. Either read step-by-step, or read arbitrary steps. The source code will be very different and there is no going back and forth. So it seems fine to force the programmer to tell ADIOS, which way the code goes.

The separate modes will allow us to write two separate reader engine classes for the BP4 engine and choose between them in the IO class at Open(). The step-by-step reader then can keep the metadata consumption down.

Options: we thought about different options:

  • engine parameter set in source or XML to differentiate
  • different engine names to differentiate
  • two separate OpenFile() and OpenStream() functions
  • two modes in Open()

All of these would achieve the same but the first two seems just to be more cumbersome for use plus its an option for the user, while the decision has to be made by the developer. The third is less elegant but very much equivalent: force the developer to tell ADIOS what is going on.

Let us know if you agree or disagree.