Unverified Commit f6917365 authored by Bolea Sanchez, Vicente Adolfo's avatar Bolea Sanchez, Vicente Adolfo Committed by GitHub
Browse files

Merge pull request #4184 from vicentebolea/backports-from-master 

Backports from master
parents c2a94648 260166dd

bindings/C/adios2/c/adios2_c_io.h

+3 −3
Original line number Diff line number Diff line
@@ -329,8 +329,8 @@ adios2_error adios2_remove_all_attributes(adios2_io *io); 
 * MPI Collective function as it calls MPI_Comm_dup

 * @param io engine owner

 * @param name unique engine identifier

 * @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append, and

 * adios2_mode_readRandomAccess

 * @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append

 * and adios2_mode_readRandomAccess

 * @return success: handler, failure: NULL

 */

adios2_engine *adios2_open(adios2_io *io, const char *name, const adios2_mode mode);
@@ -341,7 +341,7 @@ adios2_engine *adios2_open(adios2_io *io, const char *name, const adios2_mode mo 
 * MPI Collective function as it calls MPI_Comm_dup

 * @param io engine owner

 * @param name unique engine identifier

 * @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append, and

 * @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append and

 * adios2_mode_readRandomAccess

 * @param comm communicator other than adios' handler comm. MPI only.

 * @return success: handler, failure: NULL

bindings/CXX11/adios2/cxx11/Variable.h

+1 −1
Original line number Diff line number Diff line
@@ -214,7 +214,7 @@ public: 
     * variable.Count() = {Ny,Nx}, then memoryCount = {Ny+2,Nx+2}

     * </pre>

     */

    void SetMemorySelection(const adios2::Box<adios2::Dims> &memorySelection);

    void SetMemorySelection(const adios2::Box<adios2::Dims> &memorySelection = {{}, {}});



    /**

     * Sets a step selection modifying current startStep, countStep

docs/user_guide/source/components/anatomy.rst

+36 −1
Original line number Diff line number Diff line
@@ -114,7 +114,10 @@ named `adios2::Mode::ReadRandomAccess`. `adios2::Mode::Read` mode allows data ac 
current timestep. `ReadRandomAccess` can only be used with file engines and involves loading all the file metadata at

once. So it can be more memory intensive than `adios2::Mode::Read` mode, but allows reading data from any timestep using

`SetStepSelection()`. If you use `adios2::Mode::ReadRandomAccess` mode, be sure to allocate enough memory to hold

multiple steps of the variable content.

multiple steps of the variable content.  Note that ADIOS streaming

engines (like SST, DataMan, etc.) do not support `ReadRandomAccess`

mode.  Also newer file Engines like BP5 to not allow

`BeginStep/EndStep` calls in `ReadRandomAccess` mode.



.. code:: C++
@@ -134,3 +137,35 @@ multiple steps of the variable content. 
    |   |--> IO goes out of scope

    |

    |--> ADIOS goes out of scope or adios2_finalize()



Previously we explored how to read using the input mode `adios2::Mode::Read`. Nonetheless, ADIOS has another input mode

named `adios2::Mode::ReadRandomAccess`. `adios2::Mode::Read` mode allows data access only timestep by timestep using

`BeginStep/EndStep`, but generally it is more memory efficient as ADIOS is only required to load metadata for the

current timestep. `ReadRandomAccess` can only be used with file engines and involves loading all the file metadata at

once. So it can be more memory intensive than `adios2::Mode::Read` mode, but allows reading data from any timestep using

`SetStepSelection()`. If you use `adios2::Mode::ReadRandomAccess` mode, be sure to allocate enough memory to hold

multiple steps of the variable content.  Note that ADIOS streaming

engines (like SST, DataMan, etc.) do not support `ReadRandomAccess`

mode.  Also newer file Engines like BP5 to not allow

`BeginStep/EndStep` calls in `ReadRandomAccess` mode.



.. code:: C++



    ADIOS adios("config.xml", MPI_COMM_WORLD);

    |

    |   IO io = adios.DeclareIO(...);

    |   |

    |   |   Engine e = io.Open("InputFileName.bp", adios2::Mode::ReadRandomAccess);

    |   |   |

    |   |   |   Variable var = io.InquireVariable(...)

    |   |   |   |   var.SetStepSelection()

    |   |   |   |   e.Get(var, datapointer);

    |   |   |   |

    |   |   |

    |   |   e.Close();

    |   |

    |   |--> IO goes out of scope

    |

    |--> ADIOS goes out of scope or adios2_finalize()

docs/user_guide/source/components/io.rst

+1 −1
Original line number Diff line number Diff line
@@ -196,7 +196,7 @@ A particular ``Engine`` type is set to the current ``IO`` component with the ``I 
Engine polymorphism is handled internally by the ``IO`` class, which allows subclassing future derived ``Engine`` types without changing the basic API.



``Engine`` objects are created in various modes.

The available modes are ``adios2::Mode::Read``, ``adios2::Mode::Write``, ``adios2::Mode::Append``, ``adios2::Mode::Sync``, ``adios2::Mode::Deferred``, and ``adios2::Mode::Undefined``.

The available modes are ``adios2::Mode::Read``, ``adios2::Mode::ReadRandomAccess``, ``adios2::Mode::Write``, ``adios2::Mode::Append``, ``adios2::Mode::Sync``, ``adios2::Mode::Deferred``, and ``adios2::Mode::Undefined``.





.. code-block:: c++

docs/user_guide/source/engines/bp5.rst

+10 −0
Original line number Diff line number Diff line
@@ -130,6 +130,14 @@ This engine allows the user to fine tune the buffering operations through the fo 
   

   #. **Threads**: Read side: Specify how many threads one process can use to speed up reading. The default value is *0*, to let the engine estimate the number of threads based on how many processes are running on the compute node and how many hardware threads are available on the compute node but it will use maximum 16 threads. Value *1* forces the engine to read everything within the main thread of the process. Other values specify the exact number of threads the engine can use. Although multithreaded reading works in a single *Get(adios2::Mode::Sync)* call if the read selection spans multiple data blocks in the file, the best parallelization is achieved by using deferred mode and reading everything in *PerformGets()/EndStep()*.   



   #. **FlattenSteps**: This is a writer-side parameter specifies that the

      reader should interpret multiple writer-created timesteps as a

      single timestep, essentially flattening all Put()s into a single step.



   #. **IgnoreFlattenSteps**: This is a reader-side parameter that

      tells the reader to ignore any FlattenSteps parameter supplied

      to the writer.



============================== ===================== ===========================================================

 **Key**                       **Value Format**      **Default** and Examples

============================== ===================== ===========================================================
@@ -156,6 +164,8 @@ This engine allows the user to fine tune the buffering operations through the fo 
 StatsLevel                     integer, 0 or 1       **1**, 0

 MaxOpenFilesAtOnce             integer >= 0          **UINT_MAX**, 1024, 1

 Threads                        integer >= 0          **0**, 1, 32

 FlattenSteps                   boolean               **off**, on, true, false

 IgnoreFlattenSteps             boolean               **off**, on, true, false

============================== ===================== ===========================================================