# ADIOS 2 : The Adaptable Input Output System version 2
This is v2.4.0-rc1 of the ADIOS 2: The Adaptable Input/Output (I/O) System, ADIOS 2 is developed as part of the United States Department of Energy's Exascale Computing Program.
This is ADIOS 2: The Adaptable Input/Output (I/O) System, ADIOS 2 is developed as part of the United States Department of Energy's Exascale Computing Program.
ADIOS 2 is a framework designed for scientific data I/O to publish and subscribe (put/get) data when and where required.
ADIOS2 would Transport and Transform your data as groups of self-describing variables and attributes across different media (file, wide-area-network, memory-to-memory, etc.) using a common application programming interface (API) for all transport modes. ADIOS2 uses range from HPC supercomputers to personal computers and cloud based applications.
ADIOS2 transports data as groups of self-describing variables and attributes across different media types (such as files, wide-area-networks, and remote direct memory access) using a common application programming interface for all transport modes. ADIOS2 can be used on supercomputers, commercial clouds, and personal computers.
ADIOS 2 focuses on:
...
...
@@ -54,7 +54,7 @@ Please find [The ADIOS 2 User Guide at readthedocs](https://adios2.readthedocs.i
@@ -71,7 +71,7 @@ The goal is to provide specific advice and good practices about the use of ADIOS
20. C, Fortran: always call ``adios2_finalize`` for every call to ``adios2_init`` to avoid memory leaks.
21. Reminder: C++, C, Python: Row-Major, while Fortran: Column-Major. ADIOS 2 will handle interoperability between ordering. Remember that :ref:`bpls` is always a Row-Major reader.
21. Reminder: C++, C, Python: Row-Major, while Fortran: Column-Major. ADIOS 2 will handle interoperability between ordering. Remember that :ref:`bpls : Inspecting Data` is always a Row-Major reader so Fortran reader need to swap dimensions seen in bpls. bpls: (slow, ...., fast) -> Fortran(fast,...,slow).
22. Fortran API: use the type members (``var%valid``, ``var%name``, etc.) to get extra type information.
...
...
@@ -81,5 +81,5 @@ The goal is to provide specific advice and good practices about the use of ADIOS
25. Developers: explore the testing infrastructure ``ADIOS2/testing`` in ADIOS 2 as a starting point for using ADIOS 2 in your own testing environment.
26. Become a super-user of :ref:`bpls` to analyze datasets generated by ADIOS 2.
26. Become a super-user of :ref:`bpls : Inspecting Data` to analyze datasets generated by ADIOS 2.
Product Application Developers targeting finer-level control for their IO tasks for optimization should use the current full APIs. If you want to use ADIOS2 in simple use cases (*e.g.* reading a file for analysis, interactive Python, or saving some data for a small project) please refer to the :ref:`High-Level Language Bindings APIs` for a flat learning curve.
Application developers who desire fine-grained control of IO tasks should use the Full APIs. In simple cases (e.g. reading a file for analysis, interactive Python, or saving some data for a small project or tests) please refer to the :ref:High-Level APIs.
Currently ADIOS2 support bindings for the following languages and their minimum standards:
@@ -87,13 +87,11 @@ The following section provides a summary of the available functionality for each
:path: ../../bindings/CXX11/adios2/cxx11/
:members:
Debugging
---------
To help debugging, functionality is provided that creates
human-readable representations of adios2 C++11 class instances and
enums. To get a string representation, use ``ToString(object)``. You
can also directly pass objects to ``ostream``s.
For debugging, ADIOS2 C++11 class instances and enums can be passed directly to ostreams, as well as converted to human-readable strings via the ubiquitous ``ToString(object)`` member variable. You can also directly pass objects to ``ostream``s.
Product Application Developers targeting finer-level control for their IO tasks for optimization should use the current full APIs. If you want to use ADIOS2 in simple use cases (*e.g.* reading a file for analysis, interactive Python, or saving some data for a small project) please refer to the :ref:`High-Level Language Bindings APIs` for a flat learning curve.
Product Application Developers targeting finer-level control for their IO tasks for optimization should use the current full APIs. If you want to use ADIOS2 in simple use cases (*e.g.* reading a file for analysis, interactive Python, or saving some data for a small project) please refer to the :ref:`High-Level APIs` for a flat learning curve.
The full Python APIs follow very closely the full C++11 API interface.
Work in progress.
This section is Work In Progress until we figure out an automated way to pull the docs.
The current simple APIs are designed for simple and direct tasks in which performance is not a critical aspect. Unlike the :ref:`Full Language Bindings APIs` the simple APIs only require a single object/handler offering native resemblance to C++ (fstream) and Python (file I/O) native implementations. Thus, offering a nearly-flat learning curve for first-time users.
The high-level APIs are designed for simple tasks for which performance is not critical. Unlike the :ref:`Full APIs`, the high-level APIs only require a single objecthandler resembling a C++ fstream or a Python file I/O idiom. The high-level APIs are recommended to both first-time and advanced users; the low-level APIs being recommended only when performance testing identifies a bottleneck or when more control is needed.
Typical scenarios for using the simple APIs are:
Typical scenarios for using the simple high-level APIs are:
* Reading a file to perform data analysis with libraries (matplotlib, scipy, etc.)
* Interactive: few calls make interactive usage easier.
...
...
@@ -16,10 +16,6 @@ The main function calls are: ``open`` (or constructor in C++), ``write``, ``read
In addition, ADIOS2 borrows the corresponding language native syntax for advancing lines to advance the step in write mode, and for a "step-by-step" streaming basis in read mode.
See each language section in this chapter for a write/read example.
.. warning::
The simplified APIs are not meant to be used inside applications in which the end user has no access to the IO layer and/or when performance and full functionality are critical, *e.g.* running on a HPC cluster at scale. Use the :ref:`Full Language Bindings APIs` instead
.. note::
The simpified APIs are based on language native file IO interface. Hence ``write`` and ``read`` calls are always synchronized and variables data memory is ready to use immediatley after these calls.
...
...
@@ -27,15 +23,15 @@ See each language section in this chapter for a write/read example.
Currently ADIOS2 support bindings for the following languages and their minimum standards:
Python simple bindings follow closely the :ref:`C++ simple bindings`. Just like the full APIs, they rely on numpy and (optionally) on ``mpi4py`` if ADIOS2 is compiled with MPI.
Python simple bindings follow closely Python style directives. Just like the full APIs, they rely on numpy and, optionally, on ``mpi4py``, if the underlying ADIOS2 library is compiled with MPI.
For online examples on MyBinder :
...
...
@@ -79,7 +79,7 @@ Python Read "step-by-step" example
If you are doing simple tasks where performance is a non-critical aspect please go to the :ref:`High-Level Language Bindings APIs` section for a quick start. If you are an HPC application developer or you want to use ADIOS2 functionality in full please read this chapter.
If you are doing simple tasks where performance is a non-critical aspect please go to the :ref:`High-Level APIs` section for a quick start. If you are an HPC application developer or you want to use ADIOS2 functionality in full please read this chapter.
The simple way to understand the big picture for the ADIOS2 unified user interface components is to map each class to the actual definition of the ADIOS acronym.
...
...
@@ -42,7 +42,7 @@ ADIOS2's public APIs are based on the natural choice for each supported language
The following section provides a common overview to all languages based on the C++11 APIs. For each specific language go to the :ref:`Full Language Bindings APIs` section, but it's highly recommended to read this section as components map 1-to-1 in other languages.
The following section provides a common overview to all languages based on the C++11 APIs. For each specific language go to the :ref:`Full APIs` section, but it's highly recommended to read this section as components map 1-to-1 in other languages.
The following figure depicts the components hierarchy from the application's point of view.
As of version 2.4.0 this work is experimental. As it builds on top of the library in third party ecosystems, and doesn't ship with ADIOS 2 source code.
Certain ADIOS 2 bp files, and in-memory streams in the future), can be recognized by third party products to enable data visualization. The expectation is to keep adding support to the list of products enabling ADIOS 2 for visualization purposes. The goal of this section is to describe the currently cover cases and how to create an ADIOS 2 BP file to accomodate the visualization product requirements.
ADIOS BP files can now be seamlessly integrated into the `Visualization Toolkit <https://vtk.org/>`_ (VTK) and `Paraview <https://www.paraview.org/>`_. Datasets can be described with an additional attribute that conforms to the `VTK XML data model formats <https://vtk.org/wp-content/uploads/2015/04/file-formats.pdf>`_ as high-level descriptors that will allow interpretation of ADIOS 2 variables into a hierarchy understood by the VTK infrastructure. This XML data format is saved in ADIOS as either an attribute, and optionally, as an additional vtk.xml file inside file.bp.dir/vtk.xml from the default engine. For more details, see :ref:`BP3 (Default)`.
Current limitations:
* Only works with MPI builds of VTK and Paraview
* Support only one Block per ADIOS dataset
* Only supports BP Files, streams are not supported
* Currently working up to 3D (and linearized 1D) variables for scalars and vectors.
* Image Data, vti, is supported with ADIOS2 Global Array Variables only
* Unstructured Grid, vtu, is supported with ADIOS2 Local Arrays Variables only
Two "VTK File" types are supported:
1. Image data (.vti)
2. Unstructured Grid (.vtu)
The main idea is to populate the above XML format contents describing the extent and the data arrays with ADIOS variables in the BP data set. The result is a more-than-compact typical VTK data file since extra information about the variable, such as dimensions and type, as they already exist in the BP data set.
In addition, VTK can interpret physical-time or output-step varying data stored with ADIOS by resusing the special "TIME" tag. This is better illustrated in the following section.
Saving the vtk.xml data model
-----------------------------
For the full source code of the following illustration example see the `gray-scott adios2 tutorial <https://github.com/pnorbert/adiosvm/tree/master/Tutorial/gray-scott>`_
To incorporate the data model in a BP data file, the application has two options:
1) Adding a string attribute called "vtk.xml" in code. "TIME" is a special tag for adding physical time variables.
Similarly, unstructured grid (.vtu) support can be added with the limitations of using specific labels for the variable names setting the "connectivity", "vertices", and cell "types".
The following example is taken from example 2 of the `MFEM product examples website <https://mfem.org/examples/>`_ using ADIOS 2:
The resulting `bpls` output for unstructured grid data types:
Currently the implementation for ADIOS 2 readers exist in VTK and Paraview branches. We expect this to be part of the VTK and Paraview release cycle with their upcoming releases. Users must build from source and point to these branches until formal merge into their master branches is done.
Paraview and its VTK dependency must be built with the following CMake options:
1. `-DVTK_MODULE_ENABLE_VTK_IOADIOS2=YES`
2. `-DVTK_USE_MPI=ON`
3. `-DPARAVIEW_USE_MPI=ON`
For comprehensive build instructions see the documentation for `VTK <https://vtk.org/>`_ (VTK) and `Paraview <https://www.paraview.org/>`_.
Follow directions at :ref:`Building on High Performance Computing, HPC, Systems` to setup support for PGI on Titan. PGI compilers depend on GNU headers, but they must point to a version greater than gcc 4.8.1 to support C++11 features. The gcc module doesn't need to be loaded, though. Example:
Follow directions at :ref:`Building on HPC Systems` to setup support for PGI on Titan. PGI compilers depend on GNU headers, but they must point to a version greater than gcc 4.8.1 to support C++11 features. The gcc module doesn't need to be loaded, though. Example:
.. code-block:: bash
...
...
@@ -91,7 +91,7 @@ Follow directions at :ref:`Building on High Performance Computing, HPC, Systems`
How do I enable the Python bindings on Titan?
---------------------------------------------
ADIOS 2 default configuration on Titan is to build the static library. Python bindings require enabling the dynamic libraries and the Cray dynamic environment variable. See :ref:`Building on High Performance Computing, HPC, Systems` and :ref:`Enabling the Python bindings`. For example:
The default ADIOS2 configuration on Titan builds a static library. Python bindings require enabling the dynamic libraries and the Cray dynamic environment variable. See :ref:`Building on HPC Systems` and :ref:`Enabling the Python bindings`. For example:
.. code-block:: bash
...
...
@@ -125,7 +125,7 @@ ADIOS 2 default configuration on Titan is to build the static library. Python bi
-- Found PythonFull: /sw/titan/.swci/0-login/opt/spack/20180315/linux-suse_linux11-x86_64/gcc-4.3.4/python-2.7.9-v6ctjewwdx6k2qs7ublexz7gnx457jo5/bin/python2.7 found components: Interp Libs numpy mpi4py
