Unverified Commit dcece165 authored by Eisenhauer, Greg's avatar Eisenhauer, Greg Committed by GitHub
Browse files

Merge pull request #5021 from eisenhauer/mgard-lossless-and-docs

Complete MGARD Lossless spec, do docs for MGARD only (not MGARDcomplex)
parents 1fcd3b03 e2332eb9
Loading
Loading
Loading
Loading
+121 −0
Original line number Diff line number Diff line
***************
CompressorMGARD
***************

The ``CompressorMGARD`` Operator is a compressor that uses lossy
error-bounded compression to achieve high compression ratios for scientific
floating-point data.

MGARD (MultiGrid Adaptive Reduction of Data) is a multilevel compression
framework based on a hierarchical (multigrid) decomposition of the input.
It supports a rigorous error bound in either an absolute or relative sense
and lets the user tune how that error is measured through a smoothness
parameter.

ADIOS2 provides a ``CompressorMGARD`` operator that lets you compress and
decompress variables. Below there is an example of how to invoke the
``CompressorMGARD`` operator:

.. code-block:: c++

    adios2::IO io = adios.DeclareIO("Output");
    auto mgardOp = adios.DefineOperator("MGARDCompressor", adios2::ops::LossyMGARD);

    auto var_r32 = io.DefineVariable<float>("r32", shape, start, count);
    var_r32.AddOperation(mgardOp, {{adios2::ops::mgard::key::tolerance, "0.001"},
                                   {adios2::ops::mgard::key::s, "inf"}});

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CompressorMGARD Specific parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``CompressorMGARD`` operator accepts the following operator specific
parameters:

+------------------+-----------------------------------------------------------------+
| ``CompressorMGARD`` available parameters                                           |
+==================+=================================================================+
| ``tolerance``    | Error tolerance (mandatory). Interpreted according to ``mode``. |
+------------------+-----------------------------------------------------------------+
| ``accuracy``     | Alias for ``tolerance``.                                        |
+------------------+-----------------------------------------------------------------+
| ``mode``         | Error bound mode: ``ABS`` or ``REL`` (default ``REL``).         |
+------------------+-----------------------------------------------------------------+
| ``s``            | Smoothness / norm parameter (default ``0``). Use ``inf`` for    |
|                  | the L-infinity (max) norm, ``0`` for the L2 norm, or any finite |
|                  | real for the corresponding Sobolev norm.                        |
+------------------+-----------------------------------------------------------------+
| ``threshold``    | Minimum input size in bytes below which compression is skipped  |
|                  | and the block is stored uncompressed (default ``100000``).      |
+------------------+-----------------------------------------------------------------+
| ``lossless_type``| Lossless stage applied after MGARD's quantization. One of:      |
|                  | ``huffman``, ``huffman_lz4``, ``huffman_zstd``, ``cpu_lossless``|
|                  | (default ``huffman_zstd``).                                     |
+------------------+-----------------------------------------------------------------+

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CompressorMGARD Supported Data Types
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``CompressorMGARD`` supports the following data types:

- ``float`` - 32-bit floating point
- ``double`` - 64-bit floating point
- ``std::complex<float>`` - complex 32-bit floating point
- ``std::complex<double>`` - complex 64-bit floating point

For complex inputs, the operator presents the data to MGARD as a
real-valued array of the underlying scalar type (``float`` or ``double``)
with the trailing dimension doubled. MGARD therefore operates on the
real and imaginary parts interleaved along that axis, which may affect
compression quality compared to a purely real-valued input of the same
shape.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CompressorMGARD Dimension Support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``CompressorMGARD`` supports arrays with 1 to 5 dimensions. ADIOS2 pads
1D and 2D blocks up to 3 dimensions before handing them to MGARD-x.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CompressorMGARD Error Bound Semantics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The error bound is specified by two orthogonal parameters: ``mode``
selects how ``tolerance`` is interpreted, and ``s`` selects the norm in
which the error is measured.

**``mode``** controls the scale of the bound:

- **``ABS``**: ``tolerance`` is an absolute value. The error in the
  selected norm is bounded by ``tolerance``.

- **``REL``** *(default)*: ``tolerance`` is relative to the magnitude of
  the input block. The guaranteed error is ``tolerance * ||input||``.

**``s``** selects the norm used to measure the error:

- ``s=inf`` (L-infinity) bounds the worst-case pointwise error.
- ``s=0`` (L2, default) bounds the aggregate error in the least-squares
  sense.
- Finite real values select the corresponding Sobolev norms, which
  balance pointwise accuracy against smoothness.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CompressorMGARD GPU Interaction
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ADIOS2 MGARD operator passes the input buffer directly to the
underlying MGARD-x library without staging it through host memory. Whether
a GPU device pointer can be used as input therefore depends on the
MGARD-x build that ADIOS2 is linked against:

- If MGARD-x is built with a GPU backend (CUDA, HIP, or SYCL), it
  consumes a matching device pointer in place.

- If MGARD-x is built SERIAL-only, the input must reside in host memory.

The output buffer that ADIOS2 hands to the operator is host-resident, so
the decompressed result is always returned to host memory regardless of
the MGARD-x backend.
+1 −0
Original line number Diff line number Diff line
@@ -25,5 +25,6 @@ Parameters are passed at:

.. include:: CompressorZFP.rst
.. include:: CompressorSZ3.rst
.. include:: CompressorMGARD.rst
.. include:: plugin.rst
.. include:: encryption.rst
+10 −1
Original line number Diff line number Diff line
@@ -144,15 +144,24 @@ size_t CompressMGARD::Operate(const char *dataIn, const Dims &blockStart, const
        {
            config.lossless = mgard_x::lossless_type::Huffman;
        }
        else if (itLosslessType->second == "huffman_lz4")
        {
            config.lossless = mgard_x::lossless_type::Huffman_LZ4;
        }
        else if (itLosslessType->second == "huffman_zstd")
        {
            config.lossless = mgard_x::lossless_type::Huffman_Zstd;
        }
        else if (itLosslessType->second == "cpu_lossless")
        {
            config.lossless = mgard_x::lossless_type::CPU_Lossless;
        }
        else
        {
            helper::Throw<std::invalid_argument>(
                "Operator", "CompressMGARD", "Operate",
                "MGARD only supports huffman/huffman_zstd lossless types");
                "MGARD lossless_type must be one of: "
                "huffman, huffman_lz4, huffman_zstd, cpu_lossless");
        }
    }

+11 −2
Original line number Diff line number Diff line
@@ -138,15 +138,24 @@ size_t CompressMGARDComplex::Operate(const char *dataIn, const Dims &blockStart,
        {
            config.lossless = mgard_x::lossless_type::Huffman;
        }
        else if (itLosslessType->second == "huffman_lz4")
        {
            config.lossless = mgard_x::lossless_type::Huffman_LZ4;
        }
        else if (itLosslessType->second == "huffman_zstd")
        {
            config.lossless = mgard_x::lossless_type::Huffman_Zstd;
        }
        else if (itLosslessType->second == "cpu_lossless")
        {
            config.lossless = mgard_x::lossless_type::CPU_Lossless;
        }
        else
        {
            helper::Throw<std::invalid_argument>(
                "Operator", "CompressMGARD", "Operate",
                "MGARD only supports huffman/huffman_zstd lossless types");
                "Operator", "CompressMGARDComplex", "Operate",
                "MGARD lossless_type must be one of: "
                "huffman, huffman_lz4, huffman_zstd, cpu_lossless");
        }
    }