GitLab

Created by: ax3l

Describe the bug

Writing for individual variables with c-blosc (1) that write very large data fails, since c-blosc has a 2Gbyte limit.

To Reproduce Based on #2010, use this diff

diff --git a/testing/adios2/performance/largevar/TestLargeVarWrite.cpp b/testing/adios2/performance/largevar/TestLargeVarWrite.cpp
index 59d4ccad..4c1acef6 100644
--- a/testing/adios2/performance/largevar/TestLargeVarWrite.cpp
+++ b/testing/adios2/performance/largevar/TestLargeVarWrite.cpp
@@ -4,6 +4,7 @@
  */
 
 #include <memory>
+#include <random>
 #include <stdexcept>
 #include <string>
 
@@ -29,6 +30,11 @@ TEST_F(Write, Plain1D_1B)
 
     std::unique_ptr<float[]> dataBig(new float[Nx]);
 
+    auto const seed = std::random_device{}();
+    std::default_random_engine gen{ seed };
+    std::uniform_real_distribution< float > dist( 0.0f, 1.0f );
+    std::generate_n( dataBig.get(), Nx, [&]{ return dist( gen ); } );
+
 #ifdef ADIOS2_HAVE_MPI
     MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank);
     MPI_Comm_size(MPI_COMM_WORLD, &mpiSize);
@@ -53,6 +59,17 @@ TEST_F(Write, Plain1D_1B)
 
     auto varBig = io.DefineVariable<float>("varBig", shape, start, count,
                                            adios2::ConstantDims);
+    // add operations
+    adios2::Operator BloscOp =
+        adios.DefineOperator("BloscCompressor", adios2::ops::LosslessBlosc);
+
+    varBig.AddOperation(BloscOp,
+                         {{adios2::ops::blosc::key::compressor, "zstd"}});
+    varBig.AddOperation(BloscOp,
+                         {{adios2::ops::blosc::key::clevel, "1"}});
+    varBig.AddOperation(BloscOp,
+                         {{adios2::ops::blosc::key::doshuffle, "BLOSC_BITSHUFFLE"}});
+    // FIXME how to set the compressor threads?
 
     adios2::Engine writer = io.Open(fname, adios2::Mode::Write);

And execute. Repeat with a smaller array size in Nx, e.g. divide by 4.

Expected behavior

The size of the variable should not matter for the operation.

Desktop (please complete the following information):

• OS/Platform: Ubuntu 18.04
• Build: gcc 7.4.0, cmake 3.16.2

Additional context

Using threaded c-blosc compression with arbitrary variable sizes is our production ADIOS1 workflow in PIConGPU.

In ADIOS1, we added a custom-header to split up the user-input into chunks not larger than the maximum supported size in blosc: https://github.com/ornladios/ADIOS/pull/135 (This probably also affects other compressors.)

Probably even better, we can also switch directly to c-blosc2 that has 64bit buffer support.

Following up

Fixing this should complete the feature request in #1486.

cc @chuckatkins @pnorbert and @psychocoderHPC @franzpoeschel