C-Blosc: Support for Large Vars
Created by: ax3l
Describe the bug
Writing for individual variables with c-blosc (1) that write very large data fails, since c-blosc has a 2Gbyte limit.
To Reproduce Based on #2010, use this diff
diff --git a/testing/adios2/performance/largevar/TestLargeVarWrite.cpp b/testing/adios2/performance/largevar/TestLargeVarWrite.cpp
index 59d4ccad..4c1acef6 100644
--- a/testing/adios2/performance/largevar/TestLargeVarWrite.cpp
+++ b/testing/adios2/performance/largevar/TestLargeVarWrite.cpp
@@ -4,6 +4,7 @@
*/
#include <memory>
+#include <random>
#include <stdexcept>
#include <string>
@@ -29,6 +30,11 @@ TEST_F(Write, Plain1D_1B)
std::unique_ptr<float[]> dataBig(new float[Nx]);
+ auto const seed = std::random_device{}();
+ std::default_random_engine gen{ seed };
+ std::uniform_real_distribution< float > dist( 0.0f, 1.0f );
+ std::generate_n( dataBig.get(), Nx, [&]{ return dist( gen ); } );
+
#ifdef ADIOS2_HAVE_MPI
MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank);
MPI_Comm_size(MPI_COMM_WORLD, &mpiSize);
@@ -53,6 +59,17 @@ TEST_F(Write, Plain1D_1B)
auto varBig = io.DefineVariable<float>("varBig", shape, start, count,
adios2::ConstantDims);
+ // add operations
+ adios2::Operator BloscOp =
+ adios.DefineOperator("BloscCompressor", adios2::ops::LosslessBlosc);
+
+ varBig.AddOperation(BloscOp,
+ {{adios2::ops::blosc::key::compressor, "zstd"}});
+ varBig.AddOperation(BloscOp,
+ {{adios2::ops::blosc::key::clevel, "1"}});
+ varBig.AddOperation(BloscOp,
+ {{adios2::ops::blosc::key::doshuffle, "BLOSC_BITSHUFFLE"}});
+ // FIXME how to set the compressor threads?
adios2::Engine writer = io.Open(fname, adios2::Mode::Write);
And execute. Repeat with a smaller array size in Nx
, e.g. divide by 4
.
Expected behavior
The size of the variable should not matter for the operation.
Desktop (please complete the following information):
- OS/Platform: Ubuntu 18.04
- Build: gcc 7.4.0, cmake 3.16.2
Additional context
Using threaded c-blosc compression with arbitrary variable sizes is our production ADIOS1 workflow in PIConGPU.
In ADIOS1, we added a custom-header to split up the user-input into chunks not larger than the maximum supported size in blosc: https://github.com/ornladios/ADIOS/pull/135 (This probably also affects other compressors.)
Probably even better, we can also switch directly to c-blosc2 that has 64bit buffer support.
Following up
Fixing this should complete the feature request in #1486.
cc @chuckatkins @pnorbert and @psychocoderHPC @franzpoeschel