Skip to content

GitLab

  • Menu
Projects Groups Snippets
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • A ADIOS2
  • Project information
    • Project information
    • Activity
    • Labels
    • Planning hierarchy
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 97
    • Issues 97
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
  • Merge requests 9
    • Merge requests 9
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Podhorszki, Norbert
  • ADIOS2
  • Issues
  • #2018

Closed
Open
Created Mar 03, 2020 by Podhorszki, Norbert@pnbMaintainer

C-Blosc: Support for Large Vars

Created by: ax3l

Describe the bug

Writing for individual variables with c-blosc (1) that write very large data fails, since c-blosc has a 2Gbyte limit.

To Reproduce Based on #2010, use this diff

diff --git a/testing/adios2/performance/largevar/TestLargeVarWrite.cpp b/testing/adios2/performance/largevar/TestLargeVarWrite.cpp
index 59d4ccad..4c1acef6 100644
--- a/testing/adios2/performance/largevar/TestLargeVarWrite.cpp
+++ b/testing/adios2/performance/largevar/TestLargeVarWrite.cpp
@@ -4,6 +4,7 @@
  */
 
 #include <memory>
+#include <random>
 #include <stdexcept>
 #include <string>
 
@@ -29,6 +30,11 @@ TEST_F(Write, Plain1D_1B)
 
     std::unique_ptr<float[]> dataBig(new float[Nx]);
 
+    auto const seed = std::random_device{}();
+    std::default_random_engine gen{ seed };
+    std::uniform_real_distribution< float > dist( 0.0f, 1.0f );
+    std::generate_n( dataBig.get(), Nx, [&]{ return dist( gen ); } );
+
 #ifdef ADIOS2_HAVE_MPI
     MPI_Comm_rank(MPI_COMM_WORLD, &mpiRank);
     MPI_Comm_size(MPI_COMM_WORLD, &mpiSize);
@@ -53,6 +59,17 @@ TEST_F(Write, Plain1D_1B)
 
     auto varBig = io.DefineVariable<float>("varBig", shape, start, count,
                                            adios2::ConstantDims);
+    // add operations
+    adios2::Operator BloscOp =
+        adios.DefineOperator("BloscCompressor", adios2::ops::LosslessBlosc);
+
+    varBig.AddOperation(BloscOp,
+                         {{adios2::ops::blosc::key::compressor, "zstd"}});
+    varBig.AddOperation(BloscOp,
+                         {{adios2::ops::blosc::key::clevel, "1"}});
+    varBig.AddOperation(BloscOp,
+                         {{adios2::ops::blosc::key::doshuffle, "BLOSC_BITSHUFFLE"}});
+    // FIXME how to set the compressor threads?
 
     adios2::Engine writer = io.Open(fname, adios2::Mode::Write);

And execute. Repeat with a smaller array size in Nx, e.g. divide by 4.

Expected behavior

The size of the variable should not matter for the operation.

Desktop (please complete the following information):

  • OS/Platform: Ubuntu 18.04
  • Build: gcc 7.4.0, cmake 3.16.2

Additional context

Using threaded c-blosc compression with arbitrary variable sizes is our production ADIOS1 workflow in PIConGPU.

In ADIOS1, we added a custom-header to split up the user-input into chunks not larger than the maximum supported size in blosc: https://github.com/ornladios/ADIOS/pull/135 (This probably also affects other compressors.)

Probably even better, we can also switch directly to c-blosc2 that has 64bit buffer support.

Following up

Fixing this should complete the feature request in #1486.

cc @chuckatkins @pnorbert and @psychocoderHPC @franzpoeschel

Assignee
Assign to
Time tracking