Skip to content

GitLab

  • Menu
Projects Groups Snippets
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • A ADIOS2
  • Project information
    • Project information
    • Activity
    • Labels
    • Planning hierarchy
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 97
    • Issues 97
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
  • Merge requests 9
    • Merge requests 9
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Podhorszki, Norbert
  • ADIOS2
  • Issues
  • #2014

Closed
Open
Created Mar 03, 2020 by Podhorszki, Norbert@pnbMaintainer

Append: Truncate Selected Steps (e.g larger than N)

Created by: ax3l

Why is this feature important?

A very common workflow in simulations is the following:

  • run simulation for a given time
  • periodically dump data
  • restart from an earlier time step
  • overwrite some of the already existing data

We usually start to truncate existing time steps when writing in a series of files by encoding the iteration/snapshot number in the file name (convention in the openPMD 1.1.0 schema).

If we were to change to Begin/EndStep workflows that re-use variables over time, truncating after a seek to a certain position (or even random step access, which is probably way harder).

What is the potential impact of this feature in the community?

This is our production workflow in WarpX, PIConGPU, et al. (re: openPMD).

Is your feature request related to a problem? Please describe.

Related to file series #1046

Describe the solution you'd like and potential required effort

Being able to truncate steps larger than a user-provided step N - and actually freeing that memory - would work for us. Also being able to remove selected steps (truncate-on-write) would be even better.

Describe alternatives you've considered and potential required effort

We work-around this by creating new files and re-defining variables and attributes every time we write.

Additional context

cc @chuckatkins @pnorbert @franzpoeschel

Assignee
Assign to
Time tracking