Append: Truncate Selected Steps (e.g larger than N)
Created by: ax3l
Why is this feature important?
A very common workflow in simulations is the following:
- run simulation for a given time
- periodically dump data
- restart from an earlier time step
- overwrite some of the already existing data
We usually start to truncate existing time steps when writing in a series of files by encoding the iteration/snapshot number in the file name (convention in the openPMD 1.1.0 schema).
If we were to change to Begin/EndStep workflows that re-use variables over time, truncating after a seek to a certain position (or even random step access, which is probably way harder).
What is the potential impact of this feature in the community?
This is our production workflow in WarpX, PIConGPU, et al. (re: openPMD).
Is your feature request related to a problem? Please describe.
Related to file series #1046
Describe the solution you'd like and potential required effort
Being able to truncate steps larger than a user-provided step N - and actually freeing that memory - would work for us. Also being able to remove selected steps (truncate-on-write) would be even better.
Describe alternatives you've considered and potential required effort
We work-around this by creating new files and re-defining variables and attributes every time we write.
Additional context
cc @chuckatkins @pnorbert @franzpoeschel