During the maintenance period on Monday, 14 Oct 2024, all new projects will no longer create a container registry as part of the project area. Already existing projects will be unaffected at this time. Instead, users should utilize the official ORNL container registries at camden.ornl.gov (internal/moderate) and savannah.ornl.gov (external/low). Please see ORNL Today article for more info.
Randomly shuffle the training dataset
this fixes an issue where a checkpoint resume in the middle of the epoch does not point to the correct offset in the dataset
nevertheless, the dataset split should occur deterministically to avoid mixing train and val datasets between restarts