More changes to dataloaders and raps config for the simulation server

I think this PR should finally get everything working for the simulation server again. I have the dashboard and simulation server repos working in their respective develop branches as well.

Changes:

  • Fixed incorrect nodes_per_blade for selene
  • Added a raps download command that calls the download function in the dataloaders to conveniently fetch the datasets
    • Not all the dataloaders currently have a download() function, you'll still have to fetch those manually.
  • Some fixes to SimConfig replay mode
    • Validate that workload, priority and scheduler are set properly when --replay is set
    • Remove the replay scheduler as it is unused and the default scheduler handles replay
  • Shift telemetry_start back to match sim_config.start
    • With the current dataloaders, WorkloadData.start_date can end up being after sim_config.start. Most of the dataloaders set timestep_start and start_date to whatever the first job in the dataset is. This issue is most apparent when the dataloader does any filtering based on start/end so start_date will be whichever job started in the interval.
    • Eventually I think we should refactor the dataloaders to return absolute unix timestamps to avoid all these relative vs non-relative timestep complications. But for now I'm just going to shift the telemetry_start back to match the sim_config.start if needed. This does mean that telemetry_start can now be a negative value, but that seems to work fine.
  • Add dataloader option to allow passing a custom dataloader
Edited by Hines, Jesse

Merge request reports

Loading