More changes to dataloaders and raps config for the simulation server
I think this PR should finally get everything working for the simulation server again. I have the dashboard and simulation server repos working in their respective develop branches as well.
Changes:
- Fixed incorrect nodes_per_blade for selene
- Added a
raps downloadcommand that calls thedownloadfunction in the dataloaders to conveniently fetch the datasets- Not all the dataloaders currently have a download() function, you'll still have to fetch those manually.
- Some fixes to SimConfig replay mode
- Validate that workload, priority and scheduler are set properly when
--replayis set - Remove the replay scheduler as it is unused and the default scheduler handles replay
- Validate that workload, priority and scheduler are set properly when
- Shift telemetry_start back to match sim_config.start
- With the current dataloaders, WorkloadData.start_date can end up being after sim_config.start. Most of the dataloaders set timestep_start and start_date to whatever the first job in the dataset is. This issue is most apparent when the dataloader does any filtering based on start/end so start_date will be whichever job started in the interval.
- Eventually I think we should refactor the dataloaders to return absolute unix timestamps to avoid all these relative vs non-relative timestep complications. But for now I'm just going to shift the telemetry_start back to match the sim_config.start if needed. This does mean that telemetry_start can now be a negative value, but that seems to work fine.
- Add dataloader option to allow passing a custom dataloader
Edited by Hines, Jesse