Jenkins: Performance data upload from Compy, Cori
Created by: sarats
This script facilitates automatic data upload from Compy's performance archive into PACE.
It requires the Jenkins user to run this manually the first time to use the Github credentials to create an authorized token for later uploads automatically.
It stores the token information in $HOME/.pacecc.
It does not require cloning of E3SM sources or builds.
It moves certain files from the machine's PERF_ARCHIVE_DIR to the older (OLD_PERF_ARCHIVE_DIR) after data is processed.
It deletes very large files (>50MB) that are not meant to be placed in the performance archive by design.
Testing:
Successfully uploaded performance data from Compy by commenting out the boilerplate initialization.
#source $SCRIPTROOT/util/setup_common.sh
I didn't load any Python module, system Python was sufficient. In case of Python environment issues, just unload any custom Python envs.
Merge request reports
Activity
Created by: sarats
As this tool doesn't need E3SM sources/builds, think if it needs all of the standard boilerplate initialization that's used for running E3SM tests.
As mentioned, it requires manual intervention the first time it's run by the Jenkins user to create a Github token and automates future uploads.
General info: https://climatemodeling.science.energy.gov/technical-highlights/performance-analytics-computational-experiments-pace Video - How to Upload Data https://www.youtube.com/watch?v=Vd3q3mJPQFs&feature=youtu.be
Created by: sarats
Relevant excerpt from running standalone script:
+ SCRIPTROOT=/qfs/people/sree118/repos/E3SM_test_scripts + export CIME_MACHINE=compy + CIME_MACHINE=compy + PERF_ARCHIVE_DIR=/compyfs/performance_archive + OLD_PERF_ARCHIVE_DIR=/compyfs/OLD_PERF + cd /compyfs/performance_archive + '[' 0 -ne 0 ']' + wget -O pace-upload https://pace.ornl.gov/static/tools/pace-upload --2020-05-28 15:05:18-- https://pace.ornl.gov/static/tools/pace-upload Resolving pace.ornl.gov (pace.ornl.gov)... 128.219.184.19 Connecting to pace.ornl.gov (pace.ornl.gov)|128.219.184.19|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 11154 (11K) [application/octet-stream] Saving to: ‘pace-upload’ 100%[==================================================================================================================>] 11,154 --.-K/s in 0s 2020-05-28 15:05:18 (63.3 MB/s) - ‘pace-upload’ saved [11154/11154] + '[' 0 -ne 0 ']' + chmod +x pace-upload + echo 'Large file list:' Large file list: + find . -size +50M -exec ls -lh '{}' ';' + find . -size +50M -exec rm '{}' ';' + '[' 0 -ne 0 ']' ++ date +%Y_%m_%d + curdate=2020_05_28 + perl e3sm_perf_archive.perl + mv e3sm_perf_archive_compy_2020_05_28_out.txt performance_archive_compy_all_2020_05_28 + ./pace-upload --perf-archive ./performance_archive_compy_all_2020_05_28 Authorizing sarats for PACE... * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * Welcome to PACE sarats Validating Experiments in ./performance_archive_compy_all_2020_05_28... Validation Success Compressing valid experiments... Compression Success Uploading... Upload Success Parsing experiments... Parse Success Downloading Report log... Download Success Report saved as: 'pace-sarats-2020-05-28-18:11:57.log' + '[' 0 -ne 0 ']' + mv pace-sarats-2020-05-28-18:11:57.log performance_archive_compy_all_2020_05_28 + tar zcf performance_archive_compy_all_2020_05_28.tar.gz performance_archive_compy_all_2020_05_28 ++ date +%Y-%m + curmonth=2020-05 + mkdir -p /compyfs/OLD_PERF/2020-05 + mv performance_archive_compy_all_2020_05_28 performance_archive_compy_all_2020_05_28.tar.gz /compyfs/OLD_PERF/2020-05Created by: wadeburgess
You want jenkins to run your script, and this will upload a file to the E3SM CDash project? If this is correct, in which section of CDash will your file appear? None of the existing sections seem to be a good fit. The CDash is for testing results. Is there a better place for the file to land than CDash?
Created by: sarats
I think a new Cdash section for PACE might not be a bad idea. We will have an uploader for each major platform and I foresee nightly uploads. The machine POCs would be responsible for checking the logs, so would probably be better to add to Cdash rather than something else.
Created by: sarats
@rljacob These are just nightly upload tasks, not really tests. It seems like an overkill to create an entire dashboard for a handful of tasks (compy, cori, anvil, alcf) but if you really want them separate, we can.
For now, let's get the Jenkins tasks there without the dashboard stuff. If this script design looks ok, I will template the other platform scripts based on this.
- Last reply by Mametjanov, Azamat
Created by: sarats
I have refactored the common core to a utility script (pace_archive.sh) for reuse on different platforms beginning with Compy and Cori.
The first time this script is run, it stores following details in Jenkins user's $HOME/.pacecc:
$ cat ~/.pacecc [GITHUB] username = <user> token = <github_token>
