From 0ed7b6773933d01eefa6d9b6856097a912088947 Mon Sep 17 00:00:00 2001 From: Neil Vaytet <neil.vaytet@esss.se> Date: Wed, 29 Aug 2018 16:16:27 +0200 Subject: [PATCH] Refs #23285 : Updated developer documentation --- dev-docs/source/SystemTests.rst | 80 +++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/dev-docs/source/SystemTests.rst b/dev-docs/source/SystemTests.rst index b0c2cd06dfc..9380383f397 100644 --- a/dev-docs/source/SystemTests.rst +++ b/dev-docs/source/SystemTests.rst @@ -212,6 +212,86 @@ e.g. would run all of the tests whose name contains SNS. +Running the tests on multiple cores +----------------------------------- + +Running the System Tests can be sped up by distributing the list of +tests across multiple cores. This is done in a similar way to ``ctest`` +using the ``-j N`` option, where ``N`` is the number of cores you want +to use, e.g. + +.. code-block:: sh + ./systemtest -j 8 + +would run the tests on 8 cores. + +Some tests write or delete in the same directories, using the same file +names, which causes issues when running in parallel. To resolve this, +the tests are grouped in lists where all modules starting with the +same 4 letters are given to one core. This worsens the load balance +between cores (with 8 cores, core 1 performs 93 tests while core 8 only +has 44). This is not ideal but allows the suite to complete without +failures. The runtime using 8 cores still goes down from 2h to 30 min. + +This also means that in the case of running a subset of tests with the +``-R`` option, if the number of groups created from this is smaller +than the number of cores being used, some cores will have no tests to +run. Using the ``-j`` option is only really advantageous when running +a large list of tests. It does not bring much speedup up for a small +subset of tests, as these are likely to be put inside the same group +and run on the same core. + +Reducing the size of console output +----------------------------------- + +The ``systemtests`` can be run in "quiet" mode using the ``-q`` or +``--quiet`` option. This will print only one line per test instead of +the full log. + +.. code-block:: sh + ./systemtest --quiet + Updating testing data... + [100%] Built target StandardTestData + [100%] Built target SystemTestData + Running tests... + FrameworkManager-[Notice] Welcome to Mantid 3.13.20180820.2132 + FrameworkManager-[Notice] Please cite: http://dx.doi.org/10.1016/j.nima.2014.07.029 and this release: http://dx.doi.org/10.5286/Software/Mantid + [ 0%] 1/435 : DOSTest.DOSCastepTest ............................................... (success: 0.05s) + [ 0%] 2/435 : ISISIndirectBayesTest.JumpCETest .................................... (success: 0.06s) + [ 0%] 3/435 : ISISIndirectInelastic.IRISCalibration ............................... (success: 0.03s) + [ 0%] 4/435 : HFIRTransAPIv2.HFIRTrans1 ........................................... (success: 1.30s) + [ 1%] 5/435 : DOSTest.DOSIRActiveTest ............................................. (success: 0.04s) + [ 1%] 6/435 : ISISIndirectBayesTest.JumpFickTest .................................. (success: 0.06s) + [ 1%] 7/435 : AbinsTest.AbinsBinWidth ............................................. (success: 1.65s) + [ 1%] 8/435 : ISIS_PowderPearlTest.CreateCalTest .................................. (success: 1.65s) + [ 2%] 9/435 : ISISIndirectInelastic.IRISConvFit ................................... (success: 0.56s) + [ 2%] 10/435 : LiquidsReflectometryReductionWithBackgroundTest.BadDataTOFRangeTest . (success: 2.94s) + [ 2%] 11/435 : DOSTest.DOSPartialCrossSectionScaleTest ............................. (success: 0.23s) + [ 2%] 12/435 : ISISIndirectBayesTest.JumpHallRossTest .............................. (success: 0.07s) + [ 2%] 13/435 : ISISIndirectInelastic.IRISDiagnostics ............................... (success: 0.03s) + [ 3%] 14/435 : HFIRTransAPIv2.HFIRTrans2 ........................................... (success: 0.83s) + [ 3%] 15/435 : DOSTest.DOSPartialSummedContributionsCrossSectionScaleTest .......... (success: 0.15s) + [ 3%] 16/435 : ISISIndirectBayesTest.JumpTeixeiraTest .............................. (success: 0.07s) + [ 3%] 17/435 : ISISIndirectInelastic.IRISElwinAndMSDFit ............................ (success: 0.29s) + [ 4%] 18/435 : MagnetismReflectometryReductionTest.MRFilterCrossSectionsTest ....... (success: 5.30s) + [ 4%] 19/435 : DOSTest.DOSPartialSummedContributionsTest ........................... (success: 0.16s) + +One can recover the full log when a test fails by using the ``--ouptut-on-failure`` option. + +Running a cleanup run +--------------------- + +A cleanup run will go through all the tests and call the +``.cleanup()`` function for each test. It will not run the tests +(i.e. call the ``execute()`` function) themselves. This is achieved +by using the ``-c`` or ``--clean`` option, e.g. + +.. code-block:: sh + ./systemtest -c + +This is useful if some old data is left over from a previous run, +where some tests were not cleanly exited. + Adding New Data & References Files ---------------------------------- -- GitLab