Commit 9d0f54cb authored by Simon Spannagel's avatar Simon Spannagel
Browse files

Merge branch 'backport_test_conditions' into 'v2.1-stable' 

[v2.1-stable]  CI: Check for Presence of at Least One PASS or FAIl Condition

See merge request allpix-squared/allpix-squared!612
parents dc9fa700 3298f303

cmake/AllpixMacros.cmake

+4 −3
Original line number Diff line number Diff line
@@ -204,10 +204,11 @@ FUNCTION(add_allpix_test test name) 
    FILE(STRINGS ${test} PASS_LST_ REGEX "#PASS ")

    FILE(STRINGS ${test} FAIL_LST_ REGEX "#FAIL ")



    # Check for number of arguments - should only be one:

    # Check for number of pass or fail conditions - we should have at least one of them

    LIST(LENGTH PASS_LST_ listcount_pass)

    IF(listcount_pass GREATER 1)

        MESSAGE(FATAL_ERROR "More than one PASS expressions defined in test ${inp}")

    LIST(LENGTH FAIL_LST_ listcount_fail)

    IF(listcount_pass EQUAL 0 AND listcount_fail EQUAL 0)

        MESSAGE(FATAL_ERROR "Neither PASS nor FAIL defined for test \"${name}\"")

    ENDIF()



    # Escape possible regex patterns in the expected output:

doc/usermanual/chapters/testing.tex

+6 −5
Original line number Diff line number Diff line
@@ -248,12 +248,11 @@ CMake automatically searches for \apsq configuration files in the different dire 


Adding a new test is as simple as adding a new configuration file to one of the different subdirectories and specifying the pass or fail conditions based on the tags described in the following paragraph.



\paragraph{Pass and Fail Conditions}

\paragraph{Test Tags, Pass and Fail Conditions}



The output of any test is compared to a search string in order to determine whether it passed or failed.

These expressions are simply placed in the configuration file of the corresponding tests, a tag at the beginning of the line indicates whether it should be used for passing or failing the test.

Each test can only contain one passing and one failing expression.

If different functionality and thus outputs need to be tested, a second test should be added to cover the corresponding expression.

Test tags allow to influence the execution condition of the given test configuration, or to define a required condition for passing or failing the test.

These expressions are simply placed in the configuration file of the corresponding tests, a tag at the beginning of the line indicates which test tag the line corresponds to.

The following tags are available:



\begin{description}

  \item[Passing a test] The expression marked with the tag \parameter{#PASS} has to be found in the output in order for the test to pass. If the expression is not found, the test fails.
@@ -267,6 +266,8 @@ If different functionality and thus outputs need to be tested, a second test sho 


Multiple pass or fail conditions can be separated by a semicolon or by adding multiple \parameter{#PASS} or \parameter{#FAIL} expressions.

It should however be noted that test passes or fails \emph{if any of these conditions is met}, i.e.\ the conditions are combined with a logical \parameter{OR}.

At least one pass or one fail conditions must be present in every test.



Pass and fail condition are not interpreted as regular expressions but relevant characters are automatically escaped.

This allows to directly copy corresponding lines form the log into the respective condition without manually creating a matching regular expression.

A noteworthy exception to this are line breaks.

etc/unittests/test_performance/test_01_deposition.conf

+1 −0
Original line number Diff line number Diff line 
#TIMEOUT 85

#FAIL FATAL;ERROR;WARNING

[Allpix]

log_level = "STATUS"

detectors_file = "detector.conf"

etc/unittests/test_performance/test_02-1_propagation_generic.conf

+1 −0
Original line number Diff line number Diff line 
#TIMEOUT 257

#FAIL FATAL;ERROR;WARNING

[Allpix]

log_level = "STATUS"

detectors_file = "detector.conf"

etc/unittests/test_performance/test_02-2_propagation_project.conf

+1 −0
Original line number Diff line number Diff line 
#TIMEOUT 530

#FAIL FATAL;ERROR;WARNING

[Allpix]

log_level = "STATUS"

detectors_file = "detector.conf"