Manual: split testing and devtools into separate chapters

                   usermanual/chapters/modules.tex

                   usermanual/chapters/examples.tex

                   usermanual/chapters/development.tex

                   usermanual/chapters/devtools.tex

                   usermanual/chapters/faq.tex

                   usermanual/chapters/additional.tex

                   usermanual/chapters/acknowledgements.tex

% development of new modules and detectors models

\input{chapters/development}

\input{chapters/devtools}

\input{chapters/testing}

% frequently asked question

\chapter{Development Tools \& Continuous Integration}

\label{ch:devtools}

The following chapter will introduce a few tools included in the framework to ease development and help to maintain a high code quality. This comprises tools for the developer to be used while coding, as well as a continuous integration (CI) and automated test cases of various framework and module functionalities.

The chapter is structured as follows.

Section~\ref{sec:targets} describes the available \command{make} targets for code quality and formatting checks, Section~\ref{sec:ci} briefly introduces the CI, and Section~\ref{sec:tests} provides an overview of the currently implemented framework, module, and performance test scenarios.

\section{Additional Targets}

\label{sec:targets}

A set of testing targets in addition to the standard compilation targets are automatically created by CMake to enable additional code quality checks and testing.

Some of these targets are used by the project's CI, others are intended for manual checks.

Currently, the following targets are provided:

\begin{description}

  \item[\command{make format}] invokes the \command{clang-format} tool to apply the project's coding style convention to all files of the code base. The format is defined in the \file{.clang-format} file in the root directory of the repository and mostly follows the suggestions defined by the standard LLVM style with minor modifications. Most notably are the consistent usage of four whitespace characters as indentation and the column limit of 125 characters.

  \item[\command{make check-format}] also invokes the \command{clang-format} tool but does not apply the required changes to the code. Instead, it returns an exit code 0 (pass) if no changes are necessary and exit code 1 (fail) if changes are to be applied. This is used by the CI.

  \item[\command{make lint}] invokes the \command{clang-tidy} tool to provide additional linting of the source code. The tool tries to detect possible errors (and thus potential bugs), dangerous constructs (such as uninitialized variables) as well as stylistic errors. In addition, it ensures proper usage of modern \CPP standards. The configuration used for the \command{clang-tidy} command can be found in the \file{.clang-tidy} file in the root directory of the repository.

  \item[\command{make check-lint}] also invokes the \command{clang-tidy} tool but does not report the issues found while parsing the code. Instead, it returns an exit code 0 (pass) if no errors have been produced and exit code 1 (fail) if issues are present. This is used by the CI.

  \item[\command{make cppcheck}] runs the \command{cppcheck} command for additional static code analysis. The output is stored in the file \file{cppcheck_results.xml} in XML2.0 format. It should be noted that some of the issues reported by the tool are to be considered false positives.

  \item[\command{make cppcheck-html}] compiles a HTML report from the defects list gathered by \command{make cppcheck}. This target is only available if the \command{cppcheck-htmlreport} executable is found in the \dir{PATH}.

  \item[\command{make package}] creates a binary release tarball as described in Section~\ref{sec:packaging}.

\end{description}

\section{Packaging}

\label{sec:packaging}

\apsq comes with a basic configuration to generate tarballs from the compiled binaries using the CPack command. In order to generate a working tarball from the current \apsq build, the \parameter{RPATH} of the executable should not be set, otherwise the \command{allpix} binary will not be able to locate the dynamic libraries. If not set, the global \parameter{LD_LIBRARY_PATH} is used to search for the required libraries:

\begin{verbatim}

$ mkdir build

$ cd build

$ cmake -DCMAKE_SKIP_RPATH=ON ..

$ make package

\end{verbatim}

Since the CMake installation path defaults to the project's source directory, certain files are excluded from the default installation target created by CMake.

This includes the detector models in the \dir{models/} directory as well as the additional tools provided in \dir{tools/root_analysis_macros/} folder. In order to include them in a release tarball produced by CPack, the installation path should be set to a location different from the project source folder, for example:

\begin{verbatim}

$ cmake -DCMAKE_INSTALL_PREFIX=/tmp ..

\end{verbatim}

The content of the produced tarball can be extracted to any location of the file system, but requires the ROOT6 and Geant4 libraries as well as possibly additional libraries linked by individual at runtime.

For this purpose, a \file{setup.sh} shell script is automatically generated and added to the tarball.

By default, it contains the ROOT6 path used for the compilation of the binaries.

Additional dependencies, either library paths or shell scripts to be sourced, can be added via CMake for individual modules using the CMake functions described below.

The paths stored correspond to the dependencies used at compile time, it might be necessary to change them manually when deploying on a different computer.

\paragraph{\texttt{\textbf{ADD\_RUNTIME\_DEP(name)}}}

This CMake command can be used to add a shell script to be sourced to the setup file.

The mandatory argument \parameter{name} can either be an absolute path to the corresponding file, or only the file name when located in a search path known to CMake, for example:

\begin{minted}[frame=single,framesep=3pt,breaklines=true,tabsize=2,linenos]{cmake}

# Add "geant4.sh" as runtime dependency for setup.sh file:

ADD_RUNTIME_DEP(geant4.sh)

\end{minted}

The command uses the \command{GET_FILENAME_COMPONENT} command of CMake with the \parameter{PROGRAM} option.

Duplicates are removed from the list automatically.

Each file found will be written to the setup file as

\begin{verbatim}

source <absolute path to the file>

\end{verbatim}

\paragraph{\texttt{\textbf{ADD\_RUNTIME\_LIB(names)}}}

This CMake command can be used to add additional libraries to the global search path.

The mandatory argument \parameter{names} should be the absolute path of a library or a list of paths, such as:

\begin{minted}[frame=single,framesep=3pt,breaklines=true,tabsize=2,linenos]{cmake}

# This module requires the LCIO library:

FIND_PACKAGE(LCIO REQUIRED)

# The FIND routine provides all libraries in the LCIO_LIBRARIES variable:

ADD_RUNTIME_LIB(${LCIO_LIBRARIES})

\end{minted}

The command uses the \command{GET_FILENAME_COMPONENT} command of CMake with the \parameter{DIRECTORY} option to determine the directory of the corresponding shared library.

Duplicates are removed from the list automatically.

Each directory found will be added to the global library search path by adding the following line to the setup file:

\begin{verbatim}

export LD_LIBRARY_PATH="<library directory>:$LD_LIBRARY_PATH"

\end{verbatim}

\section{Continuous Integration}

\label{sec:ci}

Quality and compatibility of the \apsq framework is ensured by an elaborate continuous integration (CI) which builds and tests the software on all supported platforms.

The \apsq CI uses the GitLab Continuous Integration features and consists of seven distinct stages as depicted in Figure~\ref{fig:ci}.

It is configured via the \file{.gitlab-ci.yml} file in the repository's root directory, while additional setup scripts for the GitLab Ci Runner machines and the Docker instances can be found in the \dir{.gitlab/ci} directory.

\begin{figure}[btp]

  \centering

  \includegraphics[width=\textwidth]{ci.png}

  \caption{Typical \apsq continuous integration pipeline with 34 jobs distributed over seven distinct stages. In this example, all jobs passed.}

  \label{fig:ci}

\end{figure}

The \textbf{compilation} stage builds the framework from the source on different platforms.

Currently, builds are performed on CentOS\,7, CentOS\,8, and macOS.

On Linux type platforms, the framework is compiled with recent versions of GCC and Clang, while the latest AppleClang is used on macOS.

The build is always performed with the default compiler flags enabled for the project:

\begin{verbatim}

    -pedantic -Wall -Wextra -Wcast-align -Wcast-qual -Wconversion

    -Wuseless-cast -Wctor-dtor-privacy -Wzero-as-null-pointer-constant

    -Wdisabled-optimization -Wformat=2 -Winit-self -Wlogical-op

    -Wmissing-declarations -Wmissing-include-dirs -Wnoexcept

    -Wold-style-cast -Woverloaded-virtual -Wredundant-decls

    -Wsign-conversion -Wsign-promo -Wstrict-null-sentinel

    -Wstrict-overflow=5 -Wswitch-default -Wundef -Werror -Wshadow

    -Wformat-security -Wdeprecated -fdiagnostics-color=auto

    -Wheader-hygiene

\end{verbatim}

The \textbf{testing} stage executes the framework system and unit tests described in Section~\ref{sec:tests}.

Different jobs are used to run different test types.

This allows to optimize the CI setup depending on the demands of the test to be executed.

All tests are expected to pass, and no code that fails to satisfy all tests will be merged into the repository.

The \textbf{formatting} stage ensures proper formatting of the source code using the \command{clang-format} and following the coding conventions defined in the \file{.clang-format} file in the repository.

In addition, the \command{clang-tidy} tool is used for ``linting'' of the source code.

This means, the source code undergoes a static code analysis in order to identify possible sources of bugs by flagging suspicious and non-portable constructs used.

Tests are marked as failed if either of the CMake targets \command{make check-format} or \command{make check-lint} fail.

No code that fails to satisfy the coding conventions and formatting tests will be merged into the repository.

Furthermore, also basic sanity checks are carried out on the CMake build framework code using \command{cmake-lint}.

The \textbf{performance} stage runs a longer simulation with several thousand events and measures the execution time.

This facilitates monitoring of the simulation performance, a failing job would indicate a degradation in speed.

These CI jobs run on dedicated machines with only one concurrent job as described in Section~\ref{sec:tests}.

Performance tests are separated into their own CI stage because their execution is time consuming and they should only be started once proper  formatting of the new code is established.

The \textbf{documentation} stage prepares this user manual as well as the Doxygen source code documentation for publication.

This also allows to identify e.g.\ failing compilation of the \LaTeX documents or additional files which accidentally have not been committed to the repository.

The \textbf{packaging} stage wraps the compiled binaries up into distributable tarballs for several platforms.

This includes adding all libraries and executables to the tarball as well as preparing the \file{setup.sh} script to prepare run-time dependencies using the information provided to the build system.

This procedure is described in more detail in Section~\ref{sec:packaging}.

Finally, the \textbf{deployment} stage is only executed for new tags in the repository.

Whenever a tag is pushed, this stages receives the build artifacts of previous stages and publishes them to the \apsq project website through the EOS file system~\cite{eos}. More detailed information on deployments is provided in the following.

\section{Automatic Deployment}

The CI is configured to automatically deploy new versions of \apsq and its user manual and code reference to different places to make them available to users.

This section briefly describes the different deployment end-points currently configured and in use.

The individual targets are triggered either by automatic nightly builds or by publishing new tags.

In order to prevent accidental publications, the creation of tags is protected.

Only users with \emph{Maintainer} privileges can push new tags to the repository.

For new tagged versions, all deployment targets are executed.

\subsection{Software deployment to CVMFS}

\label{sec:cvmfs}

The software is automatically deployed to CERN's VM file system (CVMFS)~\cite{cvmfs} for every new tag.

In addition, the \parameter{master} branch is built and deployed every night.

New versions are published to the folder \dir{/cvmfs/clicdp.cern.ch/software/allpix-squared/} where a new folder is created for every new tag, while updates via the \parameter{master} branch are always stored in the \dir{latest} folder.

The deployed version currently comprises all modules as well as the detector models shipped with the framework.

An additional \file{setup.sh} is placed in the root folder of the respective release, which allows to set up all runtime dependencies necessary for executing this version.

Versions both for CentOS\,7 and CentOS\,8 are provided.

The deployment CI job runs on a dedicated computer with a GitLab SSH runner.

Job artifacts from the packaging stage of the CI are downloaded via their ID using the script found in \dir{.gitlab/ci/download_artifacts.py}, and are made available to the \emph{cvclicdp} user which has access to the CVMFS interface.

The job checks for concurrent deployments to CVMFS and then unpacks the tarball releases and publishes them to the CLICdp experiment CVMFS space, the corresponding script for the deployment can be found in \dir{.gitlab/ci/gitlab_deployment.sh}.

This job requires a private API token to be set as secret project variable through the GitLab interface, currently this token belongs to the service account user \emph{ap2}.

\subsection{Documentation deployment to EOS}

The project documentation is deployed to the project's EOS space at \dir{/eos/project/a/allpix-squared/www/} for publication on the project website.

This comprises both the PDF and HTML versions of the user manual (subdirectory \dir{usermanual}) as well as the Doxygen code reference (subdirectory \dir{reference/}).

The documentation is only published for new tagged versions of the framework.

The CI jobs uses the \parameter{ci-web-deployer} Docker image from the CERN GitLab CI tools to access EOS, which requires a specific file structure of the artifact.

All files in the artifact's \dir{public/} folder will be published to the \dir{www/} folder of the given project.

This job requires the secret project variables \parameter{EOS_ACCOUNT_USERNAME} and \parameter{EOS_ACCOUNT_PASSWORD} to be set via the GitLab web interface.

Currently, this uses the credentials of the service account user \emph{ap2}.

\subsection{Release tarball deployment to EOS}

Binary release tarballs are deployed to EOS to serve as downloads from the website to the directory \dir{/eos/project/a/allpix-squared/www/releases}.

New tarballs are produced for every tag as well as for nightly builds of the \parameter{master} branch, which are deployed with the name \file{allpix-squared-latest-<system-tag>-opt.tar.gz}.

The files are taken from the packaging jobs and published via the \parameter{ci-web-deployer} Docker image from the CERN GitLab CI tools.

This job requires the secret project variables \parameter{EOS_ACCOUNT_USERNAME} and \parameter{EOS_ACCOUNT_PASSWORD} to be set via the GitLab web interface.

Currently, this uses the credentials of the service account user \emph{ap2}.

\section{Building Docker images}

\label{sec:build-docker}

New \apsq Docker images are automatically created and deployed by the CI for every new tag and as a nightly build from the \parameter{master} branch.

New versions are published to project Docker container registry~\cite{ap2-container-registry}.

Tagged versions can be found via their respective tag name, while updates via the nightly build are always stored with the \parameter{latest} tag attached.

The final Docker image is formed from two consecutive images with different layers of software added.

The \parameter{deps} image contains all build dependencies such as compilers, CMake, and git as well as the two main dependencies of the framework are ROOT6 and Geant4.

It derives from the latest Ubuntu LTS Docker image and can be build using the \file{etc/docker/Dockerfile.deps} file via the following commands:

\begin{verbatim}

$ docker build --file etc/docker/Dockerfile.deps            \

               --tag gitlab-registry.cern.ch/allpix-squared/\

               allpix-squared/allpix-squared-deps           \

              .

$ docker push gitlab-registry.cern.ch/allpix-squared/\

              allpix-squared/allpix-squared-deps

\end{verbatim}

This image is created manually and only updated when necessary, i.e.\ if major new version of the underlying dependencies are available.

\begin{warning}

  The dependencies Docker image should not be flattened with commands like

  \command{docker export <container id> | docker import - <tag name>}

  because it strips any \parameter{ENV} variables set or used during the build process. They are used to set up the ROOT6 and Geant4 environments. When flattening, their executables and data paths cannot be found in the final \apsq image.

\end{warning}

Finally, the latest revision of \apsq is built using the file \file{etc/docker/Dockerfile}.

This job is performed automatically by the continuous integration and the created containers are directly uploaded to the project's Docker registry.

\begin{verbatim}

$ docker build --file etc/docker/Dockerfile                                \

               --tag gitlab-registry.cern.ch/allpix-squared/allpix-squared \

              .

\end{verbatim}

A short summary of potential use cases for Docker images is provided in Section~\ref{sec:docker}.

\section{List of Tests}

\label{app:testlist}

\paragraph{Framework Functionality Tests}

The framework functionality tests aim at reproducing basic features such as correct parsing of configuration keys or resolution of module instantiations.

Currently implemented tests comprise:

Currently implemented framework functionality tests comprise:

\begin{description}

    @CORE_TEST_DESCRIPTIONS@

\paragraph{Module Functionality Tests}

These tests ensure the proper functionality of each module covered and thus protect the framework against accidental changes affecting the physics simulation.

Using a fixed seed (using the \parameter{random_seed} configuration keyword) together with a specific version of Geant4~\cite{geant4} allows to reproduce the same simulation event.

One event is produced per test and the \parameter{DEBUG}-level logging output of the respective module is checked against pre-defined expectation output using regular expressions.

Once modules are altered, their respective expectation output has to be adapted after careful verification of the simulation result.

Module tests are located within the individual module source folders and are only enabled if the respective module will be built.

The following tests are currently performed:

The following module functionality tests are currently performed:

\begin{description}

    @MODULES_TEST_DESCRIPTIONS@

\paragraph{Performance Tests}

Similar to the module test implementation described above, performance tests use configurations prepared such, that one particular module takes most of the load (dubbed the ``slowest instantiation'' by \apsq), and a few of thousand events are simulated starting from a fixed seed for the pseudo-random number generator.

The \parameter{#TIMEOUT} keyword in the configuration file will ask CTest to abort the test after the given running time.

In the project CI, performance tests are limited to native runners, i.e. they are not executed on docker hosts where the hypervisor decides on the number of parallel jobs.

Only one test is performed at a time.

Despite these countermeasures, fluctuations on the CI runners occur, arising from different loads of the executing machines.

Thus, all performance CI jobs are marked with the \parameter{allow_failure} keyword which allows GitLab to continue processing the pipeline but will mark the final pipeline result as ``passed with warnings'' indicating an issue in the pipeline.

These tests should be checked manually before merging the code under review.

Current performance tests comprise:

\begin{description}