Loading doc/usermanual/04_framework/10_multithreading.md +65 −0 Original line number Diff line number Diff line Loading @@ -64,3 +64,68 @@ to handle parallelization internally which violates the Allpix Squared design. F reproducibility between its multithreaded and sequential run managers. Modules that would like to use the Geant4 library shall not use the run managers provided by Geant4. Instead, they must use the custom run managers provided by Allpix Squared as described in [Section 14.1](../14_additional/01_tools.md#geant4-interface). ### Object History, TRefs and PointerWrappers Allpix Squared uses ROOTs `TRef` objects to store the persistent links of the simulation object history. These references act similar to C pointers and allow accessing the referenced object directly without additional bookkeeping or lookup of indices. Furthermore they persist when being written to a ROOT file and read back to memory. ROOT implements this via a central lookup table that keeps track of the referenced objects and their location in memory as described [in the ROOT documentation](https://root.cern.ch/doc/master/classTRef.html). This approach comes with some drawbacks, especially in multithreaded environments. Most importantly the lookup table is a global object, which means mutexes are required for accessing it. Multiple threads generating or using `TRef` references will have to share this mutex and will consequently be subject to significant waiting for lock release. Furthermore generating more and more `TRef` relations over the course of a simulation will increase the size of the central reference table. This table is initialized with a fixed size, and once the number of `TRef` objects outgrows this pre-allocated space, new memory has to be acquired, leading to a reallocation of memory for the entire new size of the table. With potentially millions of entries, this very quickly becomes a very computationally very expensive operation, slowing down the simulation significantly. Allpix Squared solves these limitations by wrapping the `TRef` objects into a class called `PointerWrapper`. It contains both a direct, but transitional C pointer and a `TRef` to the referenced object. The latter, however, is only generated when required, i.e. if the object holding the `PointerWrapper` as well as referenced object are going to be written to file. This is achieved by first going through all relevant objects, marking them for storage: ```cpp for(auto& object : objects) { object.markForStorage(); } ``` Now, the required history references can be identified and `TRef` objects are generated *only* for relations between two objects that are both marked for storage: ```cpp for(auto& object : objects) { object.petrifyHistory(); } ``` Objects can now be written to file and will contain the persistent reference to the related object. This approach solves the above problems. File writing has to be performed single-threaded anyway, so generating `TRef` objects on the same thread does not lead to additional locking of the central reference table mutex in root. In addition, `TRef` entries are only generated and stored in the table for objects that require it - all references to objects not to be stored will be `nullptr` in either case since the target object is not available anymore when reading in the data. Since now the generation of `TRef` objects and access to the reference table is performed by a single thread and one single event at a time, it is also possible to reset the ROOT-internal object ID of `TRef` references after the event has been processed. The subsequent event will reuse the same IDs again, preventing a continuous growth of the reference table and related memory re-allocation issues. As a consequence, when reading objects back from file, the `TRef` has to be converted back to a C memory pointer, again to avoid locking access to the central reference table when looking up the memory location from there. This is performed similarly to the generation of history relations, and here only relations to valid TRefs are loaded, other relations will hold a `nullptr`: ```cpp for(auto& object : objects) { object.loadHistory(); } ``` Since `TRef` object IDs are reused and counted up only on a per-event basis to keep the size of the central reference table under control, the ROOT-internal object count has to be reset also after *reading* an entire event from file and before proceeding with the next one. Otherwise the reused IDs from the previous event will still remain in the reference table, but pointing to invalid memory locations or to objects from the previous event. Loading
doc/usermanual/04_framework/10_multithreading.md +65 −0 Original line number Diff line number Diff line Loading @@ -64,3 +64,68 @@ to handle parallelization internally which violates the Allpix Squared design. F reproducibility between its multithreaded and sequential run managers. Modules that would like to use the Geant4 library shall not use the run managers provided by Geant4. Instead, they must use the custom run managers provided by Allpix Squared as described in [Section 14.1](../14_additional/01_tools.md#geant4-interface). ### Object History, TRefs and PointerWrappers Allpix Squared uses ROOTs `TRef` objects to store the persistent links of the simulation object history. These references act similar to C pointers and allow accessing the referenced object directly without additional bookkeeping or lookup of indices. Furthermore they persist when being written to a ROOT file and read back to memory. ROOT implements this via a central lookup table that keeps track of the referenced objects and their location in memory as described [in the ROOT documentation](https://root.cern.ch/doc/master/classTRef.html). This approach comes with some drawbacks, especially in multithreaded environments. Most importantly the lookup table is a global object, which means mutexes are required for accessing it. Multiple threads generating or using `TRef` references will have to share this mutex and will consequently be subject to significant waiting for lock release. Furthermore generating more and more `TRef` relations over the course of a simulation will increase the size of the central reference table. This table is initialized with a fixed size, and once the number of `TRef` objects outgrows this pre-allocated space, new memory has to be acquired, leading to a reallocation of memory for the entire new size of the table. With potentially millions of entries, this very quickly becomes a very computationally very expensive operation, slowing down the simulation significantly. Allpix Squared solves these limitations by wrapping the `TRef` objects into a class called `PointerWrapper`. It contains both a direct, but transitional C pointer and a `TRef` to the referenced object. The latter, however, is only generated when required, i.e. if the object holding the `PointerWrapper` as well as referenced object are going to be written to file. This is achieved by first going through all relevant objects, marking them for storage: ```cpp for(auto& object : objects) { object.markForStorage(); } ``` Now, the required history references can be identified and `TRef` objects are generated *only* for relations between two objects that are both marked for storage: ```cpp for(auto& object : objects) { object.petrifyHistory(); } ``` Objects can now be written to file and will contain the persistent reference to the related object. This approach solves the above problems. File writing has to be performed single-threaded anyway, so generating `TRef` objects on the same thread does not lead to additional locking of the central reference table mutex in root. In addition, `TRef` entries are only generated and stored in the table for objects that require it - all references to objects not to be stored will be `nullptr` in either case since the target object is not available anymore when reading in the data. Since now the generation of `TRef` objects and access to the reference table is performed by a single thread and one single event at a time, it is also possible to reset the ROOT-internal object ID of `TRef` references after the event has been processed. The subsequent event will reuse the same IDs again, preventing a continuous growth of the reference table and related memory re-allocation issues. As a consequence, when reading objects back from file, the `TRef` has to be converted back to a C memory pointer, again to avoid locking access to the central reference table when looking up the memory location from there. This is performed similarly to the generation of history relations, and here only relations to valid TRefs are loaded, other relations will hold a `nullptr`: ```cpp for(auto& object : objects) { object.loadHistory(); } ``` Since `TRef` object IDs are reused and counted up only on a per-event basis to keep the size of the central reference table under control, the ROOT-internal object count has to be reset also after *reading* an entire event from file and before proceeding with the next one. Otherwise the reused IDs from the previous event will still remain in the reference table, but pointing to invalid memory locations or to objects from the previous event.
