Skip to content
  • Dan Nixon's avatar
    Implement grouped parallel event insertion · a0af5e6b
    Dan Nixon authored
    This is based on the prototype derived from the meeting at the DMSC last December.
    
    The workflow is as follows:
      - Pool all events received from the Kafka stream into a single buffer
      - When the buffer is to be flushed (i.e. events populated in the EventWorkspace):
        - Parallel sort the event buffer by the detector ID/workspace index
        - Determine group boundaries
        - Parallel insert events into EventWorkspace
    
    Also included is the optimisation of using a spec to ID mapping based on
    a vector rather than map.
    This provides better performance during the initial event buffering
    (where this lookup is made).
    
    Initial benchamrking shows 25M events is a intermediate buffer threshold
    for LOKI (9 banks) at 10^7 events per second.
    This manages to maintain well over 14Hz pulse rate (with no post
    processing).
    a0af5e6b