Rework event filtering documentation

docs/source/algorithms/FilterByLogValue-v1.rst

@@ -38,6 +38,11 @@ will be included also. If a log has a single point in time, then that
 log value is assumed to be constant for all time and if it falls within
 the range, then all events will be kept.
 
+.. warning::
+
+   :ref:`FilterByLogValue <algm-FilterByLogValue>` is not suitable for
+   fast log filtering.
+
 PulseFilter (e.g. for Veto Pulses)
 ##################################
 
@@ -63,7 +68,7 @@ rejected. For example, this call will filter out veto pulses:
 .. testsetup:: VetoPulseTime
 
    ws=CreateSampleWorkspace("Event")
-   AddTimeSeriesLog(ws, Name="veto_pulse_time", Time="2010-01-01T00:00:00", Value=1) 
+   AddTimeSeriesLog(ws, Name="veto_pulse_time", Time="2010-01-01T00:00:00", Value=1)
    AddTimeSeriesLog(ws, Name="veto_pulse_time", Time="2010-01-01T00:10:00", Value=0)
    AddTimeSeriesLog(ws, Name="veto_pulse_time", Time="2010-01-01T00:20:00", Value=1)
    AddTimeSeriesLog(ws, Name="veto_pulse_time", Time="2010-01-01T00:30:00", Value=0)
@@ -77,20 +82,20 @@ rejected. For example, this call will filter out veto pulses:
 Comparing with other event filtering algorithms
 ###############################################
 
-Wiki page :ref:`EventFiltering` has a detailed
-introduction on event filtering in MantidPlot.
+The :ref:`EventFiltering` page has a detailed introduction on event
+filtering in mantid.
 
 
 Usage
 -----
 
-**Example - Filtering by a simple time series Log**  
+**Example - Filtering by a simple time series Log**
 
 .. testcode:: FilterByLogValue
 
    ws = CreateSampleWorkspace("Event",BankPixelWidth=1)
 
-   AddTimeSeriesLog(ws, Name="proton_charge", Time="2010-01-01T00:00:00", Value=100) 
+   AddTimeSeriesLog(ws, Name="proton_charge", Time="2010-01-01T00:00:00", Value=100)
    AddTimeSeriesLog(ws, Name="proton_charge", Time="2010-01-01T00:10:00", Value=100)
    AddTimeSeriesLog(ws, Name="proton_charge", Time="2010-01-01T00:20:00", Value=100)
    AddTimeSeriesLog(ws, Name="proton_charge", Time="2010-01-01T00:30:00", Value=100)

docs/source/algorithms/FilterEvents-v1.rst

@@ -9,147 +9,120 @@
 Description
 -----------
 
-This algorithm filters events from an :ref:`EventWorkspace` to one or
-multiple :ref:`EventWorkspaces <EventWorkspace>` according to an input
-:ref:`SplittersWorkspace` containing a series of splitters (i.e.,
-:ref:`splitting intervals <SplittingInterval>`).
-
-Inputs
-######
-
-FilterEvents takes 2 mandatory input Workspaces and 1 optional
-Workspace.  One of mandatory workspace is the :ref:`EventWorkspace`
-where the events are filtered from.  The other mandatory workspace is
-workspace containing splitters.  It can be a MatrixWorkspace, a TableWorkspace or
-a :ref:`SplittersWorkspace <SplittersWorkspace>`.
-
-The optional workspace is a :ref:`TableWorkspace <Table Workspaces>`
-for information of splitters.
-
-Workspace containing splitters
-==============================
-
-This algorithm accepts three types of workspace that contains event splitters.
-- TableWorkspace: a general TableWorkspace with at three columns
-- MatrixWorkspace: a 1-spectrum MatrixWorkspace
-- SplittersWorkspace: an extended TableWorkspace with restrict definition on start and stop time.
-
-Event splitter
-++++++++++++++
-
-An event splitter contains three items, start time, stop time and splitting target (index).
-All the events belonged to the same splitting target will be saved to a same output EventWorkspace.
-
-Unit of input splitters
-+++++++++++++++++++++++
-
-- MatrixWorkspace:  the unit must be second.
-- TableWorkspace: the unit must be second.
-- SplittersWorkspace: by the definition of SplittersWorkspace, the unit has to be nanosecond.
-
-
-How to generate input workspace containing splitters
-++++++++++++++++++++++++++++++++++++++++++++++++++++
-
-There are two ways to generate
-
-Algorithm :ref:`GenerateEventsFilter <algm-GenerateEventsFilter>`
-creates both the :ref:`SplittersWorkspace <SplittersWorkspace>` and
-splitter information workspace.
-
-
-Splitters in relative time or absolute time
-+++++++++++++++++++++++++++++++++++++++++++
-
-As the SplittersWorkspace is in format of :ref:`MatrixWorkspace
-<MatrixWorkspace>`, its time, i.e., the value in X vector, can be
-relative time.
-
-Property ``RelativeTime`` flags that the splitters' time is relative.
-Property ``FilterStartTime`` specifies the starting time of the filter.
-Or the shift of time of the splitters.
-If it is not specified, then the algorithm will search for sample log ``run_start``.
-
-Outputs
-#######
-
-The output will be one or multiple workspaces according to the number of
-index in splitters. The output workspace name is the combination of
-parameter OutputWorkspaceBaseName and the index in splitter.
-
-Calibration File
-################
-
-The calibration, or say correction, from the detector to sample must be
-consider in fast log. Thus a calibration file is required. The math is
-
-``TOF_calibrated = TOF_raw * correction(detector ID).``
-
-The calibration is in column data format.
-
-A reasonable approximation of the correction is
-
-``correction(detector_ID) = L1/(L1+L2(detector_ID))``
+This algorithm filters events from a single :ref:`EventWorkspace` to
+one or multiple :ref:`EventWorkspaces <EventWorkspace>` according to
+the ``SplittersWorkspace`` property. The :ref:`EventFiltering` concept
+page has a detailed introduction to event filtering.
+
+Specifying the splitting strategy
+---------------------------------
+
+The ``SplittersWorkspace`` describes much of the information for
+splitting the ``InputWorkspace`` into the various output
+workspaces. It can have one of three types
+
++--------------------------------------------------------------+-------------+----------+
+| workspace class                                              | units       | rel/abs  |
++==============================================================+=============+==========+
+| :ref:`MatrixWorkspace <MatrixWorkspace>`                     | seconds     | either   |
++--------------------------------------------------------------+-------------+----------+
+| :class:`SplittersWorkspace <mantid.api.ISplittersWorkspace>` | nanoseconds | absolute |
++--------------------------------------------------------------+-------------+----------+
+| :ref:`TableWorkspace <Table Workspaces>`                     | seconds     | either   |
++--------------------------------------------------------------+-------------+----------+
+
+Whether the values in :ref:`MatrixWorkspace <MatrixWorkspace>` and
+:ref:`TableWorkspace <Table Workspaces>` is treated as relative or
+absolute time is dependent on the value of ``RelativeTime``. In the
+case of ``RelativeTime=True``, the time is relative to the start of
+the run (in the ``ws.run()['run_start']``) or, if specified, the
+``FilterStartTime``. In the case of ``RelativeTime=False``, the times
+are relative to the :class:`GPS epoch <mantid.kernel.DateAndTime>`.
+
+Both :ref:`TableWorkspace <Table Workspaces>` and
+:class:`SplittersWorkspace <mantid.api.ISplittersWorkspace>` have 3
+colums, ``start``, ``stop``, and ``target`` which should be a float,
+float, and string. The :ref:`event filtering <EventFiltering>` concept
+page has details on creating the :ref:`TableWorkspace <Table
+Workspaces>` by hand.
+
+If the ``SplittersWorkspace`` is a :ref:`MatrixWorkspace
+<MatrixWorkspace>`, it must have a single spectrum with the x-value is
+the time boundaries and the y-value is the workspace group index.
+
+The optional ``InformationWorkspace`` is a :ref:`TableWorkspace <Table
+Workspaces>` for information of splitters.
 
 Unfiltered Events
-#################
+-----------------
 
 Some events are not inside any splitters. They are put to a workspace
-name ended with '\_unfiltered'.
-
-If input property 'OutputWorkspaceIndexedFrom1' is set to True, then
-this workspace shall not be outputted.
+name ended with ``_unfiltered``. If
+``OutputWorkspaceIndexedFrom1=True``, then this workspace will not be
+created.
 
 Using FilterEvents with fast-changing logs
-##########################################
+------------------------------------------
+
+There are a few parameters to consider when the log filtering is
+expected to produce a large splitter table. An example of such a case
+would be a data file for which the events need to be split according
+to a log with two or more states changing in the kHz range. To reduce
+the filtering time, one may do the following:
+
+- Make sure the ``SplitterWorkspace`` input is a :ref:`MatrixWorkspace
+  <MatrixWorkspace>`. Such a workspace can be produced by using the
+  ``FastLog = True`` option when calling :ref:`GenerateEventsFilter
+  <algm-GenerateEventsFilter>`.
+- Choose the logs to split. Filtering the logs can take a substantial
+  amount of time. To save time, you may want to split only the logs
+  you will need for analysis. To do so, set ``ExcludeSpecifiedLogs =
+  False`` and list the logs you need in
+  ``TimeSeriesPropertyLogs``. For example, if we only need to know the
+  accumulated proton charge for each filtered workspace, we would set
+  ``TimeSeriesPropertyLogs = proton_charge``.
+
+Correcting time neutron was at the sample
+#########################################
+
+When filtering fast logs, the time to filter by is the time that the
+neutron was at the sample. This can be specified using the
+``CorrectionToSample`` parameter. Either the user specifies the
+correction parameter for every pixel, or one is calculated. The
+correction parameters are applied as
+
+.. math::
+
+   TOF_{sample} = TOF_{detector} * scale[detectorID] + shift[detectorID]
+
+and stored in the ``OutputTOFCorrectionWorkspace``.
+
+* ``CorrectionToSample="None"`` applies no correction
+* ``CorrectionToSample="Elastic"`` applies :math:`shift = 0` with
+  :math:`scale = L1/(L1+L2)` for detectors and :math:`scale = L1/L_{monitor}`
+  for monitors
+* ``CorrectionToSample="Direct"`` applies :math:`scale = 0` and
+  :math:`shift = L1 / \sqrt{2 E_{fix} / m_n}`.  The value supplied in
+  ``IncidentEnergy`` will override the value found in the workspace's
+  value of ``Ei``.
+* ``CorrectionToSample="Indirect"`` applies :math:`scale = 1` and
+  :math:`shift = -1 * L2 / \sqrt{2 E_{fix} / m_n}` for detectors. For
+  monitors, uses the same corrections as ``Elastic``.
+
+* ``CorrectionToSample="Customized"`` applies the correction supplied
+  in the ``DetectorTOFCorrectionWorkspace``.
 
-There are a few parameters to consider when the log filtering is expected to produce a large
-splitter table. An example of such a case would be a data file for which the events need to be split
-according to a log with two or more states changing in the kHz range. To reduce the filtering time,
-one may do the following:
-
-- Make sure the ``SplitterWorkspace`` input is a :ref:`MatrixWorkspace <MatrixWorkspace>`. Such a workspace can be produced by using the ``FastLog = True`` option when calling :ref:`GenerateEventsFilter <algm-GenerateEventsFilter>`.
-- Choose the logs to split. Filtering the logs can take a substantial amount of time. To save time, you may want to split only the logs you will need for analysis. To do so, set ``ExcludeSpecifiedLogs = False`` and list the logs you need in ``TimeSeriesPropertyLogs``. For example, if we only need to know the accumulated proton charge for each filtered workspace, we would set ``TimeSeriesPropertyLogs = proton_charge``.
 
 Difference from FilterByLogValue
-################################
-
-In FilterByLogValue(), EventList.splitByTime() is used.
-
-In FilterEvents, if FilterByPulse is selected true,
-EventList.SplitByTime is called; otherwise, EventList.SplitByFullTime()
-is called instead.
-
-The difference between splitByTime and splitByFullTime is that
-splitByTime filters events by pulse time, and splitByFullTime considers
-both pulse time and TOF.
-
-Therefore, FilterByLogValue is not suitable for fast log filtering.
-
-Comparing with other event filtering algorithms
-###############################################
-
-Wiki page :ref:`EventFiltering` has a detailed introduction on event
-filtering in MantidPlot.
-
-
-Developer's Note
-----------------
-
-Splitters given by TableWorkspace
-#################################
-
-- The ``start/stop time`` is converted to ``m_vecSplitterTime``.
-- The splitting target (in string) is mapped to a set of continuous integers that are stored in ``m_vecSplitterGroup``.
-  - The mapping will be recorded in ``m_targetIndexMap`` and ``m_wsGroupIndexTargetMap``.
-  - Class variable ``m_maxTargetIndex`` is set up to record the highest target group/index,i.e., the max value of ``m_vecSplitterGroup``.
-
-
-Undefined splitting target
-##########################
-
-Indexed as ``0`` in m_vecSplitterGroup.
-
+--------------------------------
+
+In :ref:`FilterByLogValue <algm-FilterByLogValue>`,
+``EventList.splitByTime()`` is used. In FilterEvents, it only uses
+this when ``FilterByPulse=True``. Otherwise,
+``EventList.splitByFullTime()`` is used. The difference between
+``splitByTime`` and ``splitByFullTime`` is that ``splitByTime``
+filters events by pulse time, and ``splitByFullTime`` considers both
+pulse time and TOF.
 
 Usage
 -----

docs/source/concepts/EventFiltering.rst

@@ -7,86 +7,117 @@ Event Filtering
 .. contents::
    :local:
 
-In MantidPlot, there are a few algorithms working with event
-filtering.  These algorithms are :ref:`algm-FilterByTime`,
-:ref:`algm-FilterByLogValue`, :ref:`algm-FilterEvents`, and
-:ref:`algm-GenerateEventsFilter`.
+In mantid, there are a variety of ways to filter events that are in an
+:ref:`EventWorkspace`. They are :ref:`FilterByTime
+<algm-FilterByTime>` and :ref:`FilterByLogValue
+<algm-FilterByLogValue>` which will create a filter and apply it in a
+single step. The other way to filter events is to use
+:ref:`FilterEvents <algm-FilterEvents>` which allows for a variety of
+workspaces to specify how an :ref:`EventWorkspace` is split. This
+document focuses on how the create these workspaces and will largely
+ignore :ref:`FilterByTime <algm-FilterByTime>` and
+:ref:`FilterByLogValue <algm-FilterByLogValue>`.
 
 How to generate event filters
 =============================
 
-Generating filters explicitly
------------------------------
-
-:ref:`algm-FilterEvents` reads and parses a
-:class:`mantid.api.ISplittersWorkspace` object to generate a list of
-:ref:`SplittingIntervals <SplittingInterval>`, which are used to split
-neutron events to specified output workspaces according to the times
-that they arrive detectors.
-
-There can be two approaches to create a
-:class:`mantid.api.ISplittersWorkspace`.
-
-* :ref:`algm-GenerateEventsFilter` generate event filters by either by
-  time or log value.  The output filters are stored in a
-  :ref:`SplittersWorkspace`, which is taken as an input property of
-  :ref:`algm-FilterEvents`.
-
-* Users can create a :class:`mantid.api.ISplittersWorkspace` from scrach from Python
-  script, because :class:`mantid.api.ISplittersWorkspace` inherits from
-  :ref:`TableWorkspace <Table Workspaces>`.
-
-Generating inexplicit filters
------------------------------
-
-:ref:`algm-FilterByTime` and :ref:`algm-FilterByLogValue` generate event filters during execution.
-
-* :ref:`algm-FilterByTime` generates a set of :ref:`SplittingInterval`
-  according to user-specified setup for time splicing;
-
-* :ref:`algm-FilterByLogValue` generates a set of
-  :ref:`SplittingInterval` according to the value of a specific sample
-  log.
-
-:ref:`algm-GenerateEventsFilter` and :ref:`algm-FilterEvents` vs :ref:`algm-FilterByTime` and :ref:`algm-FilterByLogValue`
---------------------------------------------------------------------------------------------------------------------------
-
-* If :ref:`algm-GenerateEventsFilter` and :ref:`algm-FilterEvents` are
-  set up correctly, they can have the same functionality as
-  :ref:`algm-FilterByTime` and :ref:`algm-FilterByLogValue`.
-
-* :ref:`algm-FilterEvents` is able to filter neutron events by either
-  their pulse times or their absolute times.  An neutron event's
-  absolute time is the summation of its pulse time and TOF.
-
-* :ref:`algm-FilterByLogValue` and :ref:`algm-FilterByTime` can only
-  split neutron events by their pulse time.
-
-Types of events filters
-=======================
-
-Filtering by :ref:`SplittingInterval`
--------------------------------------
-
-:ref:`SplittingInterval` is an individual class to indicate an
-independent time splitter.  Any event can be filtered by a
-:ref:`SplittingInterval` object.
-
-:ref:`SplittersWorkspace` is a :ref:`TableWorkspace <Table
-Workspaces>` that stors a set of :ref:`SplittingInterval`.
-
-Filtering by duplicate entries/booleans
----------------------------------------
-
-Duplicate entries in a :ref:`TimeSeriesProperty` and boolean type of
-:ref:`TimeSeriesProperty` are used in MantidPlot too to serve as time
-splitters.
-
-These two are applied in the MantidPlot log viewing functionality and
-unfortunately intrudes into :ref:`TimeSeriesProperty`.
-
-As time splitters are better to be isolated from logs, which are
-recorded in :ref:`TimeSeriesProperty`, it is not
-recommended to set up event filters by this approach.
+Implicit filters
+----------------
+
+:ref:`algm-FilterByTime` and :ref:`algm-FilterByLogValue` internally
+generate event filters during execution that are not exposed to the
+user. These algorithms can only split the neutron events by pulse
+time and do not provide the equivalent of a ``FastLog=True`` option.
+
+Explicit filters
+----------------
+
+:ref:`algm-FilterEvents` takes either a :class:`SplittersWorkspace
+<mantid.api.ISplittersWorkspace>`, :ref:`TableWorkspace <Table
+Workspaces>`, or :ref:`MatrixWorkspace <MatrixWorkspace>` as the
+``SplittersWorkspace``. The events are split into output workspaces
+according to the times that they arrive detectors.
+
+:ref:`GenerateEventsFilter <algm-GenerateEventsFilter>` will create a
+:class:`SplittersWorkspace <mantid.api.ISplittersWorkspace>` based on
+its various options. This result can be supplied as the
+``SplittersWorkspace`` input property of ref:`algm-FilterEvents`. It
+will also generate an ``InformationWorkspace`` which can be passed
+along to :ref:`GenerateEventsFilter <algm-GenerateEventsFilter>`.
+Depending on the parameters in :ref:`GenerateEventsFilter
+<algm-GenerateEventsFilter>`, the events will be filtered based on
+their pulse times or their absolute times.  An neutron event's
+absolute time is the summation of its pulse time and TOF.
+
+Custom event filters
+====================
+
+Sometimes one wants to filter events based on arbitrary conditions. In
+this case, one needs to go beyond what existing algorithms can do. For
+this, one must generate their own splitters workspace. The workspace
+is generally 3 columns, with the first two being start and stop times
+and the third being the workspace index to put the events into. For
+filtering with time relative to the start of the run, the first two
+columns are ``float``. To specify the times as absolute, in the case
+of filtering files that will be summed together, the first two columns
+should be ``int64``. For both of the examples below, the filter
+workspaces are created using the following function:
+
+.. code-block:: python
+
+   def create_table_workspace(table_ws_name, column_def_list):
+      CreateEmptyTableWorkspace(OutputWorkspace=table_ws_name)
+      table_ws = mtd[table_ws_name]
+      for col_tup in column_def_list:
+          data_type = col_tup[0]
+          col_name = col_tup[1]
+          table_ws.addColumn(data_type, col_name)
+
+      return table_ws
+
+Relative time
+-------------
+
+The easiest way to generate a custom event filter is to make one
+relative to the start time of the run or relative to a specified
+epoch. As the times in the table are seconds, a table can be created
+and used
+
+.. code-block:: python
+
+   filter_rel = create_table_workspace('custom_relative', [('float', 'start'), ('float', 'stop'), ('str', 'target')])
+   filter_rel.addRow((0,9500, '0'))
+   filter_rel.addRow((9500,19000, '1'))
+   FilterEvents(InputWorkspace='ws', SplitterWorkspace=filter_rel,
+                GroupWorkspaces=True, OutputWorkspaceBaseName='relative', RelativeTime=True)
+
+This will generate an event filter relative to the start of the
+run. Specifying the ``FilterStartTime`` in :ref:`FilterEvents
+<algm-FilterEvents>`, one can specify a different time that filtering
+will be relative to.
+
+Absolute time
+-------------
+
+If instead a custom filter is to be created with absolute time, the
+time must be processed somewhat to go into the table workspace. Much of the
+
+.. code-block:: python
+
+   abs_times = [datetime64('2014-12-12T09:11:22.538096666'), datetime64('2014-12-12T11:45:00'), datetime64('2014-12-12T14:14:00')]
+   # convert to time relative to GPS epoch
+   abs_times = [time - datetime64('1990-01-01T00:00') for time in abs_times]
+   # convert to number of seconds
+   abs_times = [float(time / timedelta64(1, 's')) for time in abs_times]
+
+   filter_abs = create_table_workspace('custom_absolute', [('float', 'start'), ('float', 'stop'), ('str', 'target')])
+   filter_abs.addRow((abs_times[0], abs_times[1], '0'))
+   filter_abs.addRow((abs_times[1], abs_times[2], '1'))
+   FilterEvents(InputWorkspace='PG3_21638', SplitterWorkspace=filter_abs,
+                GroupWorkspaces=True, OutputWorkspaceBaseName='absolute', RelativeTime=False)
+
+Be warned that specifying ``RelativeTime=True`` with a table full of
+absolute times will almost certainly generate output workspaces
+without any events in them.
 
 .. categories:: Concepts

docs/source/release/v3.14.0/framework.rst

@@ -84,6 +84,7 @@ Improvements
 - :ref:`GroupDetectors <algm-GroupDetectors>` now takes masked bins correctly into account when processing histogram workspaces.
 - :ref:`SaveNexusProcessed <algm-SaveNexusProcessed>` and :ref:`LoadNexusProcessed <algm-LoadNexusProcessed>` can now save and load a ``MaskWorkspace``.
 - :ref:`FitPeaks <algm-FitPeaks>` can output parameters' uncertainty (fitting error) in an optional workspace.
+- The documentation in :ref:`EventFiltering` and :ref:`FilterEvents <algm-FilterEvents>` have been extensively rewritten to aid in understanding what the code does.
 
 Bugfixes
 ########