Commit ecc19e4d authored by Peterson, Peter's avatar Peterson, Peter
Browse files

Rework event filtering documentation

parent 818ce27d
......@@ -38,6 +38,11 @@ will be included also. If a log has a single point in time, then that
log value is assumed to be constant for all time and if it falls within
the range, then all events will be kept.
.. warning::
:ref:`FilterByLogValue <algm-FilterByLogValue>` is not suitable for
fast log filtering.
PulseFilter (e.g. for Veto Pulses)
##################################
......@@ -63,7 +68,7 @@ rejected. For example, this call will filter out veto pulses:
.. testsetup:: VetoPulseTime
ws=CreateSampleWorkspace("Event")
AddTimeSeriesLog(ws, Name="veto_pulse_time", Time="2010-01-01T00:00:00", Value=1)
AddTimeSeriesLog(ws, Name="veto_pulse_time", Time="2010-01-01T00:00:00", Value=1)
AddTimeSeriesLog(ws, Name="veto_pulse_time", Time="2010-01-01T00:10:00", Value=0)
AddTimeSeriesLog(ws, Name="veto_pulse_time", Time="2010-01-01T00:20:00", Value=1)
AddTimeSeriesLog(ws, Name="veto_pulse_time", Time="2010-01-01T00:30:00", Value=0)
......@@ -77,20 +82,20 @@ rejected. For example, this call will filter out veto pulses:
Comparing with other event filtering algorithms
###############################################
Wiki page :ref:`EventFiltering` has a detailed
introduction on event filtering in MantidPlot.
The :ref:`EventFiltering` page has a detailed introduction on event
filtering in mantid.
Usage
-----
**Example - Filtering by a simple time series Log**
**Example - Filtering by a simple time series Log**
.. testcode:: FilterByLogValue
ws = CreateSampleWorkspace("Event",BankPixelWidth=1)
AddTimeSeriesLog(ws, Name="proton_charge", Time="2010-01-01T00:00:00", Value=100)
AddTimeSeriesLog(ws, Name="proton_charge", Time="2010-01-01T00:00:00", Value=100)
AddTimeSeriesLog(ws, Name="proton_charge", Time="2010-01-01T00:10:00", Value=100)
AddTimeSeriesLog(ws, Name="proton_charge", Time="2010-01-01T00:20:00", Value=100)
AddTimeSeriesLog(ws, Name="proton_charge", Time="2010-01-01T00:30:00", Value=100)
......
......@@ -9,147 +9,120 @@
Description
-----------
This algorithm filters events from an :ref:`EventWorkspace` to one or
multiple :ref:`EventWorkspaces <EventWorkspace>` according to an input
:ref:`SplittersWorkspace` containing a series of splitters (i.e.,
:ref:`splitting intervals <SplittingInterval>`).
Inputs
######
FilterEvents takes 2 mandatory input Workspaces and 1 optional
Workspace. One of mandatory workspace is the :ref:`EventWorkspace`
where the events are filtered from. The other mandatory workspace is
workspace containing splitters. It can be a MatrixWorkspace, a TableWorkspace or
a :ref:`SplittersWorkspace <SplittersWorkspace>`.
The optional workspace is a :ref:`TableWorkspace <Table Workspaces>`
for information of splitters.
Workspace containing splitters
==============================
This algorithm accepts three types of workspace that contains event splitters.
- TableWorkspace: a general TableWorkspace with at three columns
- MatrixWorkspace: a 1-spectrum MatrixWorkspace
- SplittersWorkspace: an extended TableWorkspace with restrict definition on start and stop time.
Event splitter
++++++++++++++
An event splitter contains three items, start time, stop time and splitting target (index).
All the events belonged to the same splitting target will be saved to a same output EventWorkspace.
Unit of input splitters
+++++++++++++++++++++++
- MatrixWorkspace: the unit must be second.
- TableWorkspace: the unit must be second.
- SplittersWorkspace: by the definition of SplittersWorkspace, the unit has to be nanosecond.
How to generate input workspace containing splitters
++++++++++++++++++++++++++++++++++++++++++++++++++++
There are two ways to generate
Algorithm :ref:`GenerateEventsFilter <algm-GenerateEventsFilter>`
creates both the :ref:`SplittersWorkspace <SplittersWorkspace>` and
splitter information workspace.
Splitters in relative time or absolute time
+++++++++++++++++++++++++++++++++++++++++++
As the SplittersWorkspace is in format of :ref:`MatrixWorkspace
<MatrixWorkspace>`, its time, i.e., the value in X vector, can be
relative time.
Property ``RelativeTime`` flags that the splitters' time is relative.
Property ``FilterStartTime`` specifies the starting time of the filter.
Or the shift of time of the splitters.
If it is not specified, then the algorithm will search for sample log ``run_start``.
Outputs
#######
The output will be one or multiple workspaces according to the number of
index in splitters. The output workspace name is the combination of
parameter OutputWorkspaceBaseName and the index in splitter.
Calibration File
################
The calibration, or say correction, from the detector to sample must be
consider in fast log. Thus a calibration file is required. The math is
``TOF_calibrated = TOF_raw * correction(detector ID).``
The calibration is in column data format.
A reasonable approximation of the correction is
``correction(detector_ID) = L1/(L1+L2(detector_ID))``
This algorithm filters events from a single :ref:`EventWorkspace` to
one or multiple :ref:`EventWorkspaces <EventWorkspace>` according to
the ``SplittersWorkspace`` property. The :ref:`EventFiltering` concept
page has a detailed introduction to event filtering.
Specifying the splitting strategy
---------------------------------
The ``SplittersWorkspace`` describes much of the information for
splitting the ``InputWorkspace`` into the various output
workspaces. It can have one of three types
+--------------------------------------------------------------+-------------+----------+
| workspace class | units | rel/abs |
+==============================================================+=============+==========+
| :ref:`MatrixWorkspace <MatrixWorkspace>` | seconds | either |
+--------------------------------------------------------------+-------------+----------+
| :class:`SplittersWorkspace <mantid.api.ISplittersWorkspace>` | nanoseconds | absolute |
+--------------------------------------------------------------+-------------+----------+
| :ref:`TableWorkspace <Table Workspaces>` | seconds | either |
+--------------------------------------------------------------+-------------+----------+
Whether the values in :ref:`MatrixWorkspace <MatrixWorkspace>` and
:ref:`TableWorkspace <Table Workspaces>` is treated as relative or
absolute time is dependent on the value of ``RelativeTime``. In the
case of ``RelativeTime=True``, the time is relative to the start of
the run (in the ``ws.run()['run_start']``) or, if specified, the
``FilterStartTime``. In the case of ``RelativeTime=False``, the times
are relative to the :class:`GPS epoch <mantid.kernel.DateAndTime>`.
Both :ref:`TableWorkspace <Table Workspaces>` and
:class:`SplittersWorkspace <mantid.api.ISplittersWorkspace>` have 3
colums, ``start``, ``stop``, and ``target`` which should be a float,
float, and string. The :ref:`event filtering <EventFiltering>` concept
page has details on creating the :ref:`TableWorkspace <Table
Workspaces>` by hand.
If the ``SplittersWorkspace`` is a :ref:`MatrixWorkspace
<MatrixWorkspace>`, it must have a single spectrum with the x-value is
the time boundaries and the y-value is the workspace group index.
The optional ``InformationWorkspace`` is a :ref:`TableWorkspace <Table
Workspaces>` for information of splitters.
Unfiltered Events
#################
-----------------
Some events are not inside any splitters. They are put to a workspace
name ended with '\_unfiltered'.
If input property 'OutputWorkspaceIndexedFrom1' is set to True, then
this workspace shall not be outputted.
name ended with ``_unfiltered``. If
``OutputWorkspaceIndexedFrom1=True``, then this workspace will not be
created.
Using FilterEvents with fast-changing logs
##########################################
------------------------------------------
There are a few parameters to consider when the log filtering is
expected to produce a large splitter table. An example of such a case
would be a data file for which the events need to be split according
to a log with two or more states changing in the kHz range. To reduce
the filtering time, one may do the following:
- Make sure the ``SplitterWorkspace`` input is a :ref:`MatrixWorkspace
<MatrixWorkspace>`. Such a workspace can be produced by using the
``FastLog = True`` option when calling :ref:`GenerateEventsFilter
<algm-GenerateEventsFilter>`.
- Choose the logs to split. Filtering the logs can take a substantial
amount of time. To save time, you may want to split only the logs
you will need for analysis. To do so, set ``ExcludeSpecifiedLogs =
False`` and list the logs you need in
``TimeSeriesPropertyLogs``. For example, if we only need to know the
accumulated proton charge for each filtered workspace, we would set
``TimeSeriesPropertyLogs = proton_charge``.
Correcting time neutron was at the sample
#########################################
When filtering fast logs, the time to filter by is the time that the
neutron was at the sample. This can be specified using the
``CorrectionToSample`` parameter. Either the user specifies the
correction parameter for every pixel, or one is calculated. The
correction parameters are applied as
.. math::
TOF_{sample} = TOF_{detector} * scale[detectorID] + shift[detectorID]
and stored in the ``OutputTOFCorrectionWorkspace``.
* ``CorrectionToSample="None"`` applies no correction
* ``CorrectionToSample="Elastic"`` applies :math:`shift = 0` with
:math:`scale = L1/(L1+L2)` for detectors and :math:`scale = L1/L_{monitor}`
for monitors
* ``CorrectionToSample="Direct"`` applies :math:`scale = 0` and
:math:`shift = L1 / \sqrt{2 E_{fix} / m_n}`. The value supplied in
``IncidentEnergy`` will override the value found in the workspace's
value of ``Ei``.
* ``CorrectionToSample="Indirect"`` applies :math:`scale = 1` and
:math:`shift = -1 * L2 / \sqrt{2 E_{fix} / m_n}` for detectors. For
monitors, uses the same corrections as ``Elastic``.
* ``CorrectionToSample="Customized"`` applies the correction supplied
in the ``DetectorTOFCorrectionWorkspace``.
There are a few parameters to consider when the log filtering is expected to produce a large
splitter table. An example of such a case would be a data file for which the events need to be split
according to a log with two or more states changing in the kHz range. To reduce the filtering time,
one may do the following:
- Make sure the ``SplitterWorkspace`` input is a :ref:`MatrixWorkspace <MatrixWorkspace>`. Such a workspace can be produced by using the ``FastLog = True`` option when calling :ref:`GenerateEventsFilter <algm-GenerateEventsFilter>`.
- Choose the logs to split. Filtering the logs can take a substantial amount of time. To save time, you may want to split only the logs you will need for analysis. To do so, set ``ExcludeSpecifiedLogs = False`` and list the logs you need in ``TimeSeriesPropertyLogs``. For example, if we only need to know the accumulated proton charge for each filtered workspace, we would set ``TimeSeriesPropertyLogs = proton_charge``.
Difference from FilterByLogValue
################################
In FilterByLogValue(), EventList.splitByTime() is used.
In FilterEvents, if FilterByPulse is selected true,
EventList.SplitByTime is called; otherwise, EventList.SplitByFullTime()
is called instead.
The difference between splitByTime and splitByFullTime is that
splitByTime filters events by pulse time, and splitByFullTime considers
both pulse time and TOF.
Therefore, FilterByLogValue is not suitable for fast log filtering.
Comparing with other event filtering algorithms
###############################################
Wiki page :ref:`EventFiltering` has a detailed introduction on event
filtering in MantidPlot.
Developer's Note
----------------
Splitters given by TableWorkspace
#################################
- The ``start/stop time`` is converted to ``m_vecSplitterTime``.
- The splitting target (in string) is mapped to a set of continuous integers that are stored in ``m_vecSplitterGroup``.
- The mapping will be recorded in ``m_targetIndexMap`` and ``m_wsGroupIndexTargetMap``.
- Class variable ``m_maxTargetIndex`` is set up to record the highest target group/index,i.e., the max value of ``m_vecSplitterGroup``.
Undefined splitting target
##########################
Indexed as ``0`` in m_vecSplitterGroup.
--------------------------------
In :ref:`FilterByLogValue <algm-FilterByLogValue>`,
``EventList.splitByTime()`` is used. In FilterEvents, it only uses
this when ``FilterByPulse=True``. Otherwise,
``EventList.splitByFullTime()`` is used. The difference between
``splitByTime`` and ``splitByFullTime`` is that ``splitByTime``
filters events by pulse time, and ``splitByFullTime`` considers both
pulse time and TOF.
Usage
-----
......
......@@ -7,86 +7,117 @@ Event Filtering
.. contents::
:local:
In MantidPlot, there are a few algorithms working with event
filtering. These algorithms are :ref:`algm-FilterByTime`,
:ref:`algm-FilterByLogValue`, :ref:`algm-FilterEvents`, and
:ref:`algm-GenerateEventsFilter`.
In mantid, there are a variety of ways to filter events that are in an
:ref:`EventWorkspace`. They are :ref:`FilterByTime
<algm-FilterByTime>` and :ref:`FilterByLogValue
<algm-FilterByLogValue>` which will create a filter and apply it in a
single step. The other way to filter events is to use
:ref:`FilterEvents <algm-FilterEvents>` which allows for a variety of
workspaces to specify how an :ref:`EventWorkspace` is split. This
document focuses on how the create these workspaces and will largely
ignore :ref:`FilterByTime <algm-FilterByTime>` and
:ref:`FilterByLogValue <algm-FilterByLogValue>`.
How to generate event filters
=============================
Generating filters explicitly
-----------------------------
:ref:`algm-FilterEvents` reads and parses a
:class:`mantid.api.ISplittersWorkspace` object to generate a list of
:ref:`SplittingIntervals <SplittingInterval>`, which are used to split
neutron events to specified output workspaces according to the times
that they arrive detectors.
There can be two approaches to create a
:class:`mantid.api.ISplittersWorkspace`.
* :ref:`algm-GenerateEventsFilter` generate event filters by either by
time or log value. The output filters are stored in a
:ref:`SplittersWorkspace`, which is taken as an input property of
:ref:`algm-FilterEvents`.
* Users can create a :class:`mantid.api.ISplittersWorkspace` from scrach from Python
script, because :class:`mantid.api.ISplittersWorkspace` inherits from
:ref:`TableWorkspace <Table Workspaces>`.
Generating inexplicit filters
-----------------------------
:ref:`algm-FilterByTime` and :ref:`algm-FilterByLogValue` generate event filters during execution.
* :ref:`algm-FilterByTime` generates a set of :ref:`SplittingInterval`
according to user-specified setup for time splicing;
* :ref:`algm-FilterByLogValue` generates a set of
:ref:`SplittingInterval` according to the value of a specific sample
log.
:ref:`algm-GenerateEventsFilter` and :ref:`algm-FilterEvents` vs :ref:`algm-FilterByTime` and :ref:`algm-FilterByLogValue`
--------------------------------------------------------------------------------------------------------------------------
* If :ref:`algm-GenerateEventsFilter` and :ref:`algm-FilterEvents` are
set up correctly, they can have the same functionality as
:ref:`algm-FilterByTime` and :ref:`algm-FilterByLogValue`.
* :ref:`algm-FilterEvents` is able to filter neutron events by either
their pulse times or their absolute times. An neutron event's
absolute time is the summation of its pulse time and TOF.
* :ref:`algm-FilterByLogValue` and :ref:`algm-FilterByTime` can only
split neutron events by their pulse time.
Types of events filters
=======================
Filtering by :ref:`SplittingInterval`
-------------------------------------
:ref:`SplittingInterval` is an individual class to indicate an
independent time splitter. Any event can be filtered by a
:ref:`SplittingInterval` object.
:ref:`SplittersWorkspace` is a :ref:`TableWorkspace <Table
Workspaces>` that stors a set of :ref:`SplittingInterval`.
Filtering by duplicate entries/booleans
---------------------------------------
Duplicate entries in a :ref:`TimeSeriesProperty` and boolean type of
:ref:`TimeSeriesProperty` are used in MantidPlot too to serve as time
splitters.
These two are applied in the MantidPlot log viewing functionality and
unfortunately intrudes into :ref:`TimeSeriesProperty`.
As time splitters are better to be isolated from logs, which are
recorded in :ref:`TimeSeriesProperty`, it is not
recommended to set up event filters by this approach.
Implicit filters
----------------
:ref:`algm-FilterByTime` and :ref:`algm-FilterByLogValue` internally
generate event filters during execution that are not exposed to the
user. These algorithms can only split the neutron events by pulse
time and do not provide the equivalent of a ``FastLog=True`` option.
Explicit filters
----------------
:ref:`algm-FilterEvents` takes either a :class:`SplittersWorkspace
<mantid.api.ISplittersWorkspace>`, :ref:`TableWorkspace <Table
Workspaces>`, or :ref:`MatrixWorkspace <MatrixWorkspace>` as the
``SplittersWorkspace``. The events are split into output workspaces
according to the times that they arrive detectors.
:ref:`GenerateEventsFilter <algm-GenerateEventsFilter>` will create a
:class:`SplittersWorkspace <mantid.api.ISplittersWorkspace>` based on
its various options. This result can be supplied as the
``SplittersWorkspace`` input property of ref:`algm-FilterEvents`. It
will also generate an ``InformationWorkspace`` which can be passed
along to :ref:`GenerateEventsFilter <algm-GenerateEventsFilter>`.
Depending on the parameters in :ref:`GenerateEventsFilter
<algm-GenerateEventsFilter>`, the events will be filtered based on
their pulse times or their absolute times. An neutron event's
absolute time is the summation of its pulse time and TOF.
Custom event filters
====================
Sometimes one wants to filter events based on arbitrary conditions. In
this case, one needs to go beyond what existing algorithms can do. For
this, one must generate their own splitters workspace. The workspace
is generally 3 columns, with the first two being start and stop times
and the third being the workspace index to put the events into. For
filtering with time relative to the start of the run, the first two
columns are ``float``. To specify the times as absolute, in the case
of filtering files that will be summed together, the first two columns
should be ``int64``. For both of the examples below, the filter
workspaces are created using the following function:
.. code-block:: python
def create_table_workspace(table_ws_name, column_def_list):
CreateEmptyTableWorkspace(OutputWorkspace=table_ws_name)
table_ws = mtd[table_ws_name]
for col_tup in column_def_list:
data_type = col_tup[0]
col_name = col_tup[1]
table_ws.addColumn(data_type, col_name)
return table_ws
Relative time
-------------
The easiest way to generate a custom event filter is to make one
relative to the start time of the run or relative to a specified
epoch. As the times in the table are seconds, a table can be created
and used
.. code-block:: python
filter_rel = create_table_workspace('custom_relative', [('float', 'start'), ('float', 'stop'), ('str', 'target')])
filter_rel.addRow((0,9500, '0'))
filter_rel.addRow((9500,19000, '1'))
FilterEvents(InputWorkspace='ws', SplitterWorkspace=filter_rel,
GroupWorkspaces=True, OutputWorkspaceBaseName='relative', RelativeTime=True)
This will generate an event filter relative to the start of the
run. Specifying the ``FilterStartTime`` in :ref:`FilterEvents
<algm-FilterEvents>`, one can specify a different time that filtering
will be relative to.
Absolute time
-------------
If instead a custom filter is to be created with absolute time, the
time must be processed somewhat to go into the table workspace. Much of the
.. code-block:: python
abs_times = [datetime64('2014-12-12T09:11:22.538096666'), datetime64('2014-12-12T11:45:00'), datetime64('2014-12-12T14:14:00')]
# convert to time relative to GPS epoch
abs_times = [time - datetime64('1990-01-01T00:00') for time in abs_times]
# convert to number of seconds
abs_times = [float(time / timedelta64(1, 's')) for time in abs_times]
filter_abs = create_table_workspace('custom_absolute', [('float', 'start'), ('float', 'stop'), ('str', 'target')])
filter_abs.addRow((abs_times[0], abs_times[1], '0'))
filter_abs.addRow((abs_times[1], abs_times[2], '1'))
FilterEvents(InputWorkspace='PG3_21638', SplitterWorkspace=filter_abs,
GroupWorkspaces=True, OutputWorkspaceBaseName='absolute', RelativeTime=False)
Be warned that specifying ``RelativeTime=True`` with a table full of
absolute times will almost certainly generate output workspaces
without any events in them.
.. categories:: Concepts
......@@ -84,6 +84,7 @@ Improvements
- :ref:`GroupDetectors <algm-GroupDetectors>` now takes masked bins correctly into account when processing histogram workspaces.
- :ref:`SaveNexusProcessed <algm-SaveNexusProcessed>` and :ref:`LoadNexusProcessed <algm-LoadNexusProcessed>` can now save and load a ``MaskWorkspace``.
- :ref:`FitPeaks <algm-FitPeaks>` can output parameters' uncertainty (fitting error) in an optional workspace.
- The documentation in :ref:`EventFiltering` and :ref:`FilterEvents <algm-FilterEvents>` have been extensively rewritten to aid in understanding what the code does.
Bugfixes
########
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment