Contributions to this project are welcome. All contributors agree to the following:
- It is assumed that the contributor is an ORNL employee and belongs to the development team. Thus the following instructions are specific to ORNL development team's process.
- You have permission and any required rights to submit your contribution.
- Your contribution is provided under the license of this project and may be redistributed as such.
- All contributions to this project are public.
...
...
@@ -13,7 +14,25 @@ Getting access to the main project
Direct commit access to the project is currently restricted to core developers.
All other contributions should be done through pull requests.
Development procedure
---------------------
1. A developer is assigned with a task during neutron status meeting and changes the task's status to **In Progress**.
2. The developer creates a branch off *next* and completes the task in this branch.
3. The developer creates a merge request (MR) off *next*.
4. The developer asks for another developer as a reviewer to review the MR. An MR can only be approved and merged by the reviewer.
5. The developer changes the task’s status to **Complete** and closes the associated issue.
Contacting the Team
-------------------
The best mechanism for contacting the team is to "open a ticket".
Otherwise, please email us at, Peter Peterson, at petersonpf <at> ornl DOT gov, Jay Jay Billings, billingsjj <at> ornl DOT gov
The best mechanism for a user to request a change is to contact the SANS CIS.
This check list is to assist developers who are new to the SANS backend software project as the onboarding process.
How to get started
##################
Accounts setup
**************
A new developer, who joins the SANS backend software project, shall need to have or request access to the environments and tools. There is a contact person for requesting access in each section.
* Analysis cluter server address: analysis.sns.gov
* Portions of the test suite requires read and write access to `/SNS/EQSANS/shared/sans-backend` on SNS analysis cluster.
* If you have any issue with accessiblity, please contact the project managers(s).
* Access to SNS archive data
SNS and HFIR SANS data files are relatively large.
Developers are expected to work with the raw data files residing on the SNS/HFIR data archive.
Therefore it is necessary for drtSANS developers to access `/SNS/EQSANS`, `/HFIR/CG2/` and `/HFIR/CG3`.
* Access can be required by emailing SNS Linux support (linuxsupport@ornl.gov) and cc-ing neutron project managers and your group leader.
* Status meeting
* All developers shall attend morning neutron status meeting.
* Contact project managers to be invited.
Set-up for development in a virtual environment
***********************************************
Refer to section *Set-up for development in a virtual environment* in :doc:`README_developer <README_developer>`.
Building the documentation
##########################
.. _building_docs:
The site can be build directly using
.. code-block:: shell
$ sphinx-build -b html docs/ build/sphinx/html
or
.. code-block:: shell
$ python setup.py build_sphinx
Development procedure
#####################
How to develop codes in drtSANS shall follow the instruction in `CONTRIBUTION <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/CONTRIBUTING.rst>`_.
..
1. A developer is assigned with a task during neutron status meeting and changes the task's status to **In Progress**.
2. The developer creates a branch off *next* and completes the task in this branch.
3. The developer creates a merge request (MR) off *next*.
4. The developer asks for another developer as a reviewer to review the MR. An MR can only be approved and merged by the reviewer.
5. The developer changes the task’s status to **Complete**.
Test Driven Development (TDD)
#############################
* Test driven Development
drtSANS development follows `test-driven development <https://en.wikipedia.org/wiki/Test-driven_development>`_ (TDD) process [1].
All software requirements for SANS data reduction shall be converted to test cases before software is fully developed.
All software developments are tracked by repeatedly testing the software against all test cases.
* Unit test
All methods and modules shall have unit tests implemented.
Unit tests are located in `repo/tests/unit/new <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/tree/next/tests/unit/new>`_.
A unit test shall be created in the corresponding directory to the method or module that it tests against.
Examples:
* `drtsans/resolution.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/drtsans/resolution.py>`_ and `tests/unit/new/drtsans/test_resolution.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/tests/unit/new/drtsans/test_resolution.py>`_.
* `drtsans/tof/eqsans/incoherence_correction_q1d.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/drtsans/tof/eqsans/incoherence_correction_1d.py>`_ and `tests/unit/new/drtsans/tof/eqsans/test_incoherence_correction_q1d.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/tests/unit/new/drtsans/tof/eqsans/test_incoherence_correction_q1d.py>`_.
* Integration test
Integration test will test the combination of Individual modules and methods.
Integration tests can be
* general for all instrument, for instance `tests/integration/new/drtsans/test_stitch.py`.
* specific to a suite of similar instruments, for instance `tests/integration/new/drtsans/mono/test_transmission.py` for all mono-wavelength instruments including Bio-SANS and GP-SANS.
* specific to an individual instrument, for instance, `tests/integration/new/drtsans/mono/gpsans/test_find_beam_center.py` for GP-SANS and
`tests/integration/new/drtsans/tof/eqsans/test_apply_solid_angle.py` for EQ-SANS.
* Testing data location
Refer to section *Testing data location* in :doc:`README_developer <README_developer>`.
* CI/CD
Glossaries
##########
* SANS
Small-angle neutron scattering (SANS) is an experimental technique that uses elastic neutron scattering at small scattering angles to investigate the structure of various substances at a mesoscopic scale of about 1–100 nm.
Data Reduction Toolkit for Small Angle Neutron Scattering
This packages is a collection of functionality for reducing SANS data developed in collaboration with the instrument scientists at the High Flux Isotope Reactor (HFIR) and Spallation Neutron Source (SNS) at Oak Ridge National Laboratory.
While much of the functionality is generic, this implementation is aimed at reducing data from BIOSANS, EQSANS, and GPSANS.
As appropriate, this work is an abstraction layer on top of the mantid project.
...
...
@@ -14,7 +15,7 @@ As appropriate, this work is an abstraction layer on top of the mantid project.
Usage from provided front-ends
------------------------------
For end users go to `next version <http://shaman.ornl.gov/>`_ or
For end users go to
`QA version <http://scse-ui.ornl.gov:8080/>`_
Use `jupyter <https://jupyter.sns.gov/>`_ to have a play with the code.
...
...
@@ -66,141 +67,3 @@ You must download the wrapper script from the above link as the build process mo
-----------------------------------------------
Set-up for development in a virtual environment
-----------------------------------------------
This is the instructions for a person who is developing *both*
drt-sans and mantid in tandem. It assumes that you are using a virtual
environment and have a local build of mantid. As one caveat, for this
method, mantid must be build against the same version of python being
Then you create a file ``.envrc`` in your source directory
.. code-block:: shell
$ echo "layout python3" > .envrc
Finally, tell direnv that you want it to work in this directory
.. code-block:: shell
$ direnv allow
Then follow steps 3-6 from above. After this, the virtual environment
with load when you enter the source tree, and unload when you leave.
-----------------
Running the tests
-----------------
.. _running_tests:
The tests for this project are all written using `pytest <https://docs.pytest.org/en/latest>`_.
The `build pipeline <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/blob/next/.gitlab-ci.yml>`_ currently `runs the unit tests and integration tests <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/blob/next/test_job.sh>`_ separately using
.. code-block:: shell
$ python -m pytest tests/unit/new/
$ python -m pytest tests/integration/new/
This is one of the ways `pytest allows for selecting tests <https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`_.
Specifying a directory or file will run all tests within that directory (recursively) or file.
Specifying a regular expression using ``-k`` will select all tests that match the regular expression independent of where they are defined
.. code-block:: shell
$ python -m pytest -k test_samplelogs
To run an individual test within an individual file add ``::`` to the filename to specify the test
Data Reduction Toolkit for Small Angle Neutron Scattering
This packages is a collection of functionality for reducing SANS data developed in collaboration with the instrument scientists at the High Flux Isotope Reactor (HFIR) and Spallation Neutron Source (SNS) at Oak Ridge National Laboratory.
While much of the functionality is generic, this implementation is aimed at reducing data from BIOSANS, EQSANS, and GPSANS.
As appropriate, this work is an abstraction layer on top of the mantid project.
**This is a python3 only package.**
-----------------------------------------------
Set-up for development in a virtual environment
-----------------------------------------------
These are the instructions for a person who is developing *both*
drt-sans and mantid in tandem. It assumes that you are using a virtual
environment and have a local build of mantid. As one caveat, for this
method, mantid must be build against the same version of python being
**Warning**: if you create the conda environment on SNS’s analysis cluster, avoid naming your environment as ‘drtsans’,
‘drtsans-qa’ and ‘drtsans-dev’, which are reserved.
3. Install the code in ``develop`` mode.
.. code-block:: shell
$ python setup.py develop
4. Try it out. Start ``python`` and try
.. code-block:: python
import mantid
import drtsans
Verify you can run the unit tests:
.. code-block:: shell
$ python -m pytest tests/unit/new/
$ python -m pytest tests/integration/new/
Some unit and integration tests require testing data. Thus these tests can be either run on
* SNS analysis cluster or
* Local computer with mount to `/SNS/` archive. Here is the `instruction <https://code.ornl.gov/pf9/sns-mounts>`_ to mount SNS data directories.
5. When done, deactivate the virtual environment using
.. code-block:: shell
$ deactivate
for virtual environment. Or
.. code-block:: shell
$ deactivate
for conda environment.
-----------------
Running the tests
-----------------
.. _running_tests:
The tests for this project are all written using `pytest <https://docs.pytest.org/en/latest>`_.
The `build pipeline <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/blob/next/.gitlab-ci.yml>`_ currently `runs the unit tests and integration tests <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/blob/next/test_job.sh>`_ separately using
.. code-block:: shell
$ python -m pytest tests/unit/new/
$ python -m pytest tests/integration/new/
This is one of the ways `pytest allows for selecting tests <https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`_.
Specifying a directory or file will run all tests within that directory (recursively) or file.
Specifying a regular expression using ``-k`` will select all tests that match the regular expression independent of where they are defined
.. code-block:: shell
$ python -m pytest -k test_samplelogs
To run an individual test within an individual file add ``::`` to the filename to specify the test
Contributing is done through merge requests of code that you have the permission to add.
See `CONTRIBUTING.rst <CONTRIBUTING.rst>`_ for more information.
-----------------------------
Test Driven Development (TDD)
-----------------------------
* Test driven Development
drtSANS development follows `test-driven development <https://en.wikipedia.org/wiki/Test-driven_development>`_ (TDD) process [1].
All software requirements for SANS data reduction shall be converted to test cases before software is fully developed.
All software developments are tracked by repeatedly testing the software against all test cases.
* Unit test
All methods and modules shall have unit tests implemented.
Unit tests are located in `repo/tests/unit/new <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/tree/next/tests/unit/new>`_.
A unit test shall be created in the corresponding directory to the method or module that it tests against.
Examples:
* `drtsans/resolution.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/drtsans/resolution.py>`_ and `tests/unit/new/drtsans/test_resolution.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/tests/unit/new/drtsans/test_resolution.py>`_.
* `drtsans/tof/eqsans/incoherence_correction_q1d.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/drtsans/tof/eqsans/incoherence_correction_1d.py>`_ and `tests/unit/new/drtsans/tof/eqsans/test_incoherence_correction_q1d.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/tests/unit/new/drtsans/tof/eqsans/test_incoherence_correction_q1d.py>`_.
* Integration test
Integration test will test the combination of Individual modules and methods.
Integration tests can be
* general for all instrument, for instance `tests/integration/new/drtsans/test_stitch.py`.
* specific to a suite of similar instruments, for instance `tests/integration/new/drtsans/mono/test_transmission.py` for all mono-wavelength instruments including Bio-SANS and GP-SANS.
* specific to an individual instrument, for instance, `tests/integration/new/drtsans/mono/gpsans/test_find_beam_center.py` for GP-SANS and
`tests/integration/new/drtsans/tof/eqsans/test_apply_solid_angle.py` for EQ-SANS.
* Testing data location
Testing data are located on SNS data archive: `/SNS/EQSANS/shared/sans-backend/data`.
Testing data for specific instruments have specific locations:
