Commit 925316f5 authored by Zhou, Wenduo's avatar Zhou, Wenduo
Browse files

Check out the docs from branch WZ_onboard_doc.

parent 4f912baf
Pipeline #169014 passed with stages
in 17 minutes and 38 seconds
......@@ -2,6 +2,7 @@ Guide to Contributing
=====================
Contributions to this project are welcome. All contributors agree to the following:
- It is assumed that the contributor is an ORNL employee and belongs to the development team. Thus the following instructions are specific to ORNL development team's process.
- You have permission and any required rights to submit your contribution.
- Your contribution is provided under the license of this project and may be redistributed as such.
- All contributions to this project are public.
......@@ -13,7 +14,25 @@ Getting access to the main project
Direct commit access to the project is currently restricted to core developers.
All other contributions should be done through pull requests.
Development procedure
---------------------
1. A developer is assigned with a task during neutron status meeting and changes the task's status to **In Progress**.
2. The developer creates a branch off *next* and completes the task in this branch.
3. The developer creates a merge request (MR) off *next*.
4. The developer asks for another developer as a reviewer to review the MR. An MR can only be approved and merged by the reviewer.
5. The developer changes the task’s status to **Complete** and closes the associated issue.
Contacting the Team
-------------------
The best mechanism for contacting the team is to "open a ticket".
Otherwise, please email us at, Peter Peterson, at petersonpf <at> ornl DOT gov, Jay Jay Billings, billingsjj <at> ornl DOT gov
The best mechanism for a user to request a change is to contact the SANS CIS.
Please email `Yingrui Shang`_ with your request.
.. _Yingrui Shang: shangy@ornl.gov
A change needs to be in the form of a:
- Story for any enhancement request
- Defect for any bug fix request.
********************
Onboarding Checklist
********************
This check list is to assist developers who are new to the SANS backend software project as the onboarding process.
How to get started
##################
Accounts setup
**************
A new developer, who joins the SANS backend software project, shall need to have or request access to the environments and tools. There is a contact person for requesting access in each section.
* Project managers
* John Hetrick
* Peter F. Peterson
* drtSANS repository
* URL: https://code.ornl.gov/ns-hfir-scse/sans/sans-backend/
* You will need permission to commit changes
* If you don't have access, please contact the project managers(s).
* Slack channels/Onboarding buddy
* ornlccsd.slack.com # sae-neutrons
Please contact the Project Managers for the access to the slack channel.
The project managers will know who to send an email to requesting access to our neutron slack channels.
There may be more than one and if the person is outside CSMD that is a special invite.
* SNS analysis cluster account and integration data access
drtSANS is designed for users to process SANS data on SNS analysis cluster or
on SNS jupyter hub.
Therefore, drtSANS shall be tested on SNS analysis cluster.
* Analysis cluster web access URL: https://analysis.sns.gov
* Jupyter hub access URL: https://jupyter.sns.gov/hub/login
* Analysis cluter server address: analysis.sns.gov
* Portions of the test suite requires read and write access to `/SNS/EQSANS/shared/sans-backend` on SNS analysis cluster.
* If you have any issue with accessiblity, please contact the project managers(s).
* Access to SNS archive data
SNS and HFIR SANS data files are relatively large.
Developers are expected to work with the raw data files residing on the SNS/HFIR data archive.
Therefore it is necessary for drtSANS developers to access `/SNS/EQSANS`, `/HFIR/CG2/` and `/HFIR/CG3`.
* Access can be required by emailing SNS Linux support (linuxsupport@ornl.gov) and cc-ing neutron project managers and your group leader.
* Status meeting
* All developers shall attend morning neutron status meeting.
* Contact project managers to be invited.
Set-up for development in a virtual environment
***********************************************
Refer to section *Set-up for development in a virtual environment* in :doc:`README_developer <README_developer>`.
Building the documentation
##########################
.. _building_docs:
The site can be build directly using
.. code-block:: shell
$ sphinx-build -b html docs/ build/sphinx/html
or
.. code-block:: shell
$ python setup.py build_sphinx
Development procedure
#####################
How to develop codes in drtSANS shall follow the instruction in `CONTRIBUTION <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/CONTRIBUTING.rst>`_.
..
1. A developer is assigned with a task during neutron status meeting and changes the task's status to **In Progress**.
2. The developer creates a branch off *next* and completes the task in this branch.
3. The developer creates a merge request (MR) off *next*.
4. The developer asks for another developer as a reviewer to review the MR. An MR can only be approved and merged by the reviewer.
5. The developer changes the task’s status to **Complete**.
Test Driven Development (TDD)
#############################
* Test driven Development
drtSANS development follows `test-driven development <https://en.wikipedia.org/wiki/Test-driven_development>`_ (TDD) process [1].
All software requirements for SANS data reduction shall be converted to test cases before software is fully developed.
All software developments are tracked by repeatedly testing the software against all test cases.
* Unit test
All methods and modules shall have unit tests implemented.
Unit tests are located in `repo/tests/unit/new <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/tree/next/tests/unit/new>`_.
A unit test shall be created in the corresponding directory to the method or module that it tests against.
Examples:
* `drtsans/resolution.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/drtsans/resolution.py>`_ and `tests/unit/new/drtsans/test_resolution.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/tests/unit/new/drtsans/test_resolution.py>`_.
* `drtsans/tof/eqsans/incoherence_correction_q1d.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/drtsans/tof/eqsans/incoherence_correction_1d.py>`_ and `tests/unit/new/drtsans/tof/eqsans/test_incoherence_correction_q1d.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/tests/unit/new/drtsans/tof/eqsans/test_incoherence_correction_q1d.py>`_.
* Integration test
Integration test will test the combination of Individual modules and methods.
Integration tests can be
* general for all instrument, for instance `tests/integration/new/drtsans/test_stitch.py`.
* specific to a suite of similar instruments, for instance `tests/integration/new/drtsans/mono/test_transmission.py` for all mono-wavelength instruments including Bio-SANS and GP-SANS.
* specific to an individual instrument, for instance, `tests/integration/new/drtsans/mono/gpsans/test_find_beam_center.py` for GP-SANS and
`tests/integration/new/drtsans/tof/eqsans/test_apply_solid_angle.py` for EQ-SANS.
* Testing data location
Refer to section *Testing data location* in :doc:`README_developer <README_developer>`.
* CI/CD
Glossaries
##########
* SANS
Small-angle neutron scattering (SANS) is an experimental technique that uses elastic neutron scattering at small scattering angles to investigate the structure of various substances at a mesoscopic scale of about 1–100 nm.
* https://en.wikipedia.org/wiki/Small-angle_neutron_scattering
* https://www.nist.gov/ncnr/neutron-instruments/small-angle-neutron-scattering-sans
* drtSANS
Data reduction tool for small angle neutron scattering.
Required libraries
##################
* numpy: https://numpy.org/
* Mantid: https://www.mantidproject.org/, https://github.com/mantidproject/mantid
* Others: h5py, docutils, jsonschema, lmfit, matplotlib, mpld3, numexpr, pandas, sortedcontainers, tinydb, ipywidgets
* For unit and integration tests: pytest, pytest-xdist
* For documentation: sphinx, sphinxcontrib-napoleon,
* For linting and formatting: autopep8, flake8, pylint
......@@ -5,6 +5,7 @@ drt-sans
Data Reduction Toolkit for Small Angle Neutron Scattering
This packages is a collection of functionality for reducing SANS data developed in collaboration with the instrument scientists at the High Flux Isotope Reactor (HFIR) and Spallation Neutron Source (SNS) at Oak Ridge National Laboratory.
While much of the functionality is generic, this implementation is aimed at reducing data from BIOSANS, EQSANS, and GPSANS.
As appropriate, this work is an abstraction layer on top of the mantid project.
......@@ -14,7 +15,7 @@ As appropriate, this work is an abstraction layer on top of the mantid project.
Usage from provided front-ends
------------------------------
For end users go to `next version <http://shaman.ornl.gov/>`_ or
For end users go to
`QA version <http://scse-ui.ornl.gov:8080/>`_
Use `jupyter <https://jupyter.sns.gov/>`_ to have a play with the code.
......@@ -66,141 +67,3 @@ You must download the wrapper script from the above link as the build process mo
-----------------------------------------------
Set-up for development in a virtual environment
-----------------------------------------------
This is the instructions for a person who is developing *both*
drt-sans and mantid in tandem. It assumes that you are using a virtual
environment and have a local build of mantid. As one caveat, for this
method, mantid must be build against the same version of python being
used in the virtual environment
1. Checkout the code.
.. code-block:: shell
$ git clone git@code.ornl.gov:sns-hfir-scse/sans/sans-backend.git
$ cd sans-backend
2. Create the virtual environment and activate it. The
``--system-site-packages`` argument lets it use things installed on
the system as a whole for compiling mantid
.. code-block:: shell
$ virtualenv -p $(which python3) --system-site-packages .venv
$ source .venv/bin/activate
3. Add the built version of mantid to the python path in the virtual
environment
.. code-block:: shell
$ python ~/build/mantid/bin/AddPythonPath.py
4. Install the requirements for running the code
.. code-block:: shell
$ pip install -r requirements.txt -r requirements_dev.txt
5. Install the code in ``develop`` mode.
.. code-block:: shell
$ python setup.py develop
6. Try it out. Start ``python`` and try
.. code-block:: python
import mantid
import drtsans
Verify you can run the unit tests:
.. code-block:: shell
$ python -m pytest tests/unit/new/
7. Be done. Deactivate the virtual environment using
.. code-block:: shell
$ deactivate
As an alternative, you can use `direnv <https://direnv.net>`_ to do a
fair amount of the work. Unfortunately, it doesn't currently handle
``--system-site-packages`` correctly so you'll have to work around
it. Create the virtual environment using
.. code-block:: shell
$ virtualenv -p $(which python3) --system-site-packages .direnv/python-$(python3 -c "import platform as p;print(p.python_version())")
Then you create a file ``.envrc`` in your source directory
.. code-block:: shell
$ echo "layout python3" > .envrc
Finally, tell direnv that you want it to work in this directory
.. code-block:: shell
$ direnv allow
Then follow steps 3-6 from above. After this, the virtual environment
with load when you enter the source tree, and unload when you leave.
-----------------
Running the tests
-----------------
.. _running_tests:
The tests for this project are all written using `pytest <https://docs.pytest.org/en/latest>`_.
The `build pipeline <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/blob/next/.gitlab-ci.yml>`_ currently `runs the unit tests and integration tests <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/blob/next/test_job.sh>`_ separately using
.. code-block:: shell
$ python -m pytest tests/unit/new/
$ python -m pytest tests/integration/new/
This is one of the ways `pytest allows for selecting tests <https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`_.
Specifying a directory or file will run all tests within that directory (recursively) or file.
Specifying a regular expression using ``-k`` will select all tests that match the regular expression independent of where they are defined
.. code-block:: shell
$ python -m pytest -k test_samplelogs
To run an individual test within an individual file add ``::`` to the filename to specify the test
.. code-block:: shell
$ python -m pytest tests/unit/new/drtsans/tof/eqsans/test_beam_finder.py::test_center_detector
--------------------------
Building the documentation
--------------------------
.. _building_docs:
The site can be build directly using
.. code-block:: shell
$ sphinx-build -b html docs/ build/sphinx/html
or
.. code-block:: shell
$ python setup.py build_sphinx
------------
Contributing
------------
Contributing is done through merge requests of code that you have the permission to add.
See `CONTRIBUTING.rst <CONTRIBUTING.rst>`_ for more information.
.. _my-reference-label: Readmedev
========
drt-sans
========
Data Reduction Toolkit for Small Angle Neutron Scattering
This packages is a collection of functionality for reducing SANS data developed in collaboration with the instrument scientists at the High Flux Isotope Reactor (HFIR) and Spallation Neutron Source (SNS) at Oak Ridge National Laboratory.
While much of the functionality is generic, this implementation is aimed at reducing data from BIOSANS, EQSANS, and GPSANS.
As appropriate, this work is an abstraction layer on top of the mantid project.
**This is a python3 only package.**
-----------------------------------------------
Set-up for development in a virtual environment
-----------------------------------------------
These are the instructions for a person who is developing *both*
drt-sans and mantid in tandem. It assumes that you are using a virtual
environment and have a local build of mantid. As one caveat, for this
method, mantid must be build against the same version of python being
used in the virtual environment
1. Checkout the code.
.. code-block:: shell
$ git clone git@code.ornl.gov:sns-hfir-scse/sans/sans-backend.git
$ cd sans-backend
2. Create the development environment.
There are two approaches to create development environment.
a) Create the virtual environment and activate it.
* It requires a local build of Mantid as the prerequisite.
* How to build `Mantid <https://developer.mantidproject.org/BuildingWithCMake.html>`_ .
* The ``--system-site-packages`` argument gives the virtual environment access to the same system site-packages
that were used for compiling `mantid <https://developer.mantidproject.org/BuildingWithCMake.html>`_.
.. code-block:: shell
$ virtualenv -p $(which python3) --system-site-packages .venv
$ source .venv/bin/activate
As an alternative, you can use `direnv <https://direnv.net>`_ to do a
fair amount of the work. Unfortunately, it doesn't currently handle
``--system-site-packages`` correctly so you'll have to work around
it. Create the virtual environment using
.. code-block:: shell
$ virtualenv -p $(which python3) --system-site-packages .direnv/python-$(python3 -c "import platform as p;print(p.python_version())")
Then you create a file ``.envrc`` in your source directory
.. code-block:: shell
$ echo "layout python3" > .envrc
Finally, tell direnv that you want it to work in this directory
.. code-block:: shell
$ direnv allow
Then follow rest steps. After those, the virtual environment
with load when you enter the source tree, and unload when you leave.
* Add the built version of mantid to the python path in the virtual environment
.. code-block:: shell
$ python ~/build/mantid/bin/AddPythonPath.py
* Install the requirements for running the code
.. code-block:: shell
$ pip install -r requirements.txt -r requirements_dev.txt
b) Create a conda environment and activate it.
* It will use Mantid’s conda package.
* miniconda/conda is required.
* Linux command:
.. code-block:: shell
$ conda config --add channels conda-forge
$ conda config --add channels mantid
$ conda create -n mydrtsans python=3.7
$ conda activate mydrtsans
$ conda install -c mantid/label/nightly mantid-framework=6 --file requirements.txt --file requirements_dev.txt
**Warning**: if you create the conda environment on SNS’s analysis cluster, avoid naming your environment as ‘drtsans’,
‘drtsans-qa’ and ‘drtsans-dev’, which are reserved.
3. Install the code in ``develop`` mode.
.. code-block:: shell
$ python setup.py develop
4. Try it out. Start ``python`` and try
.. code-block:: python
import mantid
import drtsans
Verify you can run the unit tests:
.. code-block:: shell
$ python -m pytest tests/unit/new/
$ python -m pytest tests/integration/new/
Some unit and integration tests require testing data. Thus these tests can be either run on
* SNS analysis cluster or
* Local computer with mount to `/SNS/` archive. Here is the `instruction <https://code.ornl.gov/pf9/sns-mounts>`_ to mount SNS data directories.
5. When done, deactivate the virtual environment using
.. code-block:: shell
$ deactivate
for virtual environment. Or
.. code-block:: shell
$ deactivate
for conda environment.
-----------------
Running the tests
-----------------
.. _running_tests:
The tests for this project are all written using `pytest <https://docs.pytest.org/en/latest>`_.
The `build pipeline <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/blob/next/.gitlab-ci.yml>`_ currently `runs the unit tests and integration tests <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/blob/next/test_job.sh>`_ separately using
.. code-block:: shell
$ python -m pytest tests/unit/new/
$ python -m pytest tests/integration/new/
This is one of the ways `pytest allows for selecting tests <https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`_.
Specifying a directory or file will run all tests within that directory (recursively) or file.
Specifying a regular expression using ``-k`` will select all tests that match the regular expression independent of where they are defined
.. code-block:: shell
$ python -m pytest -k test_samplelogs
To run an individual test within an individual file add ``::`` to the filename to specify the test
.. code-block:: shell
$ python -m pytest tests/unit/new/drtsans/tof/eqsans/test_beam_finder.py::test_center_detector
--------------------------
Building the documentation
--------------------------
.. _building_docs:
The site can be built directly using
.. code-block:: shell
$ sphinx-build -b html docs/ build/sphinx/html
or
.. code-block:: shell
$ python setup.py build_sphinx
------------
Contributing
------------
Contributing is done through merge requests of code that you have the permission to add.
See `CONTRIBUTING.rst <CONTRIBUTING.rst>`_ for more information.
-----------------------------
Test Driven Development (TDD)
-----------------------------
* Test driven Development
drtSANS development follows `test-driven development <https://en.wikipedia.org/wiki/Test-driven_development>`_ (TDD) process [1].
All software requirements for SANS data reduction shall be converted to test cases before software is fully developed.
All software developments are tracked by repeatedly testing the software against all test cases.
* Unit test
All methods and modules shall have unit tests implemented.
Unit tests are located in `repo/tests/unit/new <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/tree/next/tests/unit/new>`_.
A unit test shall be created in the corresponding directory to the method or module that it tests against.
Examples:
* `drtsans/resolution.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/drtsans/resolution.py>`_ and `tests/unit/new/drtsans/test_resolution.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/tests/unit/new/drtsans/test_resolution.py>`_.
* `drtsans/tof/eqsans/incoherence_correction_q1d.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/drtsans/tof/eqsans/incoherence_correction_1d.py>`_ and `tests/unit/new/drtsans/tof/eqsans/test_incoherence_correction_q1d.py <https://code.ornl.gov/sns-hfir-scse/sans/sans-backend/-/blob/next/tests/unit/new/drtsans/tof/eqsans/test_incoherence_correction_q1d.py>`_.
* Integration test
Integration test will test the combination of Individual modules and methods.
Integration tests can be
* general for all instrument, for instance `tests/integration/new/drtsans/test_stitch.py`.
* specific to a suite of similar instruments, for instance `tests/integration/new/drtsans/mono/test_transmission.py` for all mono-wavelength instruments including Bio-SANS and GP-SANS.
* specific to an individual instrument, for instance, `tests/integration/new/drtsans/mono/gpsans/test_find_beam_center.py` for GP-SANS and
`tests/integration/new/drtsans/tof/eqsans/test_apply_solid_angle.py` for EQ-SANS.
* Testing data location
Testing data are located on SNS data archive: `/SNS/EQSANS/shared/sans-backend/data`.
Testing data for specific instruments have specific locations:
- EQSANS: `/SNS/EQSANS/shared/sans-backend/data/new/ornl/sans/sns/eqsans/`
- Bio-SANS: `/SNS/EQSANS/shared/sans-backend/data/new/ornl/sans/hfir/biosans/`
- GP-SANS: `/SNS/EQSANS/shared/sans-backend/data/new/ornl/sans/hfir/gpsans/`
Data files are referenced in the tests via the reference_dir pytest fixture.
For instance, reference_dir.new.eqsans points to /SNS/EQSANS/shared/sans-backend/data/new/ornl/sans/sns/eqsans/
------------------
Required libraries
------------------
* numpy: https://numpy.org/
* Mantid: https://www.mantidproject.org/, https://github.com/mantidproject/mantid
* Others: h5py, docutils, jsonschema, lmfit, matplotlib, mpld3, numexpr, pandas, sortedcontainers, tinydb, ipywidgets
* For unit and integration tests: pytest, pytest-xdist
* For documentation: sphinx, sphinxcontrib-napoleon,
* For linting and formatting: autopep8, flake8, pylint
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment