Alex_Project issueshttps://code.ornl.gov/y8z/alex_project/-/issues2022-03-04T21:06:45Zhttps://code.ornl.gov/y8z/alex_project/-/issues/29[Task] Implement a post processing tab in ADDIE interface for Mantid2022-03-04T21:06:45ZZhang, Yuanpengzhangy3@ornl.gov[Task] Implement a post processing tab in ADDIE interface for Mantid> In the `Processing` tab, when clicking on `Launch Reduction` button, we expect to kick off the reduction and obtain reduced data stored in nexus file format. Though we are not there yet, we will. But I suppose this won't happen during ...> In the `Processing` tab, when clicking on `Launch Reduction` button, we expect to kick off the reduction and obtain reduced data stored in nexus file format. Though we are not there yet, we will. But I suppose this won't happen during this summer since there are indeed quite a few missing capabilities in the underlying reduction engine. This means, we can prepare the input json file (which will contain information concerning those missing corrections) but we won't be able to kick off the reduction. However, this does not prevent us from moving forward with the GUI design, as sketched next.
A separate tab is needed in ADDIE interface for Mantid, for extracting bank-by-bank data from the nexus file generated with data reduction. Here in the following link we can find the GUI design, [https://balsamiq.cloud/sd2jtfw/ptaj9k7](https://balsamiq.cloud/sd2jtfw/ptaj9k7) - see the second wireframe (i.e. slide).
> Concerning implementation, we can refer to the design and implementation of the `Rietveld` tab.Alex_SULI_07-12-2021Zhang, Yuanpengzhangy3@ornl.govGreenhalgh, AlexanderZhang, Yuanpengzhangy3@ornl.govhttps://code.ornl.gov/y8z/alex_project/-/issues/1[Enabler] Checklist for Alex project2022-03-04T15:38:51ZZhang, Yuanpengzhangy3@ornl.gov[Enabler] Checklist for Alex project- [ ] Badge.
- [x] Network access (UCAMS account).
- [x] Cyber security training (see email concerning `SuccessFactors Learning`).
___
Greenhalgh, Alexander Dyrk; ID: 03078525 - has a requirement to complete Cyber Security Awaren...- [ ] Badge.
- [x] Network access (UCAMS account).
- [x] Cyber security training (see email concerning `SuccessFactors Learning`).
___
Greenhalgh, Alexander Dyrk; ID: 03078525 - has a requirement to complete Cyber Security Awareness for Curriculum Cyber Security Awareness Training due on 7/1/2021 11:59 PM US/Eastern
SuccessFactors notifications are generated at 28 and 14 days prior to the due date. If the due date is 14 days from now, this is your final notice.
To access your required training, please open [your ORNL Learning Plan](https://learning.ornl.gov/) here.
___
- [x] Access to GitLab (using UCAMS account).
- [x] Meet to discuss project plan, communication and management of project, etc.
- [x] Orientation.
- [ ] Weekly seminar.
- [ ] **Time tracking -- 40 hours per week are required by the project.**
> **We agreed that you will be attending some courses during the internship. Unfortunately, you have to make up the time to guarantee that you do have 40 hours per week spent on the internship work. We can definitely have offline discussion about this.**
- [ ] Deliverables.
___
- [x] pre-survey (due -- **need to finish ASAP**).
- [x] 150-word abstract that summarizes their poster presentation (an "elevator speech").
- [x] poster presentation - the poster session will take place on **Thursday, August 12**.
- [x] 300-word abstract - a scientific abstract.
- [x] research report.
- [ ] peer review - student answer template questions of another student's poster.
- [ ] post-survey.
___
> Deliverable guidelines: [SULI_Deliverables_Guidelines.pdf](/uploads/f6649d8832f1eba20d73da91bcc36a5e/SULI_Deliverables_Guidelines.pdf)Alex_SULI_06-07-2021Zhang, Yuanpengzhangy3@ornl.govGreenhalgh, AlexanderZhang, Yuanpengzhangy3@ornl.gov2021-07-31https://code.ornl.gov/y8z/alex_project/-/issues/2[Enabler] Plan of big scope2021-08-05T02:07:56ZZhang, Yuanpengzhangy3@ornl.gov[Enabler] Plan of big scope1. ADDIE GUI interface for `mantidtotalscattering`.
2. Python 2 to Python 3 translation for those autoNOM relevant scripts.
3. Implement `nomad_extract` in either `mantidtotalscattering` or `pystog`.
4. Save ADDIE configuration as a p...1. ADDIE GUI interface for `mantidtotalscattering`.
2. Python 2 to Python 3 translation for those autoNOM relevant scripts.
3. Implement `nomad_extract` in either `mantidtotalscattering` or `pystog`.
4. Save ADDIE configuration as a project.
5. ADDIE documentation update.
6. Abstract IDL data reduction routine to programming flowchart.
> Need to involve Joerg for understanding the IDL workflow and the abstraction.Alex_SULI_06-07-2021Zhang, Yuanpengzhangy3@ornl.govGreenhalgh, AlexanderZhang, Yuanpengzhangy3@ornl.gov2021-07-31https://code.ornl.gov/y8z/alex_project/-/issues/10[Scope] Running list of issues in ADDIE interface2021-07-12T19:30:16ZGreenhalgh, Alexander[Scope] Running list of issues in ADDIE interfaceIssues
------
~~1. Under Edit tab, selecting "Reset GSAS-tab" causes ADDIE to crash
`File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/main.py", line 772, in do_reset_gsas_tab calculategr_event_handl...Issues
------
~~1. Under Edit tab, selecting "Reset GSAS-tab" causes ADDIE to crash
`File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/main.py", line 772, in do_reset_gsas_tab calculategr_event_handler.do_reset_gsas_tab(self)
AttributeError: module 'addie.calculate_gr.event_handler' has no attribute 'do_reset_gsas_tab'
Aborted (core dumped)`~~
~~2. Under `help` tab, selecting `Cheat Sheet` leads to `https://neutrons.github.io/addie/index.html`, site not found error~~
~~3. As previously mentioned `Reduction Configuration` Refresh button not working~~
4. May need to reduce default size of `Reduction Configuration` or add slider bar, the only way I could see past the `global align and focus` portion was by using `alt + F7` key command, could be unique to my monitor size though
~~5. When importing from oncat database, selecting `Change User` button and logging in again causes ADDIE to crash
`File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/processing/mantid/master_table/import_from_database/oncat_authentication_handler.py", line 111, in ok_clicked
self.next_function()
TypeError: 'NoneType' object is not callable
Aborted (core dumped)`~~
~~6. Haven't been able to get the search box/button at top of interface to work, don't think it has any functionality~~
~~7. Haven't been able to get any result other than `Unable to calculate Molecular Mass. CHECK YOUR FORMULA!` from periodic table, even retrying formulas loaded in from database~~
~~8. If `Chemical formula` is filled out in table entry, selecting `Mass Density` and switching to `Number Density`, trying to replace `N/A` in field with number causes `ADDIE` to crash: `File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/utilities/math_tools.py", line 196, in number_density2mass_density
mass_density = number_density * molecular_mass / natoms / avogadro_term
TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'
Aborted (core dumped)` Another Error With same button: `File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/processing/mantid/master_table/mass_density_handler.py", line 194, in number_density_value_changed
number_density = np.float(self.ui.number_density_line_edit.text())
ValueError: could not convert string to float:
Aborted (core dumped)`~~
9. Already knew issue existed but Error output when pressing `Launch Reduction` button is: `Traceback (most recent call last):
File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/main.py", line 559, in run_mantid
mantid_reduction_launcher.run_mantid(self)
File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/processing/mantid/launch_reduction.py", line 87, in run_mantid
reduction_input = exporter.convert_from_row_to_reduction(json_input)
File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/processing/mantid/master_table/master_table_exporter.py", line 649, in convert_from_row_to_reduction
element_section)
File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/processing/mantid/master_table/master_table_exporter.py", line 505, in density_selection_for_reduction
dictionary['MassDensity'] = float(mass_density)
ValueError: could not convert string to float: 'N/A'
Aborted (core dumped)`
10. Need to put warning for when `launch reduction` is hit with nothing in table, crashes `Addie` with error ` File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/main.py", line 559, in run_mantid
mantid_reduction_launcher.run_mantid(self)
File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/processing/mantid/launch_reduction.py", line 67, in run_mantid
raise RuntimeError('Cannot export empty table')
RuntimeError: Cannot export empty table
Aborted (core dumped)` which is exactly how it should behave right now
~~11. Sometimes when loading in from database, a second `Options to load list of runs selected` window is created that cannot be deleted. Hard to pin down the exact cause yet, has happened twice loading in different files. Still can't reliably reproduce Example:
![image](/uploads/b441d07ff93154b3efd8a6fbd4031771/image.png)~~
~~12. When trying to import from database, during authentication stage, if we click on `Cancel` button the authentication window will be closed and we are then left with the 'Import from database' window. Then in this window, if we are trying to click on some option, it will crash.~~
~~13. Very similar to #8 in this list, selecting `Mass Density` and switching to the third option `Mass` will cause addie to crash with error `Traceback (most recent call last):
File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/processing/mantid/master_table/mass_density_handler.py", line 221, in mass_value_changed
mass = np.float(self.ui.mass_line_edit.text())
ValueError: could not convert string to float: 'N/'
Aborted (core dumped)
` Sometimes allows you to edit before crashing, Only way to edit is to highlight all of N/A and replace at once, trying to delete N/A one character at a time will cause crash. If number is in space, clearing the last character is what cause will crash, understandable since data type can no longer be a float.~~
~~14. Go to `Rietveld` tab and load in the attached GSAS data file ([NOM161049tof.gsa](/uploads/8af7b9646043ed0175b761d2f204819b/NOM161049tof.gsa)), then go to `Edit` tab and click on `Reset GSAS-tab` will crash ADDIE, bringing up the following error message,~~
```
Traceback (most recent call last):
File "/SNS/users/y8z/miniconda/envs/addie_env_3/lib/python3.6/site-packages/addie/main.py", line 773, in do_reset_gsas_tab
bragg_event_handler.BraggTree.do_reset_gsas_tab(self)
File "/SNS/users/y8z/miniconda/envs/addie_env_3/lib/python3.6/site-packages/addie/rietveld/braggtree.py", line 336, in do_reset_gsas_tab
main_window._myController.delete_workspace(workspace)
File "/SNS/users/y8z/miniconda/envs/addie_env_3/lib/python3.6/site-packages/addie/addiedriver.py", line 190, in delete_workspace
raise RuntimeError('Workspace %s does not exist.' % workspace_name)
RuntimeError: Workspace Bank 1 - 15.099999999999994 does not exist.
Aborted (core dumped)
```
~~15. After resetting GSAS tab, trying to change units will result in addie crash, some of the children still remain on graph, probably need to make addition changes to fix #14~~
```
File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/main.py", line 692, in evt_switch_bragg_unit
rietveld_event_handler.switch_bragg_unit(main_window=self)
File "/SNS/users/3d0/miniconda3/envs/addie-env_test/lib/python3.6/site-packages/addie/rietveld/event_handler.py", line 377, in switch_bragg_unit
assert len(ws_group_list) > 0, 'At least 1 GSS file must be selected.'
AssertionError: At least 1 GSS file must be selected.
Aborted (core dumped)
```
~~Trying to check some of the bank boxes will also cause a crash, need to find what fully resets window~~
~~In addition, changing the units while in something is actually graphed will automatically graph all banks~~
~~16. Load in several GSAS data (as attached here, [NOM161121tof.gsa](/uploads/e73835b58e306f54a50506ec66fa302f/NOM161121tof.gsa), [NOM161122tof.gsa](/uploads/4a37555d9a929f19ddcabfdd3289aae6/NOM161122tof.gsa), [NOM161123tof.gsa](/uploads/ae73489f278b22a7c5e824e9a832b03c/NOM161123tof.gsa)) in `Rietveld` tab and we will see those loaded in data under the `Bragg Workspaces` tree. Select all loaded-in data and right click on them to select `Delete workspace`. Following that, we then go ahead to change unit and ADDIE will crash, showing the following error,~~
```
Traceback (most recent call last):
File "/SNS/users/y8z/miniconda/envs/addie_env_3/lib/python3.6/site-packages/addie/main.py", line 692, in evt_switch_bragg_unit
rietveld_event_handler.switch_bragg_unit(main_window=self)
File "/SNS/users/y8z/miniconda/envs/addie_env_3/lib/python3.6/site-packages/addie/rietveld/event_handler.py", line 377, in switch_bragg_unit
assert len(ws_group_list) > 0, 'At least 1 GSS file must be selected.'
AssertionError: At least 1 GSS file must be selected.
Aborted (core dumped)
```
~~17. In `Rietveld` tab, after loading in some data (as attached here, [NOM161121tof.gsa](/uploads/e73835b58e306f54a50506ec66fa302f/NOM161121tof.gsa), [NOM161122tof.gsa](/uploads/4a37555d9a929f19ddcabfdd3289aae6/NOM161122tof.gsa), [NOM161123tof.gsa](/uploads/ae73489f278b22a7c5e824e9a832b03c/NOM161123tof.gsa)), if we select nothing in the `Bragg Workspaces` tree and click on the change unit dropdown menu, ADDIE will crash, throwing out the message,~~
```
Traceback (most recent call last):
File "/SNS/users/y8z/miniconda/envs/addie_env_3/lib/python3.6/site-packages/addie/main.py", line 692, in evt_switch_bragg_unit
rietveld_event_handler.switch_bragg_unit(main_window=self)
File "/SNS/users/y8z/miniconda/envs/addie_env_3/lib/python3.6/site-packages/addie/rietveld/event_handler.py", line 377, in switch_bragg_unit
assert len(ws_group_list) > 0, 'At least 1 GSS file must be selected.'
AssertionError: At least 1 GSS file must be selected.
```
~~18. In `Rietveld` tab, after loading in some data (as attached here, [NOM161121tof.gsa](/uploads/e73835b58e306f54a50506ec66fa302f/NOM161121tof.gsa), [NOM161122tof.gsa](/uploads/4a37555d9a929f19ddcabfdd3289aae6/NOM161122tof.gsa), [NOM161123tof.gsa](/uploads/ae73489f278b22a7c5e824e9a832b03c/NOM161123tof.gsa)), if we click on `workspaces` in the `Bragg Workspaces` tree and click on the change unit dropdown menu, ADDIE will crash, throwing out the message,~~
```
Traceback (most recent call last):
File "/SNS/users/y8z/miniconda/envs/addie_env_3/lib/python3.6/site-packages/addie/main.py", line 692, in evt_switch_bragg_unit
rietveld_event_handler.switch_bragg_unit(main_window=self)
File "/SNS/users/y8z/miniconda/envs/addie_env_3/lib/python3.6/site-packages/addie/rietveld/event_handler.py", line 388, in switch_bragg_unit
ws_group)
File "/SNS/users/y8z/miniconda/envs/addie_env_3/lib/python3.6/site-packages/addie/addiedriver.py", line 296, in get_bank_numbers
wksp = addie.utilities.workspaces.get_ws(ws_name)
File "/SNS/users/y8z/miniconda/envs/addie_env_3/lib/python3.6/site-packages/addie/utilities/workspaces.py", line 33, in get_ws
name), 'Workspace "{}" does not exist.'.format(name)
AssertionError: Workspace "workspaces" does not exist.
Aborted (core dumped)
```
~~19. Import some example data (as attached here, [NOM161121tof.gsa](/uploads/e73835b58e306f54a50506ec66fa302f/NOM161121tof.gsa), [NOM161122tof.gsa](/uploads/4a37555d9a929f19ddcabfdd3289aae6/NOM161122tof.gsa), [NOM161123tof.gsa](/uploads/ae73489f278b22a7c5e824e9a832b03c/NOM161123tof.gsa)) in `Rietveld` tab, we then right click on `workspaces` in `Bragg Workspaces` tree and all items there in the right click menu will crash ADDIE. This is because `workspaces` is just an indicator or place holder so any time when we right click on `workspaces` followed by clicking on any of the items in the right click menu, we just want to skip any operation (i.e. do nothing).~~
Priorities
----------
| **Group-#** | **Member** | **Priority** | **Subdivision** | **Comments**|
|-------------|----------------------|--------------|--------------------------------------------------|-------------|
| 1 | 1, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15 | Critical | 1 (#11), 14 (#14), 15 (#16), 5 (#12), 8 (#17), 13 (#17), 12 (#13), 16 (#18), 17 & 18 (#19), 19 (#20) $`\rightarrow`$ 7 (#21), 3 (#22), 6 (#23), 2 (#24) $`\rightarrow`$ 4 | These are all the issues we need to fix before implementing the `real` functionality, i.e., preparing input for `mantidtotalscattering` and kick off the reduction |
| 2 | 9, 10 | High | | Two things need to happen when clicking on `Launch Reduction`, 1) prepare the input json file and 2) call `mantidtotalscattering` to run with the prepared json file |
| 3 | 11 | Low | | This will not be covered during the internship |Alex_SULI_06-14-2021Greenhalgh, AlexanderGreenhalgh, Alexanderhttps://code.ornl.gov/y8z/alex_project/-/issues/9[Task] Kick off reduction by calling mantidtotalscattering2021-06-23T14:15:37ZZhang, Yuanpengzhangy3@ornl.gov[Task] Kick off reduction by calling mantidtotalscatteringFollowing previous task #8, once the json file is ready, we need to go the next stage - call `mantidtotalscattering` to run with the prepared json file. Suppose the prepare json file is `reduction.json`, the command we are going to execu...Following previous task #8, once the json file is ready, we need to go the next stage - call `mantidtotalscattering` to run with the prepared json file. Suppose the prepare json file is `reduction.json`, the command we are going to execute is `mantidtotalscattering reduction.json`.
Refer to the following codes for the typical method to call another program within ADDIE.
[https://github.com/neutrons/addie/blob/50e28ba3f7b7e02774328087ebf9c595057c4b54/addie/processing/idl/run_sum_scans.py#L38-L45](https://github.com/neutrons/addie/blob/50e28ba3f7b7e02774328087ebf9c595057c4b54/addie/processing/idl/run_sum_scans.py#L38-L45)
> We may need some more live discussions when we come to this task since it may be a bit challenging for you @halgh007 to figure out the workflow.Greenhalgh, AlexanderGreenhalgh, Alexanderhttps://code.ornl.gov/y8z/alex_project/-/issues/7[Story] Make ADDIE interface for mantidtotalscattering functioning2021-06-11T15:37:04ZZhang, Yuanpengzhangy3@ornl.gov[Story] Make ADDIE interface for mantidtotalscattering functioningMaking sure the table view and all the buttons, input boxes, etc. are working perfectly, the very next step is to worry about the `Launch Reduction` button, by clicking on which the program will do the real job - reducing the data. Two m...Making sure the table view and all the buttons, input boxes, etc. are working perfectly, the very next step is to worry about the `Launch Reduction` button, by clicking on which the program will do the real job - reducing the data. Two main steps we need to implement for this story, i.e. when clicking on the `Launch Reduction` button, we expect the program to do the following two things,
1. Collect all information in the table view (including those necessary information from the `Reduction Configuration` window) and prepare the input json file.
2. Call `mantidtotalscattering` to run with the prepared json file. The command is simply `mantidtotalscattering INPUT_FILE_NAME.json`.
> Refer to specific tasks for details.Zhang, Yuanpengzhangy3@ornl.govGreenhalgh, AlexanderZhang, Yuanpengzhangy3@ornl.gov