plan: quicknxsv2 modularization hackathon knowledge base

# Executive summary — quicknxsv2 modularization feasibility

**Target audience:** you have ~10 minutes before the hackathon session.

You need to know: is this feasible, what are the main risks, what's

the recommended starting approach.

## Bottom line

**Modularization is feasible.**  The Qt/UI layer is already cleanly

separated in practice — there are zero Qt imports under

`src/quicknxs/interfaces/data_handling/` (the de-facto back-end

directory).  The real work is not "pulling UI out of the science code"

(that's mostly done) but **consolidating the two parallel science

back-ends that already exist in tree** — one in

`quicknxsv2/interfaces/data_handling/` and one in `mr_reduction/`.

| Evidence | File / command |

|---|---|

| Zero Qt imports in `data_handling/` | `grep -r 'qtpy\|PyQt' quicknxsv2/src/quicknxs/interfaces/data_handling/` returns no matches |

| Only 4 files import `mr_reduction` | `instrument.py`, `processing_workflow.py`, `quicknxs_io.py`, `gui.py` |

| `peak_finding.py` is duplicated verbatim | `diff quicknxsv2/.../peak_finding.py mr_reduction/.../peak_finding.py` — only docstring whitespace differs |

| `dead_time_correction.py` duplicates the Mantid `SingleReadoutDeadTimeCorrection` algorithm class | Both repos register the same `AlgorithmFactory` name |

| `data_info.py` has parallel implementations | quicknxsv2 version = 413 lines; mr_reduction = 1101 lines; neither is a subset of the other |

| mr_reduction version pinned in quicknxsv2 | `pyproject.toml`: `mr_reduction = "==2.17.0"` |

| Mantid version mismatch at HEAD | quicknxsv2 requires Mantid 6.14; mr_reduction next requires Mantid 6.15 |

## The three real choices for the hackathon

1. **Lift-and-name** — extract `quicknxsv2/interfaces/data_handling/`

   into a new library (tentatively `mr_core` or keep the `quicknxs`

   namespace but split the wheel), and `mr_reduction` continues to

   live alongside it.

   - *Robust.* Preserves the existing reduction algorithm that the

     quicknxsv2 scientists trust.

   - *Cost:* The **orchestration** in `mr_reduction.ReductionProcess`

     (the autoreduce/livereduce entry points) is *not* exercised by

     the new library — the duplication problem gets two new homes

     instead of one.  Unless the hackathon also decides to rebase

     autoreduce on top of the new library, this does not reduce

     duplication; it only moves the UI out.

2. **Merge-into-mr_reduction** — push quicknxsv2's `data_handling/`

   logic into `mr_reduction` and have the UI depend only on

   `mr_reduction` as its back-end.

   - *Robust for the autoreduce path.*  Unifies the two parallel

     reducers.

   - *Cost:* `mr_reduction.ReductionProcess` today operates on

     **run numbers and file paths** (a scripted, one-shot API), not

     on an interactive, object-oriented `DataManager` state.  The UI

     needs a different, stateful API surface (the `NexusData`/

     `CrossSectionData` / `DataManager` abstractions).  Adding that

     API without breaking autoreduce is the real task.

3. **Carve-out `mr_core`** — build a third, neutral package that

   contains the *shared* primitives (file loading, cross-section

   filtering, dead-time correction, peak finding, ORSO I/O).  Both

   quicknxsv2 and `mr_reduction`'s scripted pipeline depend on

   `mr_core`.  Each keeps its own orchestration.

   - *Most robust.* Clean layering, each package has a clear purpose,

     can be versioned independently.

   - *Cost:* Three packages to maintain.  Migration is two independent

     refactors (one per consumer) that both have to land before the

     duplication is actually eliminated.

**Recommendation: option 3 (carve-out `mr_core`) is the robust choice.**

Option 1 and option 2 each *half-solve* the problem (UI separated but

back-end still duplicated vs. back-end unified but UI awkwardly

constrained); option 3 uses each week of hackathon effort to retire

one class of duplication, and the intermediate state (after mr_core

exists but before mr_reduction adopts it) is still a net improvement

over today.  See `06-modularization-strategies.md` for the full

decision matrix.

## Red-green TDD is viable

- Pure-unit tests already exist for the trivial domain objects

  (`FilePath`, `RunNumbers`, `Configuration`): no Qt, no Mantid, no

  LFS data.  These can lead the refactor.

- Integration tests exist for the reduction pipeline

  (`test_processing_workflow.py`) but require the `quicknxs-data` LFS

  submodule and Mantid.  These can lock in reduction-output

  equivalence before / after the refactor.

- UI tests use `pytest-qt` + `pytest-xvfb`.  They run headless in CI

  and exist for most main-window interactions.  These lock in "the

  button still does what it did".

- mr_reduction has **integration tests marked `datarepo` and

  `sns_mounted`** that already validate the autoreduce pipeline on

  real run files — use these as the "back-end equivalence" harness.

## Main risks

1. **Mantid version drift** — quicknxsv2 pins `mr_reduction==2.17.0`

   (Mantid 6.14) while `mr_reduction/next` has advanced to 6.15.  Any

   "just move files between repos" refactor will collide with this

   mismatch.  The hackathon should **resolve the Mantid version first**.

2. **Two parallel `data_info.py` implementations** — 413 vs. 1101

   lines.  Neither is a superset.  The differences encode real

   behavioral differences (probably use-case specialization).  These

   must be reconciled by tests, not by diff.

3. **Threading model** — quicknxsv2 does **not** use `QThread` or any

   worker-thread pattern for reduction.  Any move that keeps reduction

   synchronous in the UI thread will ship the same UI-blocking problem.

   Threading is a separate effort orthogonal to modularization but

   may as well be designed-in now.

4. **Autoreduce/livereduce contract** — `mr_reduction` is copied into

   `/SNS/REF_M/shared/autoreduce` and `/SNS/REF_M/shared/livereduce`

   at release time.  Changing `ReductionProcess`'s signature or import

   paths breaks both deployment targets silently until a run fails.

## Out-of-scope insights worth noting

- **`lr_reduction` (sibling REF_L project) is a working model** of the

  exact split under consideration: `launcher/` is Qt, `src/lr_reduction/`

  is the library, `src/lr_autoreduce/` is the script.  The launcher

  talks to the library via **subprocess** (spawning

  `/SNS/REF_L/shared/batch_reduce.py`) rather than in-process calls.

  Worth studying before committing to an in-process API surface.

- **`quicknxsv1` (the 2012 Python/numpy/h5py original)** has a

  similar de-facto separation: `qreduce.py` / `qio.py` / `qcalc.py`

  import no Qt.  The lineage of the split is older than v2.

## What this knowledge base does NOT claim

- That the developer docs (`docs/developer/highlevel_design.rst` etc.)

  are wrong.  They are skeletal (literal "TODO" placeholders), not

  wrong.  This investigation built its evidence from code, not from

  those docs.

- That the hackathon should commit to one of the three options above

  on day one.  Spike each one on day one; choose on day two.

- That mr_reduction is "the" back-end today.  At code level,

  quicknxsv2 relies on mr_reduction for four primitives and uses

  Mantid directly for the rest.  "Back-end" is not a single package

  as of now.

# Repository structure — independent survey

This is a ground-truth map of the four repositories in the workspace.

Every claim is backed by a file path or a `git log` output so you can

verify independently.  **Do not trust this document either** — verify

against the current state of the repo when your hackathon starts.

## The four repositories, at a glance

| Repo | Role | Size | Primary language | External deps | First commit |

|---|---|---|---|---|---|

| `quicknxsv2` | Current Qt GUI for REF_M reduction | ~14,280 src LOC | Python 3.10–3.12 | Mantid, `mr_reduction==2.17.0`, PyQt5, matplotlib | 2017-10-24 |

| `mr_reduction` | Scripted reduction library + autoreduce + livereduce | ~7,620 src LOC | Python 3.11 | Mantid ≥6.15, Flask/gunicorn, plotly, plot_publisher | 2018-02-23 |

| `quicknxsv1` | Original Qt GUI (Python/numpy/h5py) | ~30,250 src LOC (inc. 7,625 from `icons_rc.py`) | Python 3.10–3.12 | PyQt5, h5py, numpy, scipy (no Mantid) | 2012-12-02 |

| `lr_reduction` | Sibling REF_L reduction project (not part of the hackathon target, but an architectural reference) | see `src/lr_reduction/` | Python 3.11 | Mantid, pure-numpy tools | 2020 (see `git log`) |

Commit counts (as of 2026-04-18 on default branches):

```bash

quicknxsv2:   1054 commits

mr_reduction:  415 commits

quicknxsv1:    579 commits

```

## quicknxsv2

### On-disk layout

```

quicknxsv2/

├── src/quicknxs/

│   ├── gui.py              # entry point → MainWindow()

│   ├── __init__.py         # just reads __version__

│   ├── config/             # plotting options, QColors, settings.json

│   ├── icons/              # PNG/SVG resources + compiled icons.qrc

│   ├── interfaces/         # THE BIG ONE — UI code + data handling

│   │   ├── main_window.py              # 627 LOC — the MainWindow class

│   │   ├── data_manager.py             # 1260 LOC — holds cached NexusData, reduction lists

│   │   ├── configuration.py            # 268 LOC — Configuration (reduction params)

│   │   ├── plotting.py                 # 776 LOC — PlotManager + matplotlib

│   │   ├── diagnostic_widget.py        # 174 LOC

│   │   ├── enums.py                    # 37 LOC

│   │   ├── offspec_slice_dialog.py     # 286 LOC

│   │   ├── reduction_dialog.py         # 133 LOC

│   │   ├── result_viewer.py            # 190 LOC

│   │   ├── smooth_dialog.py            # 566 LOC

│   │   ├── event_handlers/             # Qt signal / slot plumbing

│   │   │   ├── main_handler.py         # 2089 LOC — where the UI actions live

│   │   │   ├── plot_handler.py         # 355 LOC

│   │   │   ├── configuration_handler.py # 128 LOC

│   │   │   ├── progress_reporter.py    # 97 LOC

│   │   │   ├── status_bar_handler.py   # 13 LOC

│   │   │   └── widgets.py              # 44 LOC

│   │   └── data_handling/              # NO QT HERE — the de-facto back-end

│   │       ├── data_set.py             # 1075 LOC — NexusData, CrossSectionData, NexusMetaData

│   │       ├── data_manipulation.py    # 572 LOC — generate_script, stitch_reflectivity, merge_reflectivity

│   │       ├── data_info.py            # 413 LOC — DataInfo (peak/ROI determination)

│   │       ├── instrument.py           # 456 LOC — Instrument (per-beamline geometry + file loading)

│   │       ├── processing_workflow.py  # 910 LOC — ProcessingWorkflow (export orchestration)

│   │       ├── quicknxs_io.py          # 748 LOC — read/write QuickNXS-format reduction files

│   │       ├── peak_finding.py         # 353 LOC — (near-duplicate of mr_reduction/peak_finding.py)

│   │       ├── off_specular.py         # 460 LOC — OffSpecular class

│   │       ├── gisans.py               # 259 LOC — GISANS class

│   │       ├── diagnostic_data.py      # 124 LOC

│   │       ├── dead_time_correction.py # 126 LOC — duplicates SingleReadoutDeadTimeCorrection Mantid algorithm

│   │       ├── filepath.py             # 223 LOC — FilePath, RunNumbers (pure Python, no deps)

│   │       └── genx_templates/         # .gx template files

│   └── ui/                 # compiled Qt UI files + Qt-specific widgets

│       ├── mplwidget.py                # 897 LOC — MPLWidget, NavigationToolbar subclasses

│       ├── compare_plots.py            # 169 LOC

│       ├── active_radio_button.py      # 69 LOC

│       ├── binningtype_combobox.py     # 52 LOC

│       ├── deadtime_entry.py           # 77 LOC

│       ├── deadtime_settings.py        # 60 LOC

│       └── *.ui                        # Qt Designer files (main window, dialogs)

└── test/

    ├── conftest.py                     # data_server + main_window fixtures

    ├── unit/

    │   └── quicknxs/

    │       ├── config/                 # 1 file

    │       └── interfaces/             # ~15 files

    ├── integration/                    # 2 files (processing_workflow, load_reduced_file)

    ├── ui/                             # 21 files — pytest-qt driven

    └── data/

        └── quicknxs-data/              # LFS submodule

```

Package mapping:

- Package name on PyPI/conda: `quicknxs`

- Entry point: `[project.gui-scripts] quicknxs-gui = "quicknxs.gui:main"`

- Build: `hatchling` + `versioningit`

- Pixi tasks: `test`, `conda-build`, `audit-deps`, `build-docs`

- Platform: `linux-64` only

### Git branch model (quicknxsv2)

From `CLAUDE.md` and verified against `git branch -a`:

- `next` — integration branch, all PRs land here (currently 1 commit ahead of what I see locally)

- `qa` — staging, promoted from `next`

- `main` — stable release, promoted from `qa`

- Work branches: `EWM{number}_{description}`, `bugfix_*`, `fix_*`,

  `{user}/{description}`

- The local checkout is on `bvacaliuc/next` which merges `personal/agentic-knowledge`.

### Developer docs are not a source of truth

`docs/developer/highlevel_design.rst` is structurally:

```rst

Overview

########

TODO

Architecture

############

TODO

Reduction Workflow

##################

TODO

```

All three sections of the "high-level design" are literal `TODO`.  The

other files under `docs/developer/` cover process (branching, release

cycle, CI troubleshooting) but not architecture.  This is consistent

with the user's instruction to *independently* investigate.

## mr_reduction

### On-disk layout

```

mr_reduction/

├── src/

│   ├── mr_reduction/       # THE LIBRARY

│   │   ├── __init__.py     # just reads __version__

│   │   ├── mr_reduction.py            # 457 LOC — ReductionProcess class

│   │   ├── filter_events.py           # 589 LOC — split_events, split_error_events

│   │   ├── data_info.py               # 1101 LOC — DataInfo (peak/ROI determination; parallel to v2's)

│   │   ├── beam_options.py            # 266 LOC

│   │   ├── dead_time_correction.py    # 177 LOC — apply_dead_time_correction + Mantid algorithm

│   │   ├── inspect_data.py            # 689 LOC

│   │   ├── io_orso.py                 # 523 LOC — ORSO file format I/O

│   │   ├── logging.py                 # 10 LOC

│   │   ├── mantid_algorithm_utils.py  # 28 LOC

│   │   ├── mr_direct_beam_finder.py   # 278 LOC

│   │   ├── oncat_comm.py              # 28 LOC

│   │   ├── peak_finding.py            # 360 LOC — DUPLICATE of quicknxsv2's peak_finding.py

│   │   ├── reflectivity_merge.py      # 674 LOC — stitching runs across Q ranges

│   │   ├── reflectivity_output.py     # 115 LOC

│   │   ├── runpeak.py                 # 128 LOC — RunPeakNumber class

│   │   ├── script_output.py           # 334 LOC

│   │   ├── scripts/nexus_to_orso.py   # CLI tool

│   │   ├── settings.py                # 36 LOC — PolarizationLogs, hardcoded /SNS/REF_M paths

│   │   ├── simple_utils.py            # 107 LOC

│   │   ├── spin_setup.py              # 149 LOC

│   │   ├── types.py                   # 23 LOC — MantidWorkspace type alias

│   │   └── web_report.py              # 1002 LOC — generates HTML reports with plotly

│   ├── mr_autoreduce/      # TEMPLATED REDUCTION SCRIPT SERVICE

│   │   ├── reduce_REF_M_run.py        # 103 LOC — Flask server, reduces on HTTP POST

│   │   ├── reduce_REF_M.py.template   # 155 LOC — string.Template, substituted at runtime

│   │   └── templates/reduction_options.html

│   └── mr_livereduce/      # LIVE-DATA REDUCTION SCRIPTS

│       ├── reduce_REF_M_live_proc.py       # 15 LOC

│       ├── reduce_REF_M_live_post_proc.py  # 238 LOC

│       ├── polarization_analysis.py        # 171 LOC

│       ├── livereduce.conf

│       ├── livereduce_watchdog.sh

│       └── livereduce_watchdog.service

└── tests/

    ├── conftest.py                     # data_server + selenium browser fixture

    ├── unit/

    │   ├── mr_reduction/               # 15 files

    │   ├── mr_autoreduce/              # 1 file

    │   └── mr_livereduce/              # few files

    ├── integration/                    # 7 files

    └── mr_reduction-data/              # LFS submodule

```

Package mapping:

- Package name: `mr_reduction` (installs three top-level packages: `mr_reduction`, `mr_autoreduce`, `mr_livereduce`)

- CLI entry points:

  - `nexus_to_orso = "mr_reduction.scripts.nexus_to_orso:main"`

  - `reduce_REF_M = "mr_autoreduce.reduce_REF_M_run:main"`

- Build: `hatchling` + `versioningit`

- Pixi tasks: `test`, `autoreduce` (copies scripts to `/SNS/REF_M/shared/autoreduce`), `livereduce` (copies to `/SNS/REF_M/shared/livereduce`)

- Platform: `linux-64`

### Git branch model (mr_reduction)

From `git branch -a`: same `next` / `qa` / `main` model as quicknxsv2.

Default branch is `next`.

### Mantid version (mr_reduction)

`pyproject.toml`:

```toml

mantid = ">=6.15.0,<6.16.0"

```

This is a **mismatch with quicknxsv2**, which pins

`mr_reduction == 2.17.0` and annotates (in its own `pyproject.toml`)

that version "requires mantid >=6.14.0,<6.15.0".  The hackathon will

trip on this.

## quicknxsv1

### On-disk layout (short version)

```

quicknxsv1/

├── quicknxs/              # flat package (not src/ layout)

│   ├── main_gui.py              # 2642 LOC — the MainWindow

│   ├── default_interface.py     # 1910 LOC — Qt Designer-generated

│   ├── docked_interface.py      # 1785 LOC — alternate Qt Designer-generated

│   ├── gui_utils.py             # 1121 LOC — GUI helpers, mixed with reduction calls

│   ├── qreduce.py               # 3387 LOC — **NO QT** — NXSData, MRDataset, Reflectivity, OffSpecular, GISANS

│   ├── qio.py                   # 1131 LOC — **NO QT** — Exporter, HeaderParser, HeaderCreator

│   ├── qcalc.py                 # 496 LOC — **NO QT** — numeric helpers

│   ├── mpfit.py                 # 2437 LOC — vendored third-party (MINPACK fit)

│   ├── icons_rc.py              # 7625 LOC — compiled Qt resource (images as bytes)

│   ├── database.py              # 186 LOC — buzhug-based

│   ├── buzhug/                  # embedded read-only database

│   ├── config/                  # instrument configs, settings.json

│   ├── ipython_tools.py         # 75 LOC — AttributePloter, StringRepr

│   └── (~20 more files for specific dialogs, plot widgets, polarization UI)

├── tests/                  # pytest-based, with pixi tasks for test-core/test-gui/test-db

├── pyproject.toml          # setuptools + versioningit

└── pixi.lock

```

### Why v1 matters for v2 modularization

1. **The same de-facto split exists in v1** — `qreduce.py`, `qio.py`,

   and `qcalc.py` import **zero Qt**.  They are the back-end layer

   even though they live in the same package.  The lineage of the

   split long predates v2.

2. **v1 uses h5py/numpy directly, not Mantid.**  So when you compare

   "v1 back-end" to "v2 back-end" you're comparing two *different*

   reduction implementations (not just two UIs over the same

   library).  Any lift-and-extract must not paper over this.

3. **Active development on `feature/read-event-nexus`** — v1 is

   currently being updated to read the same `.nxs.h5` files that v2

   supports.  This is evidence of continued use of v1 at SNS: the v1

   GUI is not dead yet.

### Branch model (quicknxsv1)

From `quicknxsv1/CLAUDE.md`:

- `master` — READ-ONLY (tracks upstream `aglavic/quicknxs`, never written by this fork)

- `qa` — terminal stable (receives release tags)

- `next` — integration

- Current checkout: `feature/read-event-nexus` (work branch for event Nexus support)

## lr_reduction

Short version, for architectural comparison.  `lr_reduction` targets

REF_L (the Liquids Reflectometer), not REF_M, and so it is not the

modularization target.  But its layout is instructive:

```

lr_reduction/

├── src/

│   ├── lr_reduction/       # REDUCTION LIBRARY (no Qt)

│   │   ├── event_reduction.py         # 55861 LOC / ~1800-2000 actual lines (wc says 55861 bytes)

│   │   ├── workflow.py                # 16136 LOC

│   │   ├── nr_reduction_calc.py       # 47927 LOC

│   │   ├── template.py

│   │   ├── reduction_template_reader.py

│   │   ├── background.py

│   │   ├── binary_processing.py       # pure h5py/numpy — parallel to v1's qreduce

│   │   ├── dead_time_correction.py

│   │   ├── peak_finding.py

│   │   ├── output.py

│   │   └── web_report.py              # plotly-based report, similar to mr_reduction

│   └── lr_autoreduce/      # AUTOREDUCE SCRIPT

│       └── reduce_REF_L.py

└── launcher/               # QT GUI — SEPARATE FROM THE LIBRARY

    ├── launcher.py                    # the tab container

    ├── apps/

    │   ├── reduction.py               # Qt widget — calls out to batch_reduce.py via subprocess

    │   ├── dynamic_30Hz.py

    │   ├── dynamic_60Hz.py

    │   ├── off_spec.py

    │   ├── quick_reduce.py

    │   ├── xrr.py

    │   └── sld_calculator.py

    └── scripts/

```

**Critical architectural note.**  `lr_reduction`'s launcher calls the

library **via subprocess** (it shells out to

`/SNS/REF_L/shared/batch_reduce.py`), not via in-process imports.

This is a fundamentally different architecture from quicknxsv2's

in-process integration and has trade-offs:

- **Pro:** clean process isolation, reduction can crash without

  killing the GUI, no in-process Mantid/Qt initialization

  interactions.

- **Con:** no live data sharing, every reduction is a fresh

  interpreter start (slow), result communication has to go through

  file system.

## Summary: three reduction pipelines, one UI team

Today the SNS reflectometry ecosystem has:

| Pipeline | Who uses it | Reduction implementation | UI |

|---|---|---|---|

| quicknxsv2 (in-process) | Instrument scientists, interactive mode | Mantid `MagnetismReflectometryReduction` algorithm + v2's own orchestration in `data_set.py` / `data_manager.py` | PyQt5 GUI |

| mr_reduction (scripted, post-experiment) | Automated post-experiment reduction | Mantid `MagnetismReflectometryReduction` algorithm + `ReductionProcess` orchestration | Web form (Flask) for parameter selection; otherwise none |

| mr_livereduce (streaming, during experiment) | Live data monitoring | Subset of mr_reduction; runs inside the live-reduce service | Auto-generated HTML monitor pages |

The modularization question is really **can the first and second be

unified in a way that doesn't break the third**.  See

`04-reduction-pipeline.md` for the detailed data flows.