Commit b569a59a authored by Bogdan Vacaliuc's avatar Bogdan Vacaliuc
Browse files

Merge remote-tracking branch 'origin/hack-a-thon-2026' into hack-a-thon-2026-supplementary-material

- updated with live-edited Hack-A-Thon-2026-Agenda.docx
parents 16f7ee63 700633e7
Loading
Loading
Loading
Loading
+9.66 KiB (299 KiB)

File changed.

No diff preview for this file type.

+387 −0
Original line number Diff line number Diff line
# Hack-A-Thon 2026 — Day 1 Summary (2026-04-22)

**Location:** 8630 B-202 · **Hours:** 09:30–16:30

**Threshold Goal (from the agenda):** Assessment and agreement on whether
QuickNXSv2 is conducive for refactoring/modularization into a Qt "front-end"
and a shared reduction "back-end" (via `mr_reduction` /
[MagnetismReflectometer](https://github.com/neutrons/MagnetismReflectometer)).

**Outcome:****Agreement reached.** The scientists endorsed the direction
"QuickNXSv2 is V1\* + modularize (and must support LR); can't block on it."
See **Consensus & Decisions** below.

Sources used for this summary:

- Bogdan's handwritten notes, `~/Desktop/Hack-A-Thon-2026/20260422172809.pdf`
  (18 pages; pages 3, 7, 8, 10, 11, 12 were scanned inverted and rotated for reading).
- Meeting transcripts `2026-04-22-Part1-transcript.txt` and `Part2-transcript.txt`
  (Teams auto-transcripts — low fidelity; treated as corroboration, not ground truth).
- The Day 1 Scientist-Assessment deck (slides 0–5a + end), sourced from the
  `quicknxsv2-modularization` branch, which supplied the technical frame for
  the morning discussion.

---

## Attendees (reconstructed from notes)

Scientists & software devs present or named in the room: John (Hetrick),
Valeria Lauter, Tim Charlton, Erik Watkins, Asmaa Qdemat,Marie Backman, Becky Anderson,
Volker Urban, Jim Browning, Glass Elsarboukh, Kevin Tactac.

---

## 09:30 – 10:00 · Opening Remarks (Bogdan)

Bogdan framed the two-instrument problem on the whiteboard:

- The Reflectometry suite runs two Mantid-centric GUIs (**QuickNXSv2** on
  REF_M; **lr_reduction** GUI on REF_L) and two parallel reduction libraries
  (`mr_reduction`, `lr_reduction`). Scientists have a "mash path forward"
  expectation; Volker's opening statement on behalf of LR was roughly
  *"we want this"*.
- **Primary goal for the week** (as written on the notes): *"figure out what
  makes QuickNXS"* — i.e. define the minimum shared kernel that both
  instruments need from a modular back-end.
- The morning was structured as the "Battle of the Robots" framing from the
  pre-meeting deck — Scientist Assessment (Blue Corner, 10:00) and Developer
  Assessment (Red Corner, 11:00) — playful, explicitly not adversarial.

---

## 10:00 – 11:00 · Scientist Assessment — Marie Backman

Marie presented her assessment of the two codebases, with input from Erik,
Valeria, Becky and Tim. Key points captured in the notes:

### Where the code is today

- **lr_reduction (refined):** "good ideas" in the rewrite, but UI, business
  logic and inter-state coupling all handled in one place; a change in one
  tends to affect the others.
- **QuickNXSv2:** approximately follows the **Model-View-Presenter** pattern
  and is therefore the better starting point for a clean separation. What is
  currently inside `DataManager` is the natural candidate to migrate into a
  back-end.

### Target architecture

- **One data-reduction UI for both instruments**, sitting on top of a shared
  back-end that exposes the same surface for LR and MR, with per-instrument
  specialization encapsulated below.
- The mechanism Kevin proposed for handling the LR/MR differences: a
  **Strategy pattern** — each reduction-phase function is parameterized by an
  instrument-specific descriptor (e.g. a `LR` dict and an `MR` dict) so the
  orchestration code stays common.

### Marie's four development phases

1. **Complexity** — understand what each codebase actually does today.
2. **Architecture** — clean separation into Model-View-Presenter.
3. **Apply Strategy pattern** — encapsulate per-instrument behavior.
4. **Implement new strategy** — populate LR and MR strategies.

### Discussion that followed

- **Autoreduce in scope?** John flagged *"What does autoreduction do?"* as
  highly relevant. Volker initially questioned whether autoreduce belongs in
  scope; Marie: the back-end would cover it anyway — autoreduce is one
  consumer of the same reduction functions. Consensus: in scope.
- **Templates and defaults.** Erik noted that `lr_reduction` already has a
  generic template concept (Becky's workflow: refined / auto templates). The
  question was why LR has drifted toward a new workflow separate from
  QuickNXS.
- **Fidelity / correctness first.** John's warning (carefully addressed to
  Volker): *"the one problem in the back-end is the MR → LR mapping"* — we
  should not fix problems by putting them aside; they may resurface.
- **HTML report.** Tim observed that reproducing results via HTML reports is
  today sufficient for many users (report answers most questions modulo
  monitor state).

---

## 11:00 – 12:00 · Scientist Input  (continued) — the user experience

This block was originally the "Developer Assessment" slot on the agenda, but
in practice the scientists kept the floor and elaborated the UX requirements
a unified back-end must satisfy:

### Valeria — REF_M specifics

- Position-sensitive detector, *not* monochromatic → measured data is a 3-D
  volume (x, y, ToF). Projections onto (x, y) or (x, ToF) are the typical
  reduction views.
- "Select what is important" at reduction time. If the tool is a black box,
  it is possible to make severe errors (e.g. bad background choice → wrong
  data). **Users must see the data.**

### Erik — tools exist, but are hard to use

- The refined/dev tools show the intermediates, but the user is asked to
  configure local context sets they normally never deal with — so they won't
  actually use them.
- **Reproducibility is non-negotiable. Black-box behavior must be avoided.**
  We don't want to support "wrong discovery."
- Goal: **fewer clicks** per reduction iteration. Every tweak that currently
  requires a full re-process is a barrier.
- Inter-instrument differences acknowledged: LR has larger samples, in-line
  DB options, tools to program problem measurements of the same sample under
  different conditions (e.g. D2O/H2O contrast). *Treat reduction of the same
  kind of sample exactly the same way.*

### Template / batch workflow

- Define a template, script/batch-process, and expect **the same inputs to
  yield the same result on different days** (today they don't, which is part
  of the silent-defaults problem flagged in the pre-deck slide 3).
- Erik: allow pulling in a pre-processed file instead of `.nxs.h5` + DB.
- Becky: each instrument will still need slight variations (e.g. DB as a
  separate tool).

### Diagnostic plots during reduction — a major theme

- Not a preview. A **check of how good reduction has been, given the
  instrument setup**.
- Becky: give me the reflectivity *and* let me see intermediate steps
  (toggle solve / intermediate stages on/off; plot of projected peak vs.
  detector; modeled vs. measured shape). "Show exactly where binning is — in
  R/Q, see what the algorithm is binning along, trace along angle or curve
  on the detector."
- Valeria: "Much of this is ingrained as diagnostic plots" — that capability
  is what she cares about.
- Erik: if the intermediate plots aren't there, at least **produce the
  reflectivity without diagnostics** (don't block the minimal path).
- Tim: today these things are scattered across screens / monolithic; we want
  them modular so intermediate steps can be surfaced on demand.
- **Flagged as an important feature: a way to split / expose the
  intermediate steps.**

### Events vs histograms — Tim's design question

- *"When do we stop keeping events in the reduction process?"* If we
  histogram too early we can't back up. That decision today is made in an
  unacceptably obscure place and impacts everyone.
- Event streams are large; we want to push the event/histogram transition
  down as far as reasonable.
- **Kevin / Key finding:** formalize this as a **Strategy** choice —
  `{Event, Histogram} focus` — with different downstream behavior per
  choice. (Event focus for flexibility; histogram focus for speed.)

### Overplot / compare

- Becky: overplot reflectivity curves in a "compare" tab. Today this is not
  available and would be a back-end-and-UI feature.
- Becky: at the point a default template is chosen, the user may have
  several per-sample setups. A UI should (a) start labeling samples to
  **guide** template selection, and (b) visually indicate when you have
  **deviated from the template you loaded**, with the option to change it.
  This is consistent with keeping reduction reproducible.
- Kevin asked about a template editor — Marie noted that ROI selection and
  the binning-algorithm toggles are already the hardest part.

### Where templates live today

- XML templates; not every setting lands in the data-file header. The
  refined path today *cannot* read a header back from a saved data file.
  **ORSO format** was discussed as a candidate for chunked, self-describing
  output.

**12:08 demo:** Becky+Erik showed `nr_launcher` running the new workflow
just before lunch.

---

## 12:00 – 13:00 · Lunch (Panera)

Informal continuation of the above; nothing of note captured in the written
notes for this block.

---

## 13:00 – 15:00 · Session 1 — Deep dive into reduction semantics

The afternoon session worked through concrete reduction semantics, largely
in front of QuickNXSv2 dev and `nr_launcher`. Notes from pages 14–17 of the
PDF:

### Matching templates to runs

- "What would happen if the user pointed at (data + template file)?"
  Answer: UI file → run → see reduced. *Read templates at reduced-data time,
  then populate the UI fields.*
- Bin width: 400 µs today (standard ToF channel). "Must optimize" — but use
  metadata from the DAQ point logs to let the template alternate be
  extracted, rather than hard-coding.
- This is where the Editor's-Mode dichotomy V2 ↔ V1 came up — Valeria had
  spoken to this earlier.

### Silent defaults (the pre-deck's slide 3 content)

The five MRR silent-default disagreements between `quicknxs` (GUI) and
`mr_reduction` (autoreduce) were acknowledged in context:

| Parameter | GUI | Auto |
|---|---|---|
| `ErrorWeightedBackground` | False | True |
| `CropFirstAndLastPoints` | False | True |
| `RoundUpPixel` | False | True |
| `AcceptNullReflectivity` | True | False |
| `UseSANGLE` | via inverted flag | same (latent land-mine) |

Consensus framing: **build a single `build_mrr_kwargs(configuration, …)`
parameter-builder shared by both callers.** This is the concrete first unit
of work to retire the "same data, different R(Q)" bug and is the obvious
candidate for an early tier-1 PR.

### Side-background and LR/MR differences

- No template-of-background exists today in the refined path.
- Side-background is used in LR and will need to be carried across — which
  can cause problems in MR if applied blindly; must be a
  per-instrument/per-run switch (global vs per-run vs per-PPS).

### Match-direct-beam (back-end API)

- Principle is similar for both: match the reflect-angle in the same
  direction, account for slit open/close events.
- When slits change, compute a mean slit opening across the span; when a
  "glitter" (slit-change) event occurs, use a lookup/algorithm to locate
  the matching direct-beam run.
- In liquids the same operation differs in detail — we need parameters that
  let the user choose which matcher is appropriate, and the back-end must
  expose that choice.

### DANGLE0 / calibration

- Used for calibration. If LR has a *different* DAS value than expected it
  gets more complicated, because it isn't *just* DAS values.
- "What if the runinfo were a large text box the user edited, instead of a
  fixed series of x-ticks?" → reframed explicitly as another instance where
  the **Strategy pattern** helps.

### One UI or two? (the big ergonomics question)

- Q: Do we have two UIs?
- A: The *behavior* differs in what is shown and what names are used; the
  underlying realm is shared.
- Q: When does the user pick? A: On open, e.g. `quicknxs --ref-m` /
  `quicknxs --ref-l`, with a default saved to user config; pull-down in
  Analysis to switch.
- Valeria wanted the two apps to *look and feel the same* wherever possible.
- Consensus forming: keep **one repo, one codebase, two deployments/entry
  points**. Some prefer a drop-down to switch modes; others prefer a
  command-line switch that fixes the mode for a session. *This was not
  closed on Day 1 — revisit.*
- Related: this architecture also naturally **repurposes to a batch
  processor** (trade the UI for a LR/MR function call), which is how
  autoreduce re-enters the picture.

### Auto-select runs (after Erik rejoins)

- Q: Is there a way to auto-select runs?
- A: Input a range of runs; metadata in each run tells the series. Sort by
  `R_{1,2,3}` vs Q. Sample-by-metadata index; set an index angle below.
- In batch, Nexus run-numbers in full blocks → apply the template blindly;
  if metadata isn't set → garbage; if the template is no longer appropriate
  → garbage. Risk noted.
- Tool on the table: `mr-launcher` + calc back-end that churns through
  blindly — good for throughput, dangerous without a metadata guard.
- Conclusion: **template is only for data reduction.** Each angle has
  differ... (rest of sentence unclear, probably *differs and bigger*). Mix
  of global + per-run parameters; lots of things are not yet modeled.

### Off-specular, REF_M / REF_L

- Four-in flips in template (`thip` – Sample, `that` – Sample, `this`).
- Point: need to match up (e.g. substrate / DAS). Off-spec run numbers
  captured: `44159+44160+44161` db 44148`.
- Finishing note, written in a box on page 17: **"Agree QuickNXSv2 is V1\*
  + modularize (LR) + can't block it."** This is the Day 1 threshold-goal
  agreement.

---

## 15:00 – 16:30 · pyNEX catch-up (optional)

Susan Hubbard had reserved the original pyNEX room, so the group came to
the hack-a-thon room and hosted the discussion here.

---

## Consensus & decisions by end of Day 1

1. **Threshold goal met.** Scientists agree QuickNXSv2 is the right
   starting point for modularization. The `data_handling/` tree already
   has zero Qt imports — half the front-end/back-end separation is in
   place.
2. **Two instruments by design.** Whatever the back-end ends up being
   named (`ref_core` in the pre-deck), it must support REF_M and REF_L.
   LR cannot be blocked on MR decisions.
3. **Architecture direction.** Model-View-Presenter at the top; a
   **Strategy pattern** under it to encapsulate per-instrument differences
   (LR dict / MR dict descriptors). QuickNXSv2 is the better MVP starting
   point; lr_reduction's "refined" ideas fold in through the Strategy
   layer.
4. **First concrete tier-1 PR candidate.** A single
   `build_mrr_kwargs(configuration, …)` used by both the GUI and
   `mr_reduction` autoreduce — retires the five silent MRR default
   disagreements in one blow.
5. **Events vs histograms is a first-class design decision**, not an
   implementation detail. Formalize as a `{Event, Histogram} focus`
   strategy choice.
6. **Diagnostic plots = required capability**, not optional. The UI must
   let the user see intermediate steps (projected peak vs. detector,
   binning trace, background-band placement) without leaving the
   reduction.
7. **Overplot / compare tab and template-deviation indicator** are
   high-value UX features that require back-end cooperation.
8. **Reproducibility first.** No silent black-box behavior; same inputs
   must yield same outputs across days. ORSO-format output flagged as a
   candidate for chunked, self-describing reports.

## Explicit non-decisions (revisit Day 2+)

- **Single UI with a mode switch vs. two deployed binaries.** Strong
  preferences on both sides; the fact that they come from one codebase
  makes this largely a packaging/UX choice rather than an architectural
  one, but it is unresolved.
- **Pre-processed transmission files as an alternative to DB** —
  supported in principle; logistics (tool ownership, who runs it, per
  experiment or shared) open.
- **Scope of autoreduce touch-up.** Everyone agrees autoreduce consumes
  the same back-end; the question of *when* during the hack-a-thon the
  autoreduce call-site is re-plumbed was deferred.

## Open questions carried into Day 2

- What is the **minimum-viable** surface of the shared back-end by end of
  week? (Volker / Tim / Marie framing.)
- How modular can QuickNXSv2 realistically become in the hack-a-thon
  timebox without breaking the existing scientist workflow? (Volker's
  gating question, p.12.)
- Exactly which parameters are **global**, which are **per-run**, which
  are **per-PPS**. The side-background and match-direct-beam discussions
  both bumped into this and left it open.
- Is a templated run-metadata "large text box" edit surface preferable
  to the current fixed series? (Strategy-pattern candidate.)

---

## Notes on fidelity

- The Teams auto-transcripts are noisy (garbled words, speaker
  attribution missing). Where the transcript and the handwritten notes
  agree, I followed the notes; where only the transcript has a point,
  I've attributed it loosely.
- A few handwritten shorthand tokens were ambiguous to me — in
  particular the "UDICE" and "realm" annotations on pages 12 and 15,
  and several speaker initials where Valeria / Volker were
  hard to disambiguate. Please correct me in your review and I will
  fold the corrections into the Day 2 summary.
- The `quicknxsv2-modularization` branch's pre-meeting deck
  (`plan/quicknxsv2-modularization/slides/`) supplied the technical
  vocabulary ("silent defaults", "Strategy pattern", "peak-drag swim
  lane", `ref_core`) and was clearly the scaffold the morning
  discussion followed.

---
+648 −0

File added.

Preview size limit exceeded, changes collapsed.

+688 −0

File added.

Preview size limit exceeded, changes collapsed.

+208 −0

File added.

Preview size limit exceeded, changes collapsed.

Loading