Commit 2437500b authored by Bogdan Vacaliuc's avatar Bogdan Vacaliuc
Browse files

docs(plan): hack-a-thon-2026 knowledge base of actions and decisions



Systematic post-hack-a-thon reference (13 files, 2834 lines) modeled on
plan/quicknxsv2-modularization/ from the pre-hack-a-thon branch. Catalogs
the agreed-upon decisions, plans, EWM ticket structure (16138/16139/16140
plus gating defect 15830), child-story mapping onto the 39-item baseline,
the in-flight Phase-1 refactor work on origin/mvp-rebased, the 30 deferred
items, and the reconciliation against the pre-hack-a-thon investigation.
Source: the five daily summaries, the EWM Estimation Baseline, the Desktop/
supplementary submodule, and the quicknxsv2 mvp/mvp-rebased branches.

Co-Authored-By: default avatarClaude Opus 4.7 (1M context) <noreply@anthropic.com>
parent 34a96295
Loading
Loading
Loading
Loading
+172 −0
Original line number Diff line number Diff line
# 00 · Executive Summary

The Hack-A-Thon 2026 (2026-04-22 → 2026-04-28) was a five-day session
between the Reflectometry scientists and the QuickNXS / `mr_reduction`
developers, plus invited contributors from leadership (Addi
Malviya-Thakur, John Hetrick) and adjacent groups (Maria Patrou for
the Day-2 MVP tutorial; the Mantid Future cross-facility group on
Day 3). The week converted a long-running architectural conversation
into a concrete, ticket-shaped plan, with the first code lifts
written and pushed during and immediately after closeout.

## What the hack-a-thon decided

### The central architectural commitment

The team unanimously ratified a **three-step strategic ordering**
(Day 4 §6.1, reconfirmed Day 5 §10):

1. **Restructure QuickNXS** into front-end (Qt + UI) and back-end
   (`mr_reduction`) — the lower-risk first move that does not block
   on LR.
2. **Work on the LR `new_workflow` back-end** in parallel, hardening
   Becky's branched code with developer support.
3. **Integrate LR into the QuickNXS back-end** — the higher-risk
   capstone, deferred until steps 1 and 2 are far enough along.

### The architectural pattern

The team adopted **Model-View-Presenter** as the target pattern
(Day 2 §10–11, ratified by Maria Patrou's tutorial; refined Day 4
§14.2 with the `research_mvp_architecture.md` 6-phase migration plan;
Day 5 §10.1 settled the layout question as a **hybrid** — top-level
`model/` / `view/` / `presenter/` with feature-level subdirectories
inside, e.g. tabs as a view-subdirectory).

The **two-phase Mutation+Render pattern** (Phase 1: mutate model;
Phase 2: render view) replaces the fragile `auto_change_active`
recursive-event guard.

### The Configuration split

The Day-3 P3 problem (the `Configuration` god class with
class-attribute global state) is resolved by a **three-way split**
documented in the Copilot research artifact `research_data_objects.md`:

- `GlobalReductionSettings` — session-wide, owned by the application.
- `RunConfiguration` — per-run, owned by `NexusData` (no longer
  redundantly deep-copied across each `CrossSectionData`).
- `PlotOptions` — display-only fields, owned by the Qt frontend.

The `set_parameter` / `get_parameter` workarounds on `NexusData`
disappear with this split.

### The EWM ticket structure

Generated by COB Day 5 (Day 5 §11.5):

| EWM | Type | Title |
|---|---|---|
| **15830** | Defect (existing, located+adopted) | `[QuickNXS] Several problems in the output of Off Specular Raw, Smoothed and Binned data` |
| **16138** | EPIC | `[QuickNXS] Add support for the Liquids Reflectometer` *(parent EPIC; capability)* |
| **16139** | EPIC (child of 16138) | `[QuickNXS] Address technical debt and architecture` *(blocks 16140; the 39-item refactor backlog rolls up here)* |
| **16140** | EPIC | `[QuickNXS] Integrate with the Liquids Reflectometer backend` *(blocked by 16139)* |

[Chapter 04](04-ewm-ticket-tree.md) is the full tree with parentage,
blocking relationships, and the child-story mapping against the
Day-3 39-item backlog.

### Empirical effort baseline

From `tasking/Hack-A-Thon-2026-EWM-Estimation-Baseline.md`, derived
from 1,076 historical EWM-tracked items:

| Quantity | Median | p80 |
|---|---:|---:|
| Focused effort | 358 h | 816 h |
| Calendar at REF-cap, 2× process-tax | 5.3 mo | 12.0 mo |
| Calendar at REF-cap, 3× process-tax (central) | 7.9 mo | 18.0 mo |

Glass's verbal Day-3 estimate (≈3 mo optimistic / ≈1 yr realistic)
is consistent with the empirical bands; the dominant risk is
*process tax*, not under-estimation of work itself (the team's
median `Time Spent / Estimate` ratio is 1.00 across every work-class).

The single highest-leverage item identified is **MG6**
(`build_mrr_kwargs()`, 29 h median / 62 h p80) — retires the
five Day-1 silent default mismatches in one PR.

## What the hack-a-thon produced as code

The `Phase one refactor` commit (`d5dc9a9` in `quicknxsv2`,
authored 2026-04-28) lands on both the `mvp` and `mvp-rebased`
branches and implements:

- File moves: `interfaces/data_handling/*``models/`;
  `interfaces/event_handlers/*``presenters/`;
  `interfaces/*.py` (dialogs, main window, plotting) → `views/`;
  `ui/*.ui``views/qt/`; `ui/*.py` (widgets) → `views/widgets/`.
- Renames: `gui.py``main.py` (DI4); status-bar handler →
  `views/status_bar.py`; main_handler → `presenters/main_presenter.py`.
- Top-level promotions: `enums.py`, `exceptions.py`, `utils/`.
- Deletions: `settings.json` (Day-4 §16.5); GenX templates folder
  (Day-2 + Day-3 + Day-5 confirmed); duplicated
  `interfaces/data_handling/dead_time_correction.py` (TD1).
- Test-tree moved from `test/` to `tests/` (post-Phase-1 commit
  `2f7dff8` *fix test imports*).

[Chapter 06](06-mvp-branches-implementation.md) catalogs the file-by-file
mapping and which Day-5 stories the commit closes outright vs. partially.

## What the hack-a-thon deferred

[Chapter 08](08-non-decisions-and-deferred.md) is the full list. The
top items:

- **Whether `lr_reduction` merges into `mr_reduction`** — explicitly
  deferred Day 2 §8; EWM16140 is structured to *not* foreclose either
  way.
- **Live reduction support** — captured Day 5 §10.2 as
  out-of-hack-a-thon-scope; the intermediate *download +
  sliding-1-hour-sum + monitor.sns.gov* workflow is the bridge.
- **Mantid substrate** — the Day-3 optional Mantid Future meeting did
  not close on a substrate (SCIPP / Bluesky / streams etc.);
  visualization and performance working groups to form.
- **Hack-A-Thon #2 calendar date** — committed Day 2 to before the
  2026-06-25 SNS beam outage; not pinned during Day 5 closeout.

## What is committed going forward

[Chapter 09](09-going-forward.md) is the full action register. Headlines:

1. **Open the refactor branch in `quicknxsv2`** so Phase 1 work has
   a CI-running, scientist-deployable, team-visible target — *done*
   (`mvp` and `mvp-rebased` branches now exist).
2. **2-day LR `new_workflow` code deep-dive.** Bogdan + Becky to
   harden first; team-wide deep-dive with Marie / Glass / Kevin to
   follow. Date TBD ("this week / next week" per Addi at closeout).
3. **EWM15830 (off-spec output) is the gating critical fix**
   Valeria provides the V1 baseline output; Asmaa+Valeria own the
   comparison; Marie owns the V2 code touch.
4. **EWM16139 child stories** — being entered against `EWM16139`
   from the `Notes 04-28-2026.docx` *Proposed Stories to Write*
   list. Critical-path analysis is a post-hack-a-thon Kevin commitment.
5. **NDIP `refl1d` 0.11 → 1.01** — Bogdan's commitment from Day 4.
6. **Hack-A-Thon #2 scheduling** — open commitment, before
   2026-06-25.

## How this plan exits the week

The artifact-trail leaving the hack-a-thon, in citation-ready order:

- Slide 2 of `Desktop/2026-Hack-A-Thon-Bogdan.pptx` — the citation-
  grade closeout summary (the four EWM IDs and titles).
- The five daily summaries — long-form narrative.
- This knowledge base — systematic decision/action review.
- The EWM Estimation Baseline — empirical effort numbers.
- `Notes 04-28-2026.docx` (the Day-5 Google Doc) — the *Proposed
  Stories to Write* list, the consolidated tech-debt taxonomy, and
  the *Functionality and classes that should be in the backend* table.
- `research_mvp_architecture.md` and `research_data_objects.md`
  — the two Copilot-authored architectural references in the
  Google Drive folder.
- The `mvp` / `mvp-rebased` branches in `quicknxsv2` — the in-flight
  code work.

The top-line for the report-writer: **a multi-month back-end
refactor was scoped, ticketed, and started during this week.**

---

*Continue to [01-decisions-by-day.md](01-decisions-by-day.md) for
the chronological consensus ledger.*
+241 −0
Original line number Diff line number Diff line
# 01 · Decisions by Day — Chronological Consensus Ledger

This chapter is the **strict chronological list** of decisions and
consensus the team reached, day by day. Source: the *Consensus &
decisions* sections of the five daily summaries
(`tasking/Hack-A-Thon-2026-Day{1,2,3,4,5}-Summary.md`). Numbering
restarts each day to match the daily-summary numbering.

---

## Day 1 — 2026-04-22 · Architectural feasibility

**Threshold goal met:** scientists agreed QuickNXSv2 is the right
starting point for modularization.

1. **QuickNXSv2 is "V1\* + modularize" and must support LR.** The
   existing zero-Qt-import status of `interfaces/data_handling/` is
   the half-done starting point.
2. **Two instruments by design.** The shared back-end (working name:
   `ref_core`) must support REF_M and REF_L; LR cannot be blocked on
   MR decisions.
3. **Architecture direction:** Model-View-Presenter at the top, a
   **Strategy pattern** under it for per-instrument differences
   (LR-dict / MR-dict descriptors). *(The Strategy-pattern framing
   was Day 1; Day 2's MVP tutorial reframed the Strategy work as
   internal to the Model layer. See [Chapter 07 §2](07-pre-hack-a-thon-positions.md#2-strategy-pattern--mvp-presenter-mapping).)*
4. **First concrete tier-1 PR candidate:** `build_mrr_kwargs(...)`
   shared by GUI and `mr_reduction` autoreduce — retires the five
   Day-1 silent MRR-default disagreements in one move.
5. **Events vs histograms is a first-class design decision.**
   Formalize as an `{Event, Histogram} focus` Strategy choice.
6. **Diagnostic plots are a required capability**, not optional.
7. **Overplot / compare tab and template-deviation indicator** are
   high-value UX features that require back-end cooperation.
8. **Reproducibility first.** No silent black-box behavior. ORSO
   format flagged as a candidate for chunked, self-describing reports.

**Day-1 explicit non-decisions:** single-UI-with-mode-switch vs.
two-deployed-binaries; pre-processed transmission file logistics;
when (during the hack-a-thon) to re-plumb the autoreduce call site.

---

## Day 2 — 2026-04-23 · Computations and the back-end target

**Threshold goal met:** the team understands the computations
required across every reduction module well enough to plan the
refactor.

1. **QuickNXSv2 can solve LR.** Endorsed as the shared front-end.
2. **`new_workflow` is the target reduction lineage** for LR (Becky
   + Erik's position; the old refined path stays for compatibility).
3. **MVP pattern adopted** (pending detailed design). Maria Patrou's
   tutorial is the reference; Mantid MVP tutorial cited.
4. **Two-list planning model.** Going forward: (a) refactor list for
   QuickNXS / `mr_reduction`; (b) LR-wants list from Becky's xlsx and
   the Google Doc.
5. **Remove GenX templates** — redundant with ORSO.
6. **Tests doctrine.** Keep the small-yet-effective ones. After
   FE/BE separation, QuickNXS-layer tests mock `mr_reduction`;
   `mr_reduction` owns validation. Integration tests in GitLab-CI
   are a known debt; GitHub Actions cannot replicate them.
7. **Scope commitment.** *Do not* merge `lr_reduction` and
   `mr_reduction` this week. Keep that option open and do nothing
   this week that forecloses it.
8. **Follow-on cadence committed.** Hack-a-thon → 2-day deep-dive
   (Bogdan / Eric / LR team) → **second Hack-A-Thon dedicated to
   the back-end refactor** — all before the **2026-06-25 →
   2026-08-04 SNS beam outage**.
9. **Day 3 charge:** Marie delivers LR-wants bullet doc; Kevin /
   Glass own the Google Doc; everyone produces refactor-story
   estimates.

**Day-2 explicit non-decisions:** Qt vs. web for the front-end;
application launch mode (CLI vs. splash vs. menu); LR merge into
`mr_reduction`; match-direct-beam procedure ownership; time-slicing
/ event-filtering scope.

---

## Day 3 — 2026-04-24 · Tech-debt triage and the unanimous BE vote

**Threshold goal met; stretch goal substantially met:**
formal-requirements doc produced (Glass's Google Doc + Bogdan's
AI tech-debt summary), with a migration table.

1. **Threshold met / stretch substantially met** — Google Doc plus
   AI tech-debt summary.
2. **UNANIMOUS: all reduction computations land in the back-end.**
   Front-end holds only pointers / run numbers / file paths /
   template references / plotting choices. Kevin's framing:
   *"Model as transfer point."*
3. **Event boundary pushed downstream.** *"Keep in event until
   render / plot / write out."*
4. **No black boxes for numerical transforms.** Use explicit
   numpy/scipy linear operations where possible; Mantid's `Rebin`
   is no longer the default for inspectable intermediates.
5. **Tests:** 60–70% coverage is the sweet spot; whole-workflow
   tests beat unit tests alone. MR team will provide representative
   run numbers; per-test fixtures can be public (DOI-pinnable).
   GISANS test data is a gap that must be filled.
6. **Concrete tech-debt backlog** — see Chapter 04 and the Google
   Doc; 15 items captured.
7. **Back-end migration table** — 14 items captured (GISANS calc,
   off-spec calc, Python-script generation, reduced-data `.dat`,
   unit conversion, MRR consolidation, `DataInfo`, plot-facing
   pre-processing, `interfaces/data_handling/` tree, rebinning,
   summing, stitching, direct-beam matching, peak finding).
8. **Developer-insight cleanups** — 7 items captured (obsolete
   `#pylint`, `auto_change_active``blockSignals()`, params
   from `CrossSectionData``NexusData`, `gui.py``main.py`,
   bare `except` cleanup, prune TODOs, naming consistency).
9. **Scientist TODOs captured.**
10. **Ordering heuristic for Monday's deep-dive:** *(modularity,
    add LR, which BE)*; Tim — *"choose order wisely — gets us to
    finish sooner."*
11. **Effort bracketing — not a commitment.** Glass's verbal
    ≈3 mo optimistic / ≈1 yr realistic; later corroborated by the
    EWM Estimation Baseline at 358 h median / 816 h p80.

**Day-3 explicit non-decisions:** monorepo vs. separate repos;
data-ownership philosophy (all-data-in-BE vs. pass-back-and-forth);
Mantid substrate; error-weighting Poisson↔Gaussian ruling.

---

## Day 4 — 2026-04-27 · The most consequential day

**Threshold goal:** generate EWM tickets — *on track*; **stretch
goal:** code review of agentic AI development artifacts —
*partially met* via the Copilot-authored
`research_mvp_architecture.md`.

1. **Three-step strategic ordering — UNANIMOUS.**
   (1) restructure QuickNXS BE/FE; (2) work on LR new BE;
   (3) integrate LR into QuickNXS BE. The QuickNXS work does not
   need to wait for LR. *(See [Chapter 02](02-strategic-three-step-ordering.md).)*
2. **Three critical scientific issues catalogued and assigned**
   (off-spec output: Asmaa+Valeria; run summing: Becky/QA; off-spec
   binning: Marie UI-hide). Independent of refactor work.
3. **Working Plan ratified.** Freeze main dev branches (`next` /
   `qa` / `prod`); branch from a *refactor branch* (not from
   `next`); address low-hanging fruit while designing the structure.
4. **MVP architecture adopted as the target.**
   `research_mvp_architecture.md` is the working reference.
   Two-phase Mutation+Render replaces `auto_change_active`.
5. **Eliminate `settings.json` and the `Settings` singleton.**
6. **`Configuration` class to be split** — Global / Per-Run / Plot.
7. **GenX templates removed** *(Day-2 decision now confirmed in the
   Working Plan).*
8. **Refactor branch isolation strategy committed.** Refactor
   branches branch from the refactor branch, not from `next`.
9. **NDIP confirmed as deployment substrate** for both Qt and web
   frontends. **Bogdan committed to NDIP `refl1d` 0.11 → 1.01.**
10. **AI-augmented refactor is in scope.** Copilot's
    `research_mvp_architecture.md` (962 lines) is itself a working
    artefact.
11. **Better developer docs** — committed as a Working-Plan side
    deliverable (`git-lfs`, pre-commit hooks, dev setup in README).
12. **Tomorrow's threshold goal narrowed:** generate technical-debt
    stories.

**Day-4 explicit non-decisions:** feature-based vs. top-level MVP
structure; how many distinct presenters; whether tabs are features;
when to start touching UI files; specific CI/test strategy for
off-spec V1↔V2 parity.

---

## Day 5 — 2026-04-28 · Closeout

**Threshold goal met.** Three EPICs + one gating defect committed
in EWM. *(Stretch goal — full BE/FE separation with passing tests
— always understood as a multi-month arc; the in-scope-for-the-week
portion (start Phase 1 refactor work) was begun and is now in the
repo as the `mvp` / `mvp-rebased` branches.)*

1. **Threshold goal met.** EWM15830 + EWM16138 + EWM16139 + EWM16140.
2. **Three-step strategic ordering survives Day 5 unchanged.**
3. **Hybrid MVP layout — agreed.** Top-level `model/` / `view/` /
   `presenter/` with feature-level subdirectories *inside*
   (e.g. tabs as a view-subdirectory).
4. **camelCase → snake_case migration** committed (bundled into the
   general-cleanup story).
5. **Configuration three-way split**`GlobalReductionSettings` /
   `RunConfiguration` / `PlotOptions` per `research_data_objects.md`
   §12.
6. **EWM15830 is the one critical issue gating refactor work.**
   Valeria provides V1 baseline output; Marie owns V2 code touch.
7. **Run summing fix confirmed by Asmaa** in QA. Day-4 §6.2 item 2
   closed.
8. **Off-specular rebinning hidden in UI** — confirmed; Marie owns
   the UI change as part of EWM15830.
9. **Live reduction is out of hack-a-thon scope.** Recorded as a
   side-quest. Intermediate three-step workflow (download →
   sliding-1-hour-sum → monitor.sns.gov) is the bridge.
10. **TODO-comment triage rule.** Bogdan to catalogue
    `{FIXME, INFO, …}` markers; every marker becomes a story with
    an owner, or it gets removed. **No orphan stories.**
11. **CIS-input gate.** Three items need a developer ruling before
    their stories enter the queue:
    (a) two `_as_ints` in `data_set.py`; (b) event-count cutoff
    mismatch (QuickNXS=100 vs. mr_reduction=200);
    (c) `quicknxs_scaling_factor` triplication and `+1`
    divide-by-zero hacks.
12. **2-day code deep-dive on LR `new_workflow` committed.** Bogdan
    + Becky to harden first. John wants it in a separate branch
    from `next`. Date TBD.
13. **Critical-path analysis** is a post-hack-a-thon action
    (Kevin's commitment).

**Day-5 explicit non-decisions:** EWM16140 LR-integration scope (the
deferred Day-2 merge question stays deferred); Hack-A-Thon #2
calendar date (open commitment, before 2026-06-25); live-reduction
return path (no owner / no ticket); `quicknxs_io.py` placement;
whether each tab gets its own presenter; critical-path analysis
itself.

---

## Cumulative count

| Day | Threshold goal | Stretch goal | Decisions logged | Non-decisions |
|---|---|---|---:|---:|
| 1 | ✅ met | n/a | 8 | 3 |
| 2 | ✅ met | n/a | 9 (last 2 are charges, not decisions) | 5 |
| 3 | ✅ met | substantial | 11 | 4 |
| 4 | ✅ on-track | partial (via Copilot) | 12 | 5 |
| 5 | ✅ met | not in scope | 13 | 6 |
| **Total** | — | — | **53** | **23** |

The **23 non-decisions** are tracked in
[Chapter 08](08-non-decisions-and-deferred.md). The
**53 decisions** distill into the EPIC structure in
[Chapter 04](04-ewm-ticket-tree.md) and the architectural
position in [Chapter 03](03-mvp-architecture-final.md).

---

*Continue to [02-strategic-three-step-ordering.md](02-strategic-three-step-ordering.md)
for the central architectural commitment.*
+145 −0

File added.

Preview size limit exceeded, changes collapsed.

+343 −0

File added.

Preview size limit exceeded, changes collapsed.

+257 −0

File added.

Preview size limit exceeded, changes collapsed.

Loading