update after 2nd prompt

# Analysis: CHI Scan Integration into the Scantools Framework

**Date:** 2026-04-06

**Date:** 2026-04-08 (updated from 2026-04-06 draft)

**Branch:** bl4b:issue/4850, scantools:issue/4850

**Redmine:** #4850

**Status:** Work in progress — functional for xprofile/xy detector selection, discrepancies remain

> **Note on file locations:** The scantools shared module lives at

> `/home/controls/common/scantools/{main,issue/4850}/` (symlink resolves to

> `/media/ssd1/beamlines/common/scantools/...`). The bl4b repo symlinks

> `python/scantools/ScanTools -> ../../../common/scantools/issue/4850/python/ScanTools`.

> The scan server's Jython scripts are at `/home/controls/share/scan/` (resolves to

> `/media/ssd1/beamlines/share/scan/`). Phoebus source at `/home/controls/src/phoebus/`.

---

## 1. Executive Summary

## 5. Data Flow in the IOC-Integrated Version

### 5.1 Scan Execution Sequence

### 5.1 The `WriteDataToPV` Jython Script — The Critical Bridge

The scan server does **not** directly write to FittingIOC PVs. Instead, it executes Jython scripts via `Script` commands. The `WriteDataToPV` script at `/home/controls/share/scan/writedatatopv.py` is the bridge:

```python

# Jython, runs inside scan server JVM

class WriteDataToPV(ScanScript):

    def __init__(self, device, pv, norm_device=None, norm_value=1.0):

        self.device = device        # source device name (e.g., motor or detector)

        self.pv = pv                # target PV (e.g., BL4B:CS:Fitting:Data:X)

        self.norm_device = norm_device

        self.norm_value = norm_value

    def run(self, context):

        if self.norm_device:

            data = np.array(context.getData(self.device, self.norm_device))

            normed = data[0] * float(self.norm_value) / data[1]

            data = array(normed, 'd')       # Java double array

        else:

            data = context.getData(self.device)

            data = data[0]                  # first (and only) column

        context.write(self.pv, data, False)  # False = no completion wait

```

**Key behavior:** `context.getData(device)` retrieves **all logged samples for that device** accumulated since the scan started. So at step N, it returns an array of N values (all motor positions logged so far, or all detector values). The script writes this **entire accumulated array** to the target PV.

This means:

- `Data:X` receives `[chi_0]`, then `[chi_0, chi_1]`, then `[chi_0, chi_1, chi_2]`, etc.

- For scalar detectors: `Data:Y` receives the same accumulated pattern

- For 2D detectors: `Data:XY` receives the **latest waveform** (context.getData returns the most recent logged waveform, not an accumulated array of waveforms)

### 5.2 Scan Execution Sequence (Corrected)

The `Alignment.createScan()` method (in `ScanTools/align/__init__.py`) constructs this scan sequence:

```python

# From Alignment.createScan() (lines 168-186)

x_pv = f'{self.fitting_pv}Data:X'

if self.detector == resolveAlias('xy'):

    y_pv = f'{self.fitting_pv}Data:XY'          # route to Data:XY PV

elif self.detector == resolveAlias('xprofile'):

    y_pv = f'{self.fitting_pv}Data:XPROFILE'     # route to Data:XPROFILE PV

elif self.detector == resolveAlias('yprofile'):

    y_pv = f'{self.fitting_pv}Data:YPROFILE'     # route to Data:YPROFILE PV

else:

    y_pv = f'{self.fitting_pv}Data:Y'            # standard scalar path

loop_body = [

    wait,                                         # Wait for pcharge criteria

    Log(log_devs),                                # Log all devices to scan data

    Script('WriteDataToPV', [self.motor, x_pv]),  # Write motor positions → Data:X

    Script('WriteDataToPV', [self.detector, y_pv, norm_device, norm_value]),  # Write detector → Data:XY/XPROFILE/Y

    Wait(y_pv, None)                              # WAIT for FittingIOC completion callback

]

```

**Full execution timeline:**

```

 User clicks "Run" in Alignment OPI

           │

           ▼

 AlignmentIOC receives Run=1

 AlignmentIOC.run() → creates Alignment object → calls alignment.run()

           │

           ▼

 AlignmentIOC constructs scan config:

   motor='BL4B:Mot:chis', start=-5.0, end=5.0, step=0.5

   criteria='pcharge', criteriaInc=1.0e-3 (1 mC)

   detector='xprofile'  (or 'xy')

   method='Gauss+const' (via Fitting:Method)

 Alignment.createScan() builds scan command sequence

 Alignment.scan() submits to Scan Server via scan_client.submit()

           │

           ▼

 AlignmentIOC submits scan to Scan Server via scan_client

           │

           ▼

 ┌─────── Scan Server Loop ──────────────────────────────────────────┐

 ┌─────── Scan Server Loop (Java) ──────────────────────────────────────┐

 │  For each step i:                                                     │

 │    1. Set BL4B:Mot:chis = start + i*step                          │

 │    2. Wait for motor completion                                     │

 │    3. Wait for criteria (accumulate pcharge)                        │

 │    4. Log Data:X (append motor position)                            │

 │    5. Log detector PV → writes to Fitting:Data:XPROFILE (or XY)    │

 │       └──▶ FittingIOC.write('Data:XPROFILE', ...) is triggered     │

 │            └──▶ caget(resolveAlias('xprofile'))  [fetch real data]  │

 │    1. Set BL4B:Mot:chis = start + i*step (with completion)            │

 │    2. Wait for pcharge criteria                                        │

 │    3. Log(all_devices) → records scalar values to internal Derby DB    │

 │    4. Script('WriteDataToPV', [motor, Data:X])                         │

 │       └──▶ Jython: context.getData(motor) → all chi values so far     │

 │       └──▶ context.write(Data:X, [chi_0,...,chi_i], False)             │

 │       └──▶ FittingIOC.write('Data:X', [chi_0,...,chi_i])              │

 │            └──▶ setParam('Mark:Xlast', chi_i)  ◄── tracks current pos │

 │    5. Script('WriteDataToPV', [detector, Data:XPROFILE])               │

 │       └──▶ Jython: context.getData(xprofile) → latest 1D projection   │

 │       └──▶ context.write(Data:XPROFILE, projection_data, False)        │

 │       └──▶ FittingIOC.write('Data:XPROFILE', ...) triggered            │

 │            └──▶ caget(resolveAlias('xprofile'))  [re-fetch real data]  │

 │            └──▶ trigger_data_to_y_and_err('Data:XPROFILE')             │

 │                 └──▶ extract_XProfile_pos(Xproj)                       │

 │                      └──▶ returns (MeanX, ErrX, SigmaX, ...)       │

 │                      returns (MeanX, ErrX, SigmaX, ErrSigmaX, DiagXY) │

 │                 └──▶ Data:Y.append(SigmaX)                             │

 │                 └──▶ Data:Err.append(ErrSigmaX)                        │

 │            └──▶ trigger_fit()  [fit accumulated Data:X vs Data:Y]      │

 │            └──▶ callbackPV('Data:XPROFILE')  [signal completion]       │

 │    6. Scan server proceeds to next step                             │

 └────────────────────────────────────────────────────────────────────┘

 │    6. Wait(Data:XPROFILE, None) → blocks until callbackPV fires        │

 │    7. Scan server proceeds to next step                                │

 └────────────────────────────────────────────────────────────────────────┘

           │

           ▼

 Alignment.scan() returns → data table from scan_client.getData()

           │

           ▼

 AlignmentIOC reads fit results:

   Fit:Pos = optimal chi position

   Fit:Amp, Fit:Wid, Fit:Base = Gaussian parameters

   State = OK/Error

 Alignment.align() checks detector type:

   if detector in ['xy','xprofile','yprofile']:

       x = y = None        # skip adjustData — IOC has accumulated Data:Y

   else:

       (x, y) = adjustData(data)   # extract from scan log, normalize

           │

           ▼

 If MoveToCenter: caput(BL4B:Mot:chis, Fit:Pos)

 Alignment.fit(x, y):

   if x is not None and y is not None:

       caput(Data:X, x)    # overwrite with potentially filtered data

       caput(Data:Y, y)    # overwrite with normalized data

   # For 2D detectors: x=y=None, so skip — IOC has our data already

   caput(Method, self.method, wait=True)   # triggers final fit with completion

   position = caget(Fit:Pos)

           │

           ▼

 If MoveToCenter: Alignment.setDevice(motor, position)

```

### 5.2 What Gets Accumulated in Data:Y

- **09cd8b8** ("add perform_fitXdist_for_chis()"): Simplified chi_scan.py, removed the standalone `perform_fitXdist_for_chis()` function

- **e1ccfb0** ("glitched"): Changed `if True:` to `if FitXdist:` in `extract_XProfile_pos()`, added `else: plt.cla()` branches

### 6.2 Syntax Issue at Line 390

### 6.2 `chifit` Undefined Variable in Scantools chi_scan.py

**Critical bug.** In the scantools `issue/4850` copy of `chi_scan.py` (at `/home/controls/common/scantools/issue/4850/ioc/fit/chi_scan.py`), the `extract_XY_pos()` function references `chifit` at lines 149 and 164, but this variable is never defined within that function:

In the commented-out code block, line 390 has a malformed comment:

```python

#3    iy=GaussDip_ConstantBkg(chifit, py[0],py[1],py[2],py[3])

def extract_XY_pos(XYarray):

    ...

    if True:

        print('Chi = ',str(np.round(chifit,3)),' StdDevX: ', np.round(SigmaX,2))  # ← chifit undefined!

    ...

    return chifit, ErrX, SigmaX, ErrSigmaX, ...  # ← chifit undefined!

```

This should be `##` not `#3`. This is in dead code (doubly-commented) so it has no runtime effect, but indicates hasty editing.

### 6.3 `Data:Y` Accumulation Semantics

The bl4b repo copy has `if FitXdist:` guarding this code, which means it only runs when FitXdist=True. But `chifit` is still referenced in the return statement. In the bl4b copy, the `FitImage` code path sets `chifit` when it runs, and there's an `else: chifit=0.0` fallback — but this only works if the `if FitImage:` block precedes the `if FitXdist:` block.

**Potential discrepancy:** When the scan server writes `Data:Y` directly (the normal, non-chi case), the `write()` handler replaces `Data:Y` with the incoming value and computes `Data:Err = sqrt(value)`. But when a waveform PV (XPROFILE/XY/YPROFILE) triggers `trigger_data_to_y_and_err()`, the function *appends* to the existing `Data:Y`.

**Impact:** When using `extract_XY_pos()` with the scantools version, this will raise a `NameError` at runtime if `FitImage=False` (which it is by default). The bl4b version partially fixes this but the code paths are fragile.

This means:

- If the scan server also writes `Data:Y` (as it normally does for the detector counts), the two paths would conflict — the scan server would overwrite the accumulated SigmaX values

- The current code handles this by having `Data:Y`'s write handler call `callbackPV` in `finally`, which signals completion. If the detector PV writes happen *before* `Data:Y` writes, the accumulated values could be overwritten

### 6.3 Two Copies of chi_scan.py — Divergence Risk

There are **two copies** of `chi_scan.py`:

1. `/home/controls/common/scantools/issue/4850/ioc/fit/chi_scan.py` — the scantools repo version (199 lines)

2. `/home/controls/bl4b/applications/bl4b-ScanSupport/iocBoot/bl4b-Fit/chi_scan.py` — the bl4b repo version (454 lines)

These have already diverged. The bl4b version has more code (includes `FitImage` mode logic, `FitXdist` conditional), while the scantools version has `perform_fitXdist_for_chis()` (which references undefined `curve_fit`, `ChiRange`, `ChiValue`). The bl4b `st.cmd` uses the local copy, not the symlinked scantools version. **Future changes must be coordinated across both repos.**

### 6.4 `Data:Y` Accumulation — Resolved by Design

**This is likely the source of one of the observed discrepancies.** The scan configuration needs to ensure that only the waveform detector PV is logged (not the normal scalar detector), or the `Data:Y` write handler needs to be aware of chi-scan mode and not overwrite.

**Previously hypothesized as a problem, now confirmed as intentionally correct:**

The `Alignment.createScan()` method routes detector writes to different PVs based on detector type. When `detector='xprofile'`, the `WriteDataToPV` script writes to `Data:XPROFILE` (not `Data:Y`). The FittingIOC's `trigger_data_to_y_and_err()` then internally appends sigma values to `Data:Y`.

Since the scan server never writes to `Data:Y` for 2D/1D detectors, there is no conflict. However:

- The `WriteDataToPV` script calls `context.write(Data:XPROFILE, data, False)` with `False` (no completion wait). The scan server then separately executes `Wait(Data:XPROFILE, None)` to wait for the FittingIOC's `callbackPV()`.

- The FittingIOC's `write('Data:XPROFILE')` handler does a **second `caget()`** to fetch the real waveform data from the detector PV. This is because `context.write()` may not deliver the full waveform — it writes whatever `context.getData()` returned from the scan log, which may be a summary.

**Remaining concern:** The `context.getData(detector)` call in `WriteDataToPV` retrieves accumulated scan log data. For waveform PVs logged via `Log()`, the scan server stores the full array in its Derby database. But the FittingIOC ignores the incoming value and re-fetches via `caget()` anyway. This double-fetch is intentional but inefficient — the comment says "get vector from PV b/c scan server wont."

### 6.4 Module-Level Flags Are Static

- The flags must be edited in chi_scan.py and the IOC restarted

- A future enhancement should expose these as PVs or derive them from the scan configuration

### 6.5 Diagnostic Image Size Mismatch

### 6.5 `perform_fitXdist_for_chis()` Has Undefined References

The scantools version of `chi_scan.py` includes `perform_fitXdist_for_chis()` (lines 166-198) which is intended to fit the accumulated chi-vs-sigma data. This function references:

- `curve_fit` — imported as `scipy.optimize.curve_fit` in the bl4b version but **commented out** (`##from scipy.optimize import curve_fit`) in the scantools version

- `ChiRange` — a module-level constant that was commented out

- `ChiValue` — a loop variable from the original Spyder script that doesn't exist in the module

This function is not currently called from `FittingIOC.py`. The post-loop fit is instead handled by `trigger_fit()` which uses the `fittings` module. The `perform_fitXdist_for_chis()` function appears to be dead code left from an incomplete attempt to do custom post-loop fitting.

### 6.6 Diagnostic Image Size Mismatch

The `Diag:XY` PV is sized at 640×480 (307,200 elements), but `get_img_array_from_fig_iobuf()` returns whatever size the matplotlib figure canvas produces. The `reshape(w, h)` uses the canvas dimensions, which may not be 640×480 unless the figure is explicitly sized. If the figure size doesn't match, the PV write could fail or produce garbled output.

---

## 10. File Inventory

## 10. Scan Server Infrastructure

### 10.1 Scan Server Process

The scan server is a Java application running on the instrument control system:

```

java -jar service-scan-server-4.6.4.jar \

  -settings /home/controls/css/phoebus.ini \

  -settings /home/controls/bl4b/css/phoebus.ini \

  -config /home/controls/bl4b/scan_config.xml

```

It exposes an HTTP REST API (used by PyScanClient) and executes scan command sequences (Loop, Set, Wait, Log, Script, etc.). The scan server maintains an internal Derby database for logged data.

### 10.2 `scan_config.xml` — Scan Server Configuration

The configuration at `/home/controls/bl4b/scan_config.xml` defines:

- **Data log location:** `/home/controls/var/scan/db`

- **Pre/post scan scripts:** `/home/controls/bl4b/scan/pre_scan.scn`, `post_scan.scn`

- **Script paths** (for Jython Script commands):

  - `/home/controls/bl4b/scan`

  - `/home/controls/bl4b/python`

  - `/home/controls/share/master/python`

  - `/home/controls/share/master/scan`

- **79 named PVs** with optional aliases — these are the PVs the scan server knows about

**Important:** `Data:X`, `Data:Y`, `Data:XY`, `Data:XPROFILE` are **not** listed in scan_config.xml. They are accessed via the `WriteDataToPV` Jython script, which uses `context.write()` with full PV names passed as arguments.

### 10.3 Jython Script Commands

The scan server can execute Jython classes that implement `org.csstudio.scan.command.ScanScript`. Key scripts:

**`WriteDataToPV`** (`/home/controls/share/scan/writedatatopv.py`):

- Called via `Script('WriteDataToPV', [device, pv, norm_device, norm_value])`

- `context.getData(device)` retrieves all logged samples for that device from the scan

- Optionally normalizes: `data[0] * norm_value / data[1]`

- `context.write(pv, data, False)` writes the accumulated array to the target PV

- **No completion wait** (`False`) — the calling scan sequence must add its own `Wait()` if needed

**`ClearDataInPV`** (`/home/controls/bl4b/scan/cleardatainpv.py`):

- Writes a zeros array to a PV to clear previous scan data

- Used before alignment scans to reset the FittingIOC state

### 10.4 The Scantools Shared Module

The ScanTools module (`/home/controls/common/scantools/`) is shared across all SNS beamlines:

```

common/scantools/

├── main/                          # production release

│   ├── python/ScanTools/          # core module

│   │   ├── __init__.py

│   │   ├── devices.py             # Device class, find(), getNames()

│   │   ├── commands.py            # Set, Loop, Wait, Log, Script, etc.

│   │   ├── align/                 # Alignment class

│   │   │   └── __init__.py

│   │   ├── table.py               # TableScan

│   │   ├── macros.py              # Start/Stop/Diagnostic macros

│   │   └── util.py

│   └── ioc/

│       ├── align/AlignmentIOC.py

│       └── fit/

│           ├── FittingIOC.py

│           ├── fittings.py

│           └── EdgeFinderLib.py

└── issue/4850/                    # development branch

    ├── python/ScanTools/          # modified: align/__init__.py (detector routing)

    └── ioc/fit/

        ├── FittingIOC.py          # modified: 2D PV support, trigger_data_to_y_and_err()

        ├── chi_scan.py            # NEW: extraction functions

        └── st.cmd                 # modified: env_calc venv, PYTHONPATH

```

Each beamline has its own `python/scantools/` directory with:

- `__init__.py` — creates ScanClient, sets pv_prefix

- `devices.py` — beamline-specific Device definitions

- `ScanTools` — symlink to `common/scantools/{release}/python/ScanTools`

### 10.5 Scantools issue/4850 Changes (Summary)

Key commits on the scantools issue/4850 branch:

| SHA | Description |

|-----|-------------|

| `df6e3e7` | work-in-progress: merge with CHI_scan.py |

| `410cb2e` | checkpoint: CHI_scan works for both XPROFILE and XY |

| `7ef4c01` | use env_calc venv for matplotlib; report version |

| `ec5c144` | stripped down CHI_scan.py extraction |

| `7ce8971` | handle xy, xprofile, yprofile alternate detectors |

| `a31034d` | work-in-progress: FittingIOC 2D data for Redmine 4850 |

Changes span 8 files (+478/-112 lines), primarily:

1. **`align/__init__.py`**: Detector type routing, `Wait(y_pv, None)` for completion, skip adjustData for 2D

2. **`ioc/fit/FittingIOC.py`**: New async PVs, `trigger_data_to_y_and_err()`, callbackPV on all write handlers

3. **`ioc/fit/chi_scan.py`**: New file — extraction functions

4. **`ioc/fit/st.cmd`**: Uses env_calc venv, pre-flight syntax check

---

## 11. File Inventory

### Source Files (issue/4850 branch)

| `alignmentParameters.py` | `bl4b/python/beamline/alignmentParameters.py` | Scan parameters (CAXIS_STANDARD) |

| `pv_names.py` | `bl4b/python/beamline/pv_names.py` | PV short name cross-reference |

### Scantools Shared Module (issue/4850 branch)

| File | Path | Role |

|------|------|------|

| `align/__init__.py` | `common/scantools/issue/4850/python/ScanTools/align/__init__.py` | Alignment class — detector routing, scan construction |

| `commands.py` | `common/scantools/issue/4850/python/ScanTools/commands.py` | Scan command wrappers (Set, Loop, Wait, Log, Script) |

| `devices.py` | `common/scantools/issue/4850/python/ScanTools/devices.py` | Device class definition, find(), getNames() |

| `chi_scan.py` | `common/scantools/issue/4850/ioc/fit/chi_scan.py` | Analysis functions (scantools copy, diverged from bl4b) |

| `FittingIOC.py` | `common/scantools/issue/4850/ioc/fit/FittingIOC.py` | FittingIOC (scantools copy) |

### Scan Server Scripts

| File | Path | Role |

|------|------|------|

| `writedatatopv.py` | `/home/controls/share/scan/writedatatopv.py` | Jython: writes accumulated scan data to FittingIOC PVs |

| `cleardatainpv.py` | `/home/controls/bl4b/scan/cleardatainpv.py` | Jython: clears PV data before scan |

| `scan_config.xml` | `/home/controls/bl4b/scan_config.xml` | Scan server PV list, script paths, pre/post scan |

| `pre_scan.scn` | `/home/controls/bl4b/scan/pre_scan.scn` | Pre-scan: wait for idle, stop detector |

| `post_scan.scn` | `/home/controls/bl4b/scan/post_scan.scn` | Post-scan: set run control stop |

### Spyder Scripts (reference originals)

| File | Path | Integration Status |

---

## 11. Tasks for Future Agents

## 12. Tasks for Future Agents

### Immediate (Complete CHI Scan Integration)

1. **Resolve `Data:Y` conflict** — determine whether the scan server also writes `Data:Y` when a waveform detector is selected, and if so, prevent overwriting the accumulated sigma values. May need to conditionally skip the normal `Data:Y` write path when in chi-scan mode.

1. **Fix `chifit` undefined variable** — in both copies of `chi_scan.py`, the `extract_XY_pos()` function references `chifit` before it is defined when `FitImage=False`. The bl4b version partially guards this with `if FitXdist:` but still returns `chifit` which may be unset. Add `chifit = 0.0` initialization at the top of `extract_XY_pos()`.

2. **Unify the two chi_scan.py copies** — the bl4b and scantools repos have diverged. Decide which is canonical and ensure the other is a symlink or identical copy. The `st.cmd` PYTHONPATH currently uses the local bl4b copy.

2. **Verify fit function compatibility** — confirm that the `fittings` module's "Gauss+const" or "Gauss+slope" methods produce equivalent results to the Spyder script's `Gauss_ConstantBkg` for fitting sigma-vs-chi curves.

3. **Fix `perform_fitXdist_for_chis()` or remove it** — the scantools version has this function but it references undefined variables (`curve_fit`, `ChiRange`, `ChiValue`). Either fix the imports and integrate it into the workflow, or remove it as dead code.

3. **Add runtime mode selection** — expose `FitXdist`/`FitImage`/`FitYdist`/`FitIdist` as PVs or derive from configuration, so operators don't need to edit code and restart the IOC.

4. **Verify fit function compatibility** — confirm that the `fittings` module's "Gauss+const" or "Gauss+slope" methods produce equivalent results to the Spyder script's `Gauss_ConstantBkg` for fitting sigma-vs-chi curves.

4. **Fix `Diag:XY` sizing** — ensure the matplotlib figure is sized to match the 640×480 PV allocation, or make the PV size dynamic.

5. **Fix `Diag:XY` sizing** — ensure the matplotlib figure is sized to match the 640×480 PV allocation, or make the PV size dynamic.

5. **Test with real beam data** — run a chi scan from the alignment OPI and compare results to a simultaneous Spyder execution. Verify:

6. **Test with real beam data** — run a chi scan from the alignment OPI and compare results to a simultaneous Spyder execution. Verify:

   - Same sigma values at each chi position

   - Same fitted chi optimum

   - Correct error bars

---

## 12. PV Reference

## 13. PV Reference

### Motor PVs

---

## 13. Glossary

## 14. Glossary

| Term | Definition |

|------|-----------|