@@ -82,4 +82,107 @@ Ok! I have downloaded the files you requested:
I could not obtain 'Wireshark capture of 10.111.8.46:502 during a DANGLE move' because during this time, the wireshark capture system was not running. Regarding the 'bl4a-DANGLE-persistent-scan-fail.png', I hope that 'bl4a-DANGLE-operation-fault-2026-04-08_1335.csv' will suffice as a data export of the values. The bl4a-Galil1.sav* files are all available now at '/home/controls/var/bl4a-Galil1/*' with the two most likely ones being 'bl4a-Galil1.sav_260326-125116' (good?) and 'bl4a-Galil1.sav_260409-120044' (bad?). The EPICS archiver data for 'BL4A:Mot:mDANGLE:Enc, .DRBV, .MISS, BL4A:Mot:AirPadStatus covering 2026-04-08 13:35–13:37' should be in 'bl4a-DANGLE-operation-fault-2026-04-08_1335.csv'. The 'Scan server log from bl4a-dassrv1 for the 04-08 window' should be in '/home/controls/var/scan/console.log'. Thank you for asking! Good luck in your analysis.
```
● The analysis document has been updated with the new verified findings. Here is a summary of what the new data revealed:
Major Revisions from the First Draft
1. The "1.054° encoder jump" hypothesis was wrong
The archiver CSV shows DANGLE.RBV moving smoothly from 8.2148 → 7.9463 → 8.0401 — no sub-second 1° excursion. The Profibus encoder is not glitching. The PRD=-6321 retry is instead triggered by some combination of the URIP=Yes readback path and what
I believe is the backlash post-process (MIP_MOVE_BL) state transition resetting RCNT, which is why 7 retries appear instead of the RTRY=5 budget.
2. The stepper is achieving only 17–26 % of commanded steps
From the archiver CSV vs the Galil command log on 04-08: PRD=-6321 (−1.054°) produced only −0.269° of actual motion. PRD=+3358 (+0.559°) produced only +0.094°. This is a physical stall, not a software issue. Most likely causes are the air-pad
cushion, a mechanical coupling, or a suspicious VELO/SPD factor-of-2 mismatch (see #5).
3. The scan failure on 2026-04-08 is a baseline-rate event, not a regression
Across the full console log, scans at ±0.1° DANGLE tolerance have an 8.4 % failure rate (39/463). The 04-08 "Data 3/" scan had 35 DANGLE moves and 2 failed (5.7 %). The 04-07 day only had 1 move so nothing failed and it looked fine.
4. The "very often" failures after 04-09 are from new scan definitions
On 2026-04-09 the beamline staff loaded new experiment templates (Si_PEG2k_DOPC_*, Pump D2O/Pump H2O) with per-scan DANGLE tolerances of ±0.005° and ±0.01°. Those tolerances are below the motor's residual error floor of ~0.005–0.02°, so they fail
constantly (58.8 % at ±0.005°, 27.6 % at ±0.01°). The 0.05° tolerance has 0 failures in 51 attempts and is the right default.
5. The autosave diff yielded exactly one DANGLE-relevant runtime change
mDANGLE.RDBD 0.001 → 0.01 between 2026-03-26 and 2026-04-09. This is the only difference touching mDANGLE, and it is in the correct direction — at 0.001° the motor could never have settled. Everything else (VELO, RTRY, BDST, URIP, FOFF, etc.) was
unchanged.
There is a latent VELO/SPD factor-of-2 inconsistency worth chasing separately: the autosave stores VELO=1.452 deg/s but the Galil log shows SPD=18310 steps/sec (= 3.046 deg/s), and neither matches the substitutions default of 6.0906. Please caget
BL4A:Mot:mDANGLE.VELO during active operations to confirm whether the autosave is stale or the driver is genuinely running at the wrong speed.
6. The 2026-04-09 15:29 devices.py tolerance commit is a no-op
The scan server was last restarted 2026-03-24 07:26:52 and has been running continuously since, so it is still using the devices.py values it loaded then. The commit changing the default from 0.1 → 0.05 will only take effect at the next scan-server
restart. The failing 04-09 scans override the default anyway, so the commit would not have helped them.
Key recommendations (revised)
- Raise the per-scan DANGLE tolerance on the new Si_PEG2k_DOPC_* / Pump * scans to ±0.05° — this has a proven 0 % failure rate.
- Keep mDANGLE.RDBD = 0.01 — do not re-tighten.
- Resolve the VELO/SPD factor-of-2 mystery via live caget.
- Diagnose the 80 % step-loss via archiver comparison of _DMotPos vs _DEncPos vs DRBV during a deliberate DANGLE move. This is the actual physical problem — fix it and the tight-tolerance scans will work.
- Long-term: make profibus.template URIP override opt-in so heavy compliant axes like mDANGLE can be excluded from unconditional URIP behavior.
```
and revised the document with commit da8db64f7bb6f270575a2c7ec9919c093781393c
### Prompt 1.2 - 1.3
Very good! Here is the output from the caget command you requested:
the (third) revision with commit a0cc207c956569a2ffa869b03ba98114536ab296
```
What started as "why did a 0.17° motor error happen on 2026-04-08" turned into a surprisingly deep archaeology dig. Let me mark the checkpoints:
First draft (before beamline data) — wrong. I convinced myself the "-6330 step retries" were a phantom Profibus encoder glitch of 1.054°. The math checked out using the substitutions-file MRES, the magnitude was eerily consistent, and the URIP=Yes
override in profibus.template was a genuinely attractive architectural smell. But it was wrong because the MRES I was dividing by wasn't the MRES the IOC was actually using.
Second draft (after autosave + CSV + scan log) — partially right. The archiver CSV killed the encoder-glitch hypothesis (no 1° excursion in the raw trace) and pointed at a real physical motion problem. The scan log revealed the "very often"
failures after 04-09 were from a completely separate cause (new experiment templates with tight tolerances), and that the baseline 04-08 failure was just one sample from a long-standing 8.4% failure rate at ±0.1°. But I still had a spurious
"VELO/SPD factor-of-2 mystery" because I hadn't yet checked MRES.
Third draft (after caget + pass0 autosave) — I believe correct. The caget was the tipoff: VELO=1.45202 matched the autosave exactly, so it wasn't a stale-autosave bug. Searching bl4a-Galil1_pass0.sav* for mDANGLE.MRES yielded 7.93e-05 (not the
substitutions-file 1.663e-04), and the factor of 2.097× between them made every previously-inconsistent number snap into place:
The "jump" was never mysterious — it was the EPICS motor record doing exactly what it was designed to do, pre-positioning by BDST when preferred_dir flipped. What was mysterious was that BDST (in engineering units) had quietly become 2× larger in
motor-step units when someone recalibrated MRES on 2026-02-19/20 without touching BDST. The first backlash cycle at the new step count fired 4 days later on 2026-02-24, and has been firing ~150+ times since.
What I wish I had asked for earlier:
- The live caget output — would have resolved the VELO question in one call
- The pass0 autosave files — would have revealed the MRES-substitutions mismatch immediately
- The scan server console.log — the Timeout while waiting for DANGLE.RBV (8.0401) = 8.213476 message cut through the misleading display artifact in the Scan GUI screenshot in one line
What the data trail taught me:
- "All motor retries converge monotonically, then suddenly there's a huge reverse retry" is a backlash pre-positioning signature, not an encoder glitch. The tell is that the magnitude is exactly BDST/MRES and it pairs with a +BDST/MRES move shortly
after.
- When two motor-record fields have tight numerical coupling (BDST, MRES, SREV, UREV), changing one without the others is a latent trap that only fires when the retry phase happens to dance across the sign boundary.
- The substitutions file and the runtime are not the same thing on a long-running IOC, and the substitutions file is what dominates assumptions (because the running IOC is opaque). Always, always check caget or _pass0.sav before trusting the
substitutions defaults.
Three caputs fix the acute problem (BDST=0, RDBD=0.05, RTRY=10); one substitutions-file update protects against the next IOC restart; one per-scan-tolerance fix protects the new experiments. The deep mechanical question (why the stepper is
achieving only 30–50% of commanded steps) is still open and will need hands-on investigation, but that is a secondary reliability problem — not the primary failure mode.
Thanks for the patience with the back-and-forth and for pulling the files from the beamline. That second round of data was what made the diagnosis possible; without it I'd have shipped the "Profibus encoder glitch" story with high confidence and
been completely wrong.
```
### Epilogue
Let us update our knowledge-transfer system to remember these new understandings. Be sure to apply the correct instructions at the various layers that we operate in (machine, cross-project and project. Be sure to update any cross-project understandings on the 'main' branch and propagate it to the machine-specific branch we are running this session in.
