@@ -22,3 +22,27 @@ Under this environment, I created a branch in [tasking](https://code.ornl.gov/6o
You are working on the tasking project, instrument-motion-investigations branch. The code uses the Experimental Physics Industrial Control System (EPICS) as its basis. You had previously performed investigations of problems with the S3 and DANGLE motion at *another* beamline, and I reference your work in tasking/S3-Gap-Failure-Analysis.md, tasking/DANGLE-Motion-Failure-Analysis.md and tasking/tthd-Motion-Failure-Analysis.md which you may profit from to orient yourself. A previous session developed a plan at tasking/plan/bl4b-hs-invstigation-plan.md please read it. The task is to analyze the reason for motion control problems that were observed on the 'hs' motor on 2026-04-10 23:35. There were unexpected (possibly spurious) high limit triggers in the motion. The process variable is BL4B:Mot:hs, which is a key axis in sample positioning used often that you will need to understand fully. You are not executing on the same computer that the system ran on. I have made as much of the environment available to you on this machine, please let me know if there is document that you require. The code that configures the motion is one of two modules; either /home/controls/bl4b/applications/bl4b-Galil1/, or /home/controls/bl4b/applications/bl4b-Galil2/. The code for the component software modules is in /home/controls/common/*. The log files are in /home/controls/var/log/*. Your task is to do a deep and thorough investigation of the reported files, the documents, the code, the configuration files and try to determine the root cause of the problem. Please be diligent and resourceful. If you need a tool to analyze the data that you do not have, use venv, uv and/or pixi to install it ask me to help you.
```
Bottom-line findings
Andre's hypothesis is confirmed in every particular, and more: the hs fault is a chronic, intermittent electrical fault on the DMC3 axis-E limit-switch signal path — not a software bug, not a soft-limit issue, not a physical limit contact.
- HLS asserted at 2026-04-10 23:35:36.298 EDT while hs.RBV = 63.132 mm (187 mm below DHLM = 250 mm). Motor was stationary; no setpoint had been written in ~46 minutes. HLS cleared at 00:10:21.929 EDT, 34 m 45 s later, spontaneously with no IOC
command, no operator action, no IOC restart.
- The 599 ms flip-flop at 23:35:36 (HLS 0→1→0→1 inside 800 ms) is an unmistakable intermittent-contact fingerprint.
- 13 positive BGE attempts all refused with ?; 2 negative BGE attempts both succeeded — textbook Galil hardware-limit protection behavior.
- This is at least the third spurious limit event on axis E in 8 weeks: 2026-02-19 PLUS_LS glitch (300 ms), 2026-02-20 MINUS_LS (6 bursts, 400 ms–84 s), 2026-04-10 PLUS_LS (34:45). Both forward and reverse inputs have glitched — this rules out a
single bad microswitch and points to shared wiring/connector or Galil input-stage issue.
- Other DMC3 axes (sib/sit/sir/sil) are clean outside of IOC-restart transients — fault is localised to axis E.
Recommended fix (high priority → low)
1. Physical inspection of DMC3 axis-E limit-switch cabling and connector (wiggle-test while watching HLS/LLS).
2. Swap axis-E wiring to a spare axis (F/G/H on DMC3 are free) if #1 doesn't find the fault — will disambiguate cable vs Galil input stage.
3. IOC-side plausibility alarm for "HLS asserted while motor >X mm from DHLM" — would have caught the Feb 19 event at first occurrence.
4. Galil digital-input filter (CI command) as a lower-priority defensive measure for short glitches (can't filter 34-min events).
The sister tthd runaway that night is a separate software fault (see tthd-Motion-Failure-Analysis.md) — correlated only because operator frustration with stuck hs led them to retry Move Align.
```
Claude developed [hs-HLS-Failure-Analysis.md](hs-HLS-Failure-Analysis.md) and [hs-HLS-Failure-Analysis.pdf](hs-HLS-Failure-Analysis.pdf).
