gather BL4A/BL4B motion investigations into a common module

# bl4b-tthd-investigation instructions

# instrument-motion-investigations

This project contains correspondence, screenshots and log files developed during the course

of an investigation into motion control problems of an instrument with chronic concerns.

of several investigations into motion control problems of two instruments with chronic concerns.

You may read any and all files that are in this folder and which are provided in the /home/controls/**.

There are also files that may be referenced from the read-only experiment filesystem mounts at /SNS/REF_L/**.

There are also files that may be referenced from the read-only experiment filesystem mounts at /SNS/${INSTRUMENT}/**.

You may use the internet freely to research specifications and documentation.

## Capabilites and Role

You are an EPICS motion control specialist who is expert at EPICS database record syntax and have a deep understanding of the Galil multi-axis controller.

You are able to direct agent teams who are expert system programmers and software developers who have a deep understanding of the C/C++ runtime model and how to diagnose and fix memory, concurrency and file system errors.

## Problem specific files

## General configuration, log and code references

There are files that have been collected during the investigation of the fault. These are noted in the following files and folders:

Your prompts will indicate any specific files you should use. Generally, these will come into these paths:

* /SNS/users/6ov/BL4B/2026/04/12/

* /SNS/users//${USER}/${BL}/YYYY/MM/DD/: folders in user area containing artifacts obtained at the instrument or otherwise

* /home/controls/var: symbolic link to ../../ssd2/servers/${BEAMLINE}-dassrv1.sns.gov/home/controls/var

* /home/controls/common/*: folders containing support modules for the codes that can be inspected

* /home/controls/${BEAMLINE}/: folders that contain the instrument control system repository. You may interrogate the git log for context as needed.

## Direct archiver query (use this instead of asking for CSV exports)

Full docs: `setup/docs/sns-archiver-query.md` on `main`.

### Authoritative file locations on bl4b-dassrv1

### Authoritative file locations on ${BEAMLINE}-dassrv1

| Purpose | Path |

|---|---|

| Motor record autosave (`Galil1`) — VELO, BVEL, BDST, RDBD, RTRY… | `/home/controls/var/bl4b-Galil1/bl4b-Galil1.sav` + `.sav0/.sav1/.sav2` (rotating) |

| Motor record pass0 autosave (`Galil1`) — MRES, ERES, DVAL, OFF | `/home/controls/var/bl4b-Galil1/bl4b-Galil1_pass0.sav*` |

| Motor record dated snapshots (`Galil1`) — on IOC restart | `/home/controls/var/bl4b-Galil1/bl4b-Galil1.sav_YYMMDD-hhmmss` |

| Motor record autosave (`Galil2`) | `/home/controls/var/bl4b-Galil2/bl4b-Galil2.sav` + rotating |

| Motor record pass0 autosave (`Galil2`) | `/home/controls/var/bl4b-Galil2/bl4b-Galil2_pass0.sav*` |

| Motor record autosave (`Galil1`) — VELO, BVEL, BDST, RDBD, RTRY… | `/home/controls/var/${BEAMLINE}-Galil1/${BEAMLINE}-Galil1.sav` + `.sav0/.sav1/.sav2` (rotating) |

| Motor record pass0 autosave (`Galil1`) — MRES, ERES, DVAL, OFF | `/home/controls/var/${BEAMLINE}-Galil1/${BEAMLINE}-Galil1_pass0.sav*` |

| Motor record dated snapshots (`Galil1`) — on IOC restart | `/home/controls/var/${BEAMLINE}-Galil1/${BEAMLINE}-Galil1.sav_YYMMDD-hhmmss` |

| Motor record autosave (`Galil2`) | `/home/controls/var/${BEAMLINE}-Galil2/${BEAMLINE}-Galil2.sav` + rotating |

| Motor record pass0 autosave (`Galil2`) | `/home/controls/var/${BEAMLINE}-Galil2/${BEAMLINE}-Galil2_pass0.sav*` |

| Galil1 command log — every PR/BG/MG/SH on DMC1-DMC4 (env `GALIL_DEBUG_FILE` in `st.cmd`) | `/home/controls/var/log/dassrv1/galil_dmc.log` |

| Galil2 command log — every PR/BG/MG/SH on DMC5-DMC6 (separate file from Galil1) | `/home/controls/var/log/bl4b-Galil2.log` |

| Galil2 command log — every PR/BG/MG/SH on DMC5-DMC6 (separate file from Galil1) | `/home/controls/var/log/${BEAMLINE}-Galil2.log` |

| Galil1 IOC stdout (Encoder-stall, move-begin-failure, CA Link Exceptions) | `/home/controls/var/log/ioc_bl4b-Mot-Galil1.log` |

| Galil2 IOC stdout | `/home/controls/var/log/ioc_bl4b-Galil2.log` |

| Scan server stdout (real RBV values in `TimeoutException`) | `/home/controls/var/scan/console.log` |

| Scan device definitions (tolerances, timeouts) | `/home/controls/bl4b/python/scantools/devices.py` |

| Motor substitutions — `Galil1` | `/home/controls/bl4b/applications/bl4b-Galil1/bl4b-Galil1App/Db/bl4b-Galil1.substitutions` |

| Motor substitutions — `Galil2` | `/home/controls/bl4b/applications/bl4b-Galil2/bl4b-Galil2App/Db/bl4b-Galil2.substitutions` |

| IOC startup — `Galil1` | `/home/controls/bl4b/applications/bl4b-Galil1/iocBoot/iocbl4b-Galil1/st.cmd` |

| IOC startup — `Galil2` | `/home/controls/bl4b/applications/bl4b-Galil2/iocBoot/iocbl4b-Galil2/st.cmd` |

| Scan device definitions (tolerances, timeouts) | `/home/controls/${BEAMLINE}/python/scantools/devices.py` |

| Motor substitutions — `Galil1` | `/home/controls/${BEAMLINE}/applications/${BEAMLINE}-Galil1/${BEAMLINE}-Galil1App/Db/${BEAMLINE}-Galil1.substitutions` |

| Motor substitutions — `Galil2` | `/home/controls/${BEAMLINE}/applications/${BEAMLINE}-Galil2/${BEAMLINE}-Galil2App/Db/${BEAMLINE}-Galil2.substitutions` |

| IOC startup — `Galil1` | `/home/controls/${BEAMLINE}/applications/${BEAMLINE}-Galil1/iocBoot/iocbl4b-Galil1/st.cmd` |

| IOC startup — `Galil2` | `/home/controls/${BEAMLINE}/applications/${BEAMLINE}-Galil2/iocBoot/iocbl4b-Galil2/st.cmd` |

Note: the old path `/home/controls/var/log/dassrv1/ioc_bl4b-Galil1.log` is a

pre-RHEL9-upgrade artefact (last-written 2019). The live IOC stdout is at

`/home/controls/var/log/ioc_bl4b-Mot-Galil1.log`, written to by the current

`procServ` wrapper.

Note: some beamlines will use '/home/controls/var/log/dassrv1/*' while others will use '/home/controls/var/log/*'. Prefer the former, but tolerate the latter.

### Galil controller-to-axis mapping (two IOCs, overlapping numbering)

BL4B uses **two** Galil IOCs. Each creates different controllers on the same

beamline network (10.112.9.x). DMC5 (10.112.9.49) is *commented out* in

`bl4b-Galil1`'s st.cmd but is *active* in `bl4b-Galil2` — so `BL4B:Mot:zd`

`${BEAMLINE}-Galil1`'s st.cmd but is *active* in `${BEAMLINE}-Galil2` — so `BL4B:Mot:zd`

and siblings are live motors served by Galil2, not ghosts. Always consult

both st.cmd files when mapping a PV to its controller.

BL4A controller analysis is contained in branches ' bl4a-S3-investigation' and 'bl4a-DANGLE-investigation' that will be merged into 'instrument-motion-investigations'

#### `bl4b-Galil1` IOC

| Controller | IP | Notable axes |

#### Cross-IOC quirks to watch for

* **`BL4B:Mot:zd.RBV`** is read by `bl4b-Galil1`'s `virtual_pdLift.template`

  (for the `tthd` forward kinematics) from `bl4b-Galil2` over CA. If you see

* **`BL4B:Mot:zd.RBV`** is read by `${BEAMLINE}-Galil1`'s `virtual_pdLift.template`

  (for the `tthd` forward kinematics) from `${BEAMLINE}-Galil2` over CA. If you see

  a suspicious `zd` value in a tthd calc, `dbpr BL4B:Mot:tthd:REQ_pd,4` on

  the running Galil1 IOC is the quickest way to see the actual input value;

  inspecting the `Galil1` substitutions alone is misleading because they

### Test data for development

Files in `/SNS/REF_M/` and `/SNS/users/6ov/` are accessed via sshfs mounts. See the

Files in `/SNS/${INSTRUMENT}/` and `/SNS/users/${USER}/` are accessed via sshfs mounts. See the

parent project's `CLAUDE.md` for network mount handling rules.

**Do not revert the `read_only` parameter** — the production mount is `-o ro` and tests

The SNS sshfs mounts **must** include `-o intr` for SIGALRM to work:

```bash

sshfs ${USER}@analysis.sns.gov:/SNS/REF_M/ ~/SNS/REF_M \

sshfs ${USER}@analysis.sns.gov:/SNS/${INSTRUMENT}/ ~/SNS/${INSTRUMENT} \

  -o ro,intr,reconnect,ServerAliveInterval=15,ServerAliveCountMax=3

```

# bl4b-hs-investigation

Using the [Claude](https://code.ornl.gov/6ov/claude) agentic-engineering knowledge transfer workflow, I gathered information on the failures we are experiencing at BL4A and BL4B following the Galil motion control upgrade. As is my custom, I collect files and screenshots in my home folder on the DAQ/Analysis unified autohome folder mount, at the systematic path `${HOME}/${BL}/YYYY/MM/DD/*`. The agent runs on a machine that has the `/SNS/${INSTRUMENT}/` and `/SNS/users/${USER}/` mounted via an sshfs filesystem mount ( *which is setup easily by the workflow via `setup/mount-sshfs.sh`* ). The [architecture](https://code.ornl.gov/6ov/claude/-/blob/main/setup/docs/architecture.md?ref_type=heads) of this workflow and how knowledge transfers is documented.

```

6ov@uvdl3:/media/ssd2/Projects/Claude/1$ setup/mount-sshfs.sh --backend sshfs users/6ov

Backend: sshfs

--- SNS/users/6ov ---

  Mounting /SNS/users/6ov -> /home/6ov/SNS/users/6ov (sshfs)

  Updating symlink: /SNS/users/6ov -> /home/6ov/SNS/users/6ov

[sudo] password for 6ov: 

=== Mount status ===

sftp-analysis-sns-gov:/SNS/REF_L     fuse.rclone  5.0P  2.9P  2.2P  57% /home/6ov/SNS/REF_L

sftp-analysis-sns-gov:/SNS/REF_M     fuse.rclone  5.0P  2.9P  2.2P  57% /home/6ov/SNS/REF_M

6ov@analysis.sns.gov:/SNS/users/6ov/ fuse.sshfs   256G  139G  118G  55% /home/6ov/SNS/users/6ov

```

Under this environment, I created a branch in [tasking](https://code.ornl.gov/6ov/tasking/-/tree/instrument-motion-investigations) to collect the results of several investigations, based on previous work and an understanding of how to prompt the AI assistant to produce high quality results. From that step, I engaged in a the dialog that is captured below:

## Prompt 1

You are working on the tasking project, instrument-motion-investigations branch. The code uses the Experimental Physics Industrial Control System (EPICS) as its basis. You had previously performed investigations of problems with the S3 and DANGLE motion at *another* beamline, and I reference your work in tasking/S3-Gap-Failure-Analysis.md, tasking/DANGLE-Motion-Failure-Analysis.md and tasking/tthd-Motion-Failure-Analysis.md which you may profit from to orient yourself. A previous session developed a plan at tasking/plan/bl4b-hs-invstigation-plan.md please read it. The task is to analyze the reason for motion control problems that were observed on the 'hs' motor on 2026-04-10 23:35. There were unexpected (possibly spurious) high limit triggers in the motion. The process variable is BL4B:Mot:hs, which is a key axis in sample positioning used often that you will need to understand fully. You are not executing on the same computer that the system ran on. I have made as much of the environment available to you on this machine, please let me know if there is document that you require. The code that configures the motion is one of two modules; either /home/controls/bl4b/applications/bl4b-Galil1/, or /home/controls/bl4b/applications/bl4b-Galil2/. The code for the component software modules is in /home/controls/common/*. The log files are in /home/controls/var/log/*. Your task is to do a deep and thorough investigation of the reported files, the documents, the code, the configuration files and try to determine the root cause of the problem. Please be diligent and resourceful. If you need a tool to analyze the data that you do not have, use venv, uv and/or pixi to install it ask me to help you.