analysis: Phoebus RDB archive-reader source review

# CSS / Phoebus RDB Archive Reader — Source Analysis

**Date:** 2026-04-11

**Branch:** `tasking/css-archiver-query-tool-development`

**Author:** investigation by assistant, sources read end-to-end

**Source tree:** `/home/controls/src/phoebus/app/trends/archive-reader/`

**Reference deployment:** `~/opt/css/product-sns-4.7.4-SNAPSHOT/` (office CSS)

This document captures what we learned by reading the Phoebus archive-reader

source code in preparation for building a Python CLI that talks directly to the

SNS PV archive Oracle databases. It exists to make the refinements to

`plan/archiver-query-tool.md` defensible: every claim in the refined plan can

be cross-referenced to a specific file and line in the Phoebus source.

The reading list:

```

app/trends/archive-reader/src/main/java/org/phoebus/archive/reader/rdb/

├── RDBArchiveReader.java          (446 lines)  — entry point, channel lookup, status/severity caches

├── RDBArchiveReaderFactory.java   (31 lines)   — SPI for "jdbc:" URLs (just a thin dispatch)

├── RDBPreferences.java            (32 lines)   — preference declarations

├── SQL.java                       (124 lines)  — *the* schema definition (templated SQL strings)

├── AbstractRDBValueIterator.java  (379 lines)  — sample-row decoder (the coalescing logic lives here)

├── RawSampleIterator.java         (215 lines)  — raw-sample fetch with initial-sample carry-forward

└── StoredProcedureValueIterator.java (284 lines) — OPTIMIZED min/max/avg path

```

Plus the runtime preferences default file:

```

app/trends/archive-reader/src/main/resources/archive_reader_rdb_preferences.properties

```

And the SNS office Phoebus configuration that overrides those defaults:

```

~/opt/css/product-sns-4.7.4-SNAPSHOT/settings.ini   (mode 644 — credentials are intentionally shared)

```

---

## 1. Authoritative schema for SNS (`chan_arch.*`)

The schema strings are constructed in `SQL.java` from a `prefix` parameter. SNS

sets `org.phoebus.archive.reader.rdb/prefix=chan_arch.` in `settings.ini`, so

every reference below is to the `chan_arch` schema in Oracle. The dialect

detection routes to the **Oracle** branch of the SQL builder.

### `chan_arch.channel`

- `channel_id` PK, `name` UNIQUE.

- Lookup by exact name: `SELECT channel_id FROM chan_arch.channel WHERE name=?`

  (`SQL.java:73`).

- Lookup by glob pattern (case-insensitive Oracle dialect):

  `SELECT name FROM chan_arch.channel WHERE LOWER(name) LIKE LOWER(?) ESCAPE '\' ORDER BY name`

  (`SQL.java:66`).

### `chan_arch.sample` — the time-series table

Columns referenced by SQL.java's Oracle branch (`SQL.java:85-95`):

```sql

SELECT smpl_time, severity_id, status_id, num_val, float_val, str_val

  FROM chan_arch.sample

 WHERE channel_id = ?

   AND smpl_time BETWEEN ? AND ?

 ORDER BY smpl_time

```

When `use_array_blob=true` (the SNS default — see prefs file

`archive_reader_rdb_preferences.properties:39`), two more columns are added:

```sql

SELECT smpl_time, severity_id, status_id, num_val, float_val, str_val,

       datatype, array_val

```

**Critical Oracle/Postgres difference for sub-second precision**

(`AbstractRDBValueIterator.java:181-185`):

```java

final java.sql.Timestamp stamp = result.getTimestamp(1);

// Oracle has nanoseconds in TIMESTAMP, other RDBs in separate column

if (reader.getPool().getDialect() != Dialect.Oracle)

    stamp.setNanos(result.getInt(7));

```

→ **For SNS Oracle, the `smpl_time` TIMESTAMP carries fractional seconds inline.

There is no separate `nanosecs` column.** The original plan assumed there was

one (it appears in MySQL/Postgres dialects). This is the single most important

schema correction.

### `chan_arch.severity` and `chan_arch.status`

Two small lookup tables, slurped into in-memory hashmaps at reader

construction time (`RDBArchiveReader.java:113-180`).

```sql

SELECT severity_id, name FROM chan_arch.severity

SELECT status_id,   name FROM chan_arch.status

```

The status text is then used as a *behavioural* flag for the no-value markers

documented in §4.

### `chan_arch.num_metadata` — display info per channel

```sql

SELECT low_disp_rng, high_disp_rng,

       low_warn_lmt, high_warn_lmt,

       low_alarm_lmt, high_alarm_lmt,

       prec, unit

  FROM chan_arch.num_metadata

 WHERE channel_id=?

```

Used to render units and apply numeric precision (`SQL.java:55-58`,

`AbstractRDBValueIterator.java:106-132`). Useful for the assistant to know "what

unit is this PV in" without having to introspect the running IOC.

### `chan_arch.enum_metadata` — enum string labels

```sql

SELECT enum_nbr, enum_val

  FROM chan_arch.enum_metadata

 WHERE channel_id=?

 ORDER BY enum_nbr

```

(`SQL.java:60-61`). When this returns rows, the channel is treated as an enum:

the integer or float value coming back from `chan_arch.sample` is *cast to int*

and used as an index into the label list (`AbstractRDBValueIterator.java:200`,

`221`). For example, `BL4A:Mot:AirPadStatus` would have rows

`(0, "Off"), (1, "On")` and a sample with `num_val=1` becomes `"On"`.

### `chan_arch.array_val` — legacy waveform table

Only consulted if `use_array_blob=false` (`SQL.java:96-97`,

`AbstractRDBValueIterator.java:267-276`). At SNS the new BLOB format is used —

waveform data lives in the `sample.array_val` BLOB column, decoded inline by

`AbstractRDBValueIterator.readBlobArrayElements()` (`AbstractRDBValueIterator.java:313-343`).

The BLOB format is extremely simple: 4-byte big-endian element count followed

by `count` consecutive 8-byte big-endian doubles. Datatype `'d'` is the only

type currently decoded; integer (`'i'`) and long (`'l'`) BLOBs throw

`"Sample BLOBs of type 'X' are not decoded"`. Out of scope for v1 of our tool.

### What the schema does **NOT** have

- **No `archive_id` column on `sample`.** The original plan guessed there was a

  multi-archive engine join key. Wrong: each Oracle instance is a single

  archive. Selecting between Accelerator and Instruments is done by connecting

  to the right URL.

- **No `seq_nbr` column on the scalar `sample` table.** That column exists only

  in the `array_val` table for waveform-element ordering.

- **No retention or sampling-mode columns referenced** by the reader. They may

  exist in the database (the archive engine writes them), but the reader does

  not query them.

---

## 2. The two SNS archive endpoints

From `~/opt/css/product-sns-4.7.4-SNAPSHOT/settings.ini` (lines 78-79):

```ini

org.csstudio.trends.databrowser3/urls=

    jdbc:oracle:thin:@(DESCRIPTION=(LOAD_BALANCE=OFF)(FAILOVER=ON)

        (ADDRESS=(PROTOCOL=TCP)(HOST=snsappa.sns.ornl.gov)(PORT=1610))

        (ADDRESS=(PROTOCOL=TCP)(HOST=snsappb.sns.ornl.gov)(PORT=1610))

        (CONNECT_DATA=(SERVICE_NAME=prod_controls)))|Accelerator*

    jdbc:oracle:thin:@snsoroda-scan.sns.gov:1521/scprod_controls|Instruments

```

Two physically distinct Oracle databases:

| Logical name | Service name      | Endpoint                                         | Notes                                  |

|--------------|-------------------|--------------------------------------------------|----------------------------------------|

| Accelerator  | `prod_controls`   | `snsappa.sns.ornl.gov:1610` + `snsappb:1610`     | HA pair, FAILOVER on, LOAD_BALANCE off |

| Instruments  | `scprod_controls` | `snsoroda-scan.sns.gov:1521`                     | DNS round-robin to .65 / .63           |

For BL4A motion / DASlogs work, **Instruments is the relevant endpoint.**

Accelerator carries beam-current PVs, vacuum, RF, etc. — relevant only when an

investigation needs to correlate instrument behaviour against beam state.

**Network reach from this machine** (`uvdl3`, dragonfly clone) verified

2026-04-11:

```

$ timeout 5 bash -c '</dev/tcp/snsoroda-scan.sns.gov/1521' && echo OK

OK

$ timeout 5 bash -c '</dev/tcp/snsappa.sns.ornl.gov/1610' && echo OK

OK

$ timeout 5 bash -c '</dev/tcp/snsappb.sns.ornl.gov/1610' && echo OK

OK

```

Direct local connection works → no SSH proxy needed → tool can run in-process

on the dragonfly machine without an extra network hop. Other machines

(particularly the laptop branches) need to be re-checked at deploy time and

fall back to ssh-tunnel mode if blocked.

---

## 3. Authentication is a shared read-only account, baked into settings.ini

Lines 71-76 of `~/opt/css/product-sns-4.7.4-SNAPSHOT/settings.ini`:

```ini

# Archived Data

org.phoebus.archive.reader.rdb/user=sns_reports

# org.phoebus.archive.reader.rdb/user=css_arch_user

org.phoebus.archive.reader.rdb/password=sns

org.phoebus.archive.reader.rdb/prefix=chan_arch.

org.phoebus.archive.reader.rdb/stored_procedure=chan_arch.archive_reader_pkg.get_browser_data

org.phoebus.archive.reader.rdb/starttime_function=SELECT chan_arch.archive_reader_pkg.get_actual_start_time (?, ?, ?)  FROM DUAL

```

Implications for the tool design:

1. **The credentials are not actually a secret.** The settings file is mode 644

   and is distributed to every CSS install at every workstation site-wide.

   `sns_reports` is a shared read-only Oracle account that ORNL Controls

   intends every operator's CSS instance to use. The original plan's "ask the

   user for credentials" step is unnecessary — we already have them, in the

   same place CSS reads them from.

2. We should still **store our copy in a mode-600 file** as defense-in-depth

   (e.g. an Oracle ACL change in the future could revoke world-read on the

   shared password without us noticing if we baked it into the script). But

   provenance should be documented: the credentials came from CSS settings.ini,

   not from a private operator handoff.

3. **Verifying auth before writing the tool requires actually connecting to

   the live Oracle.** That's a state-affecting action (audit-trail entry on

   snsoroda-scan) so should be greenlit by the operator first. After greenlight,

   a 10-line throwaway Python script can confirm:

   - `oracledb.connect(user="sns_reports", password="sns", dsn="snsoroda-scan.sns.gov:1521/scprod_controls")`

   - `SELECT name FROM chan_arch.channel WHERE LOWER(name) LIKE 'bl4a:mot:dangle%' ORDER BY name`

   - `SELECT smpl_time, num_val, float_val FROM chan_arch.sample WHERE channel_id = (...) AND smpl_time BETWEEN ... AND ...`

4. **`css_arch_user` is the commented-out alternative**, suggesting it was the

   previous account name. Worth knowing in case `sns_reports` ever stops working.

---

## 4. The coalescing rules — what `value` actually means

The single most error-prone area of the plan was "how do you coalesce

`num_val`, `float_val`, and `str_val` into a single `value` column". The

authoritative answer is in `AbstractRDBValueIterator.decodeSampleTableValue()`

(`AbstractRDBValueIterator.java:178-229`). The logic is:

1. **First, look at `float_val` (column 5).** If non-null:

   - If the channel has enum labels → return `VEnum.of((int) float_val, labels, ...)`

   - Else → return `VDouble.of(float_val, ...)`

2. **Then, look at `num_val` (column 4).** If non-null:

   - If the channel has enum labels → return `VEnum.of(num_val, labels, ...)`

   - Else → return `VDouble.of(num_val, ...)` (note: cast to double, label still "VDouble")

3. **Then, look at `str_val` (column 6).** Always returned as `VString` if reached.

So the precedence is **`float_val` > `num_val` > `str_val`**, irrespective of

whether the PV is "really" an enum or analog. The "is this an enum?" question

is answered separately by the existence of rows in `chan_arch.enum_metadata`

for that `channel_id`. Enum rendering happens *after* coalescing.

The original plan's intuition — "use float for analog, num for enum" — was

half-right. In practice:

- An analog PV (e.g. `BL4A:Mot:DANGLE.RBV`) writes to `float_val`, leaves

  `num_val` and `str_val` null.

- A digital/enum PV (e.g. `BL4A:Mot:DANGLE.DMOV` or `BL4A:Mot:AirPadStatus`)

  may write to *either* `num_val` *or* `float_val` depending on how the

  archive engine was configured for that channel. The coalescing precedence

  handles both cases. The enum_metadata table provides the textual rendering.

- A string PV (e.g. an alarm message or a `.DESC` field) writes to `str_val`.

**Practical consequence for the tool:** the default `--coalesce` behavior

should mirror CSS exactly: pick `float_val` first, fall back to `num_val`, fall

back to `str_val`. The optional `--raw` mode emits all three columns plus

`datatype` so the assistant can debug coalescing surprises.

---

## 5. Status-driven "no-value" markers

`AbstractRDBValueIterator.filterSeverity()` (`AbstractRDBValueIterator.java:236-248`):

```java

if (status.equalsIgnoreCase("Archive_Off") ||

    status.equalsIgnoreCase("Disconnected") ||

    status.equalsIgnoreCase("Write_Error"))

    return AlarmSeverity.UNDEFINED;

```

These three status strings are how the archive engine *records gaps* — they're

written into the `sample` table to mark "the engine lost the channel here" and

"the engine resumed here". The data values in those rows are typically zero or

NaN and should not be plotted.

**For the agent's use case this is critical**: a query that returns a sample

with status `Disconnected` means the IOC was offline at that time. If the

agent is investigating a motion failure and sees a `Disconnected` marker in

the middle of the failure window, that's evidence of a totally different

problem class (network glitch vs mechanical fault).

The tool **must surface status alongside the value** so the agent can detect

these markers, even if the default coalesced output looks like a normal sample.

---

## 6. Initial-sample carry-forward

`RawSampleIterator.determineInitialSample()` (`RawSampleIterator.java:76-142`):

```java

// Get time of initial sample

final PreparedStatement statement =

        connection.prepareStatement(reader.getSQL().sample_sel_initial_time);

...

statement.setInt(1, channel_id);

statement.setTimestamp(2, start_stamp);

if (statement.getParameterMetaData().getParameterCount() == 3)

    statement.setTimestamp(3, end_stamp);

final ResultSet result = statement.executeQuery();

if (result.next())

{

    final Timestamp actual_start = result.getTimestamp(1);

    if (actual_start != null)

    {

        start_stamp = actual_start;

        ...

    }

}

```

CSS resolves the *actual* start time before issuing the main `BETWEEN` query.

For Oracle, the configured initial-sample lookup is the stored procedure:

```sql

SELECT chan_arch.archive_reader_pkg.get_actual_start_time(?, ?, ?) FROM DUAL

```

This procedure (per the parameter-count check) takes 3 args: `channel_id`,

some second timestamp, and the requested start. We don't have the procedure

source, but the *fallback* SQL when `starttime_function` is empty

(`SQL.java:80-84`) tells us the semantics:

```sql

SELECT smpl_time FROM (

  SELECT smpl_time FROM chan_arch.sample

   WHERE channel_id = ? AND smpl_time <= ?

   ORDER BY smpl_time DESC

) WHERE ROWNUM = 1

```

Plain English: "give me the timestamp of the most recent sample at or before

the requested start". The main query then uses *that* as its start, so the

result set always begins with the carry-forward value that was active at the

left edge of the requested window.

**For the agent's use case this is also exactly right.** When the agent asks

"what was DANGLE doing between 13:35 and 13:36", it almost always wants to

know "and what was the value AT 13:35" — even if the most recent change was

at 13:32. Otherwise the agent has to do a separate "give me the last value

before 13:35" query, or wrongly conclude the PV had no value at the start

of the window.

The tool should **default to carry-forward mode** (matching CSS) and offer a

`--strict-window` flag for the rare case where the agent wants samples

*strictly inside* the time window.

We can implement this in two ways:

1. **Stored procedure path:** call

   `chan_arch.archive_reader_pkg.get_actual_start_time(channel_id, start, end)`

   from Python, then issue the main `SELECT`. Same as CSS.

2. **Pure SQL path:** issue the fallback SQL above, then issue the main

   `SELECT`. Doesn't depend on the stored procedure existing.

Path 2 is simpler and more portable. Path 1 is identical to what CSS does, so

results are guaranteed bit-for-bit comparable. Probably do path 2 as the

default and add a `--use-stored-procedures` flag for path 1 verification.

---

## 7. The OPTIMIZED stored procedure (out of scope for v1, documented for v2)

`StoredProcedureValueIterator.executeProcedure()`

(`StoredProcedureValueIterator.java:90-186`).

For Oracle, the JDBC call form is:

```java

String sql = "begin ? := chan_arch.archive_reader_pkg.get_browser_data(?, ?, ?, ?); end;";

final CallableStatement statement = connection.prepareCall(sql);

statement.registerOutParameter(1, OracleTypes.CURSOR);

statement.setInt(2, channel_id);

statement.setTimestamp(3, Timestamp.from(start));

statement.setTimestamp(4, Timestamp.from(end));

statement.setInt(5, count);    // desired number of buckets

statement.execute();

ResultSet result = (ResultSet) statement.getObject(1);

```

The returned cursor has 9 columns

(`StoredProcedureValueIterator.java:198-201`):

```

WB | SMPL_TIME | SEVERITY_ID | STATUS_ID | MIN_VAL | MAX_VAL | AVG_VAL | STR_VAL | CNT

```

Decoding rules (`StoredProcedureValueIterator.java:202-238`):

- **`WB == -1`** → string sample (column 8: `STR_VAL`).

- **`WB >= 0` and `CNT == 1`** → single sample bucket; return `AVG_VAL` as a

  scalar `VDouble`.

- **`WB >= 0` and `CNT > 1`** → averaged bucket; return `VStatistics` with

  `min=MIN_VAL`, `max=MAX_VAL`, `avg=AVG_VAL`, `count=CNT`.

For Python `oracledb`, calling a function that returns a `SYS_REFCURSOR` is

straightforward: register an output variable of type `oracledb.CURSOR`,

execute, then iterate the cursor. The 9-column result decodes the same way.

We won't implement this in v1 (raw is enough for minute-to-hour windows) but

when v2 is needed, this is a roughly 50-line addition.

---

## 8. PV-name variants and equivalent prefixes

`RDBArchiveReader.getChannelID()` (`RDBArchiveReader.java:332-368`):

```java

for (String variant : PVPool.getNameVariants(name, RDBPreferences.equivalent_pv_prefixes))

{

    statement.setString(1, variant);

    ...

}

```

The default `equivalent_pv_prefixes=ca, pva` means CSS will try:

1. The exact name as given.

2. Each listed prefix prepended (`ca://name`, `pva://name`).

3. If the input already has a prefix, the bare name as well.

This matters because the archive engine may have stored a channel as

`BL4A:Mot:DANGLE.RBV` while the CS-Studio data browser is configured to

request `ca://BL4A:Mot:DANGLE.RBV`, or vice versa. The lookup tries every

plausible spelling.

For the Python tool we should mimic this: try the exact name first, then

strip any `ca://` / `pva://` prefix, then prepend each prefix and try those,

to maximize the chance of finding the channel without the operator having to

guess what spelling the archive engine used.

---

## 9. Connection pooling and JDBC tuning

CSS uses an `RDBConnectionPool` (`RDBArchiveReader.java:49`,

`RDBArchiveReader.java:68`) which is a Phoebus utility wrapping a small

JDBC connection pool. For our Python tool, a single short-lived connection

per CLI invocation is fine — `oracledb` thin-mode connections set up in

≪ 500 ms and the assistant's invocation pattern is one-shot queries.

JDBC fetch size is set to **1000** (`archive_reader_rdb_preferences.properties:75`):

```properties

# JDBC Statement 'fetch size':

# For Oracle, the default is 10.

# Tests resulted in a speed increase up to fetch sizes of 1000.

# Bigger numbers can result in java.lang.OutOfMemoryError.

fetch_size=1000

```

The Python `oracledb` cursor equivalent is `cursor.arraysize = 1000`. We

should set this to match.

CSS uses a 120-second timeout for "metadata" queries (lookups of channel_id,

status, severity) but no timeout on data queries (`RDBPreferences.timeout_secs`

= 120, but the data path doesn't call `setQueryTimeout` based on the comment

in the prefs file). For our tool, a per-CLI-invocation `--timeout` flag with

a 60-second default is reasonable.

---

## 10. Timezone — the one open question that needs live confirmation

`AbstractRDBValueIterator.decodeSampleTableValue()`

(`AbstractRDBValueIterator.java:181`):

```java

final java.sql.Timestamp stamp = result.getTimestamp(1);

...

final Time time = TimeHelper.fromInstant(stamp.toInstant());

```

`java.sql.Timestamp.toInstant()` returns an `Instant` interpreted in the

*JVM's default timezone*. On a CSS office instance running on a Linux machine

configured for `America/New_York`, this means the JDBC driver fetches the

Oracle TIMESTAMP, attaches the JVM's local TZ to it, then converts to UTC

epoch milliseconds.

In symmetry, `RawSampleIterator.determineInitialSample()`

(`RawSampleIterator.java:78-79`):

```java

Timestamp start_stamp = Timestamp.from(start);

final Timestamp end_stamp = Timestamp.from(end);

```

These take a UTC `Instant` and produce a `Timestamp` that the JDBC driver

will *also* interpret in the JVM's local TZ when sending to Oracle.

So as long as **client-side and Oracle-side timezones agree**, the conversions

cancel out and everything works. The dragonfly clone is `America/New_York` per

the operator's environment. The Oracle server snsoroda-scan is presumably also

on Eastern time (the SNS site standard). Both Python `oracledb` and Java

`java.sql.Timestamp` will treat naive `TIMESTAMP` as local-server-time on read

and local-client-time on write, so this is a wash.

**Action item for the live test:** issue a query for a sample whose epoch-ms

timestamp we know exactly (from a CSV export the operator already did during

the DANGLE investigation), and confirm round-trip equivalence to within ±1 ms.

If it's off by 4 hours (3600 \* 4 \* 1000 = 14 400 000 ms), the JVM/Python TZ

assumption is wrong and the tool needs to do an explicit `at_timezone` cast in

the SQL.

---

## 11. Implications for the plan

Concrete edits the plan needs (will be applied in a follow-up commit to the

parent project's `plan/archiver-query-tool.md`):

| § in current plan | Edit |

|---|---|

| "What's in the database (expected schema, to be verified)" | Replace guessed schema with confirmed `chan_arch.*` schema from §1 above |

| "How CS-Studio picks which value field to use" | Rewrite per §4: precedence is float→num→str, enum rendering is via separate `enum_metadata` table |

| "Open questions: credentials" | Mark resolved — credentials are `sns_reports`/`sns` from `settings.ini`, document provenance |

| "Open questions: schema verification" | Mark resolved — see §1 |

| "Open questions: multi-archive selection" | Mark resolved — single archive per Oracle instance, no archive_id JOIN |

| "Open questions: network reach" | Mark resolved — verified 2026-04-11, all three endpoints reachable |

| "Step 1 — Network & auth feasibility check" | Reduce to a single 10-line oracledb thin-mode test script + the four PVs from the DANGLE CSV |

| "Step 2 — Minimum viable tool" | Add: implement the initial-sample carry-forward (per §6); add `--strict-window` flag for off-by-default behaviour |

| "Step 3 — Verification against the DANGLE CSV" | Tighten: now we know the timezone risk (§10) and the status-driven null-marker pattern (§5), the diff script must check both |

| "Non-obvious design decisions" | Add subsections on initial-sample carry-forward, status-marker handling, equivalent PV name lookup |

| Estimated work | Step 1 drops from "15 min + operator wait" to "10 min, just need greenlight" since we already have credentials |

Net effect: most of the plan stays the same but the *uncertainty budget*

collapses dramatically. We can go from "review the plan" → "run the live test"

→ "implement the MVP" in a single session instead of needing three back-and-

forths with the operator.

---

## 12. Next concrete steps

1. **Operator greenlight to do the live read-test** (the only state-affecting

   step — generates an Oracle audit-trail entry).

2. **Live test** — 10-line throwaway script: connect, list a few channel rows,

   read 60 seconds of `BL4A:Mot:DANGLE.RBV` from the DANGLE incident window,

   verify timezone round-trip against the operator CSV.

3. **Apply plan refinements** to `plan/archiver-query-tool.md` per §11.

4. **Implement MVP** per the refined plan.

5. **Verify against the DANGLE CSV** — exact same step the original plan

   describes, just better-prepared.

6. **Commit tool to `main`**, merge into machine branches, push.

7. **Capture cross-project knowledge** in the parent `CLAUDE.md` per the

   knowledge-routing rules.