sientia-dataops-opc-ingestor/e2e/scenarios.md

# E2E Scenario Documentation — OPC Gateway

Functional reference for the gateway end-to-end scenarios: slot leasing, subscription reads,
NATS writes, gRPC reads, active reads, certificates and the pod lifecycle. Tests live under
`e2e/`, are marked with `@pytest.mark.integration` and run with:

```bash
pytest e2e/ -o testpaths=e2e -m integration
```

**Every section below maps 1:1 to an actual test function** (or to one `pytest.param` id of a
parametrized test) — this file describes what the suite verifies today. Adding a scenario here
without a matching test (or vice versa) is a doc bug; fix the mismatch instead of leaving it
for later.

## Execution Context

### One-time setup: disable the testcontainers reaper

Same as the Laborious and Orchestrator suites. Once per machine:

```bash
echo "ryuk.disabled=true" >> ~/.testcontainers.properties
```

Under Docker Desktop on Linux the reaper ("Ryuk") cannot mount the Docker socket and every
test errors at setup (`ERROR`, not `FAILED`). Every fixture stops its own containers in
teardown; after a killed run, clean up by hand:

```bash
docker ps -a --filter "label=org.testcontainers=true" -q | xargs -r docker rm -f
```

### What runs for real

| Dependency | How | Scope |
|---|---|---|
| MongoDB (`mongo:7`, user/password) | testcontainers | session |
| Redis (`redis:7-alpine`) | testcontainers | session |
| PostgreSQL (`postgres:15`, `e2e/db_schema.sql`) | testcontainers | session |
| MinIO (`bitnamilegacy/minio`, client certificates) | testcontainers | session |
| NATS (`nats:2.10-alpine`) | testcontainers | session |
| OPC UA servers | in process (`e2e/opc_server.py`, asyncua `Server`) | per test |

- Gateway code under `ingestor/**` is not mocked. A pod (`e2e/helpers.Pod`) is built from the
  environment exactly like `ingestor/app.py` builds it — `Ingestor.prepare_ingestor`, then
  `GatewayRuntime` on the same data manager and connection pool — minus the Prometheus HTTP
  server. `Pod.tick()` runs one main loop iteration (`Ingestor.loop` + `GatewayRuntime.idle_sweep`)
  without the `POLL_INTERVAL` sleep, so every scenario controls exactly how many iterations run.
- The environment (`gateway_env` in `e2e/conftest.py`) points every setting at the containers;
  scenarios override single variables per pod (`LEASE_TTL`, `WRITE_TIMESTAMP`, ...).
- State is reset before every test: Redis `FLUSHDB`, Mongo database dropped, Postgres schema
  recreated from `e2e/db_schema.sql`, certificate bucket emptied.
- Inputs live in `e2e/scenario_inputs/*.json` and are loaded with `load_scenario_input`;
  `@server:<name>` markers resolve to the URL of that test's in-process OPC server.
- Default `POLL_INTERVAL=1` and tag `frequency=1000`: one OPC "cycle" per loop iteration.
- Not covered by the default `pytest` run (`testpaths = ["tests"]`) nor by the unit coverage.
  `--strict-markers` is on: a test without `@pytest.mark.integration` is not collected, and a run
  that collects nothing still exits 0 — check the collected count.

---

## 1. Slot Leasing and Load Balancing
Source: `e2e/test_slot_leasing.py`

### 1.1.1 Single Pod Leases the Slot and Heartbeats
**Summary**: A lone pod takes the only slot on its first loop iteration.

**Description**:
- Slot 1 reads `plant-a`; one pod starts and runs one iteration.

**Expected Outcome**:
- The pod manages slot `1`; `lease:opc_tags:1 = "pod-a"` with TTL ≤ `LEASE_TTL` (10 s).
- `heartbeat:ingestor:pod-a = 1` with TTL ≤ `HEARTBEAT_TTL` (20 s).
- Readings of the slot reach MongoDB.

### 1.1.2 Idle Pod Takes a Slot Created After Startup
**Summary**: A pod without slots picks up a slot the Orchestrator creates later.

**Description**:
- The pod starts with Redis empty and runs one iteration (nothing to lease).
- Slot 1 is written; the pod runs another iteration.

**Expected Outcome**:
- The pod manages slot `1`, holds its lease and its readings reach MongoDB.

### 1.2.1 Two Pods Split Two Slots
**Summary**: The rebalancing hands one slot to each pod.

**Description**:
- Slot 1 reads `plant-a`, slot 2 reads `plant-b`. `pod-a` starts alone, then `pod-b` starts.
- Iterations: `pod-a`, `pod-b`, `pod-a`, `pod-b`.

**Expected Outcome**:
- Alone, `pod-a` takes both slots; `pod-b` finds them leased.
- `pod-a` sees a pod without slots and gives slot `2` back (unsubscribing first); `pod-b` takes it.
- Leases: slot 1 → `pod-a`, slot 2 → `pod-b`; each pod only has the reader of its own server.
- New values of both servers reach MongoDB.

### 1.2.2 Extra Pod Stays Idle
**Summary**: With more pods than slots, no slot is read twice.

**Description**:
- Two slots balanced between `pod-a` and `pod-b`; `pod-c` joins and every pod iterates twice.

**Expected Outcome**:
- `pod-c` manages nothing; `pod-a` and `pod-b` hold disjoint slots covering `{1, 2}`.
- Leases belong to `pod-a` and `pod-b` only.

### 1.2.3 Failover After Lease Expiry
**Summary**: A slot left by a dead pod is taken over once its lease expires.

**Description**:
- `pod-a` (`LEASE_TTL=2`) reads slot 1; `pod-b` runs idle.
- `pod-a` stops without releasing the lease (crash-like).

**Expected Outcome**:
- The lease still names `pod-a` right after the stop, then expires.
- On its next iteration `pod-b` leases slot `1` and new values reach MongoDB.

### 1.3.1 Deleted Slot Stops the Reads
**Summary**: Removing a slot from Redis unsubscribes it.

**Description**:
- A pod reads slot 1; the slot key is deleted; the pod iterates.

**Expected Outcome**:
- The pod manages no slot and has no OPC reader.
- A value changed afterwards is not persisted.

### 1.3.2 Invalid Slot Is Notified and Not Read
**Summary**: A slot v1 (`topics`, no `security`/`names`) is rejected.

**Description**:
- Slot 1 holds the legacy v1 shape (`slot_v1_invalid.json`).

**Expected Outcome**:
- Notification `OPC_SLOT_CONFIG_ERROR_1` in `notification_queue`.
- The pod manages nothing; no reading is persisted.

### 1.3.3 Fixed Slot Is Read Once Its Lease Expires
**Summary**: Fixing an invalid slot takes effect after the lease of the failed acquisition expires.

**Description**:
- `pod-a` (`LEASE_TTL=2`) leases the invalid slot 1 (the lease is kept, the slot is not managed).
- The slot is fixed; the pod iterates before and after the lease expires.

**Expected Outcome**:
- Before expiry the pod still manages nothing (its own lease blocks the re-acquisition).
- After expiry the pod manages slot `1` and its readings reach MongoDB.

---

## 2. Subscription Reads (OPC UA → MongoDB)
Source: `e2e/test_subscription_reads.py`

### 2.1.1 Value Change Is Persisted
**Summary**: Each data change becomes one document in the name's collection.

**Description**:
- Slot 1 subscribes `ns=2;s=Temperature` of `plant-a` into `raw_plant_a`; the value changes.

**Expected Outcome**:
- The initial value and the new one are persisted.
- Document fields: `tag` (address), `name` (`tag_name`), `model_id`, `value`, `timestamp`
  (source timestamp, `YYYY-MM-DD HH:MM:SS-0300`) and `inserted_at` — nothing else.

### 2.1.2 One Address Fans Out to Every Name
**Summary**: An address with several `names` is written to each collection.

**Description**:
- `ns=2;s=Pressure` has two names: `raw_furnace`/`furnace_pressure`/model 10 and
  `raw_boiler`/`boiler_inlet_pressure`/model 20 (`slot_multi_name.json`).

**Expected Outcome**:
- One document per collection for the change, each with its own `name`/`model_id`, the same
  `tag` and the same `timestamp`.

### 2.1.3 Two Servers in One Slot
**Summary**: A slot can hold several OPC servers.

**Description**:
- Slot 1 holds `plant-a` and `plant-b` (`slot_two_servers.json`); both values change.

**Expected Outcome**:
- Both changes are persisted; the pod has one reader per server.

### 2.1.4 Values Keep Their OPC Type
**Summary**: Double, Int32, Boolean and String values are stored with their native type.

**Description**:
- Four tags of different OPC types change value (`slot_data_types.json`).

**Expected Outcome**:
- Each document stores the value with the matching Python/BSON type (`float`, `int`, `bool`, `str`).

### 2.2.1 Tag Added to the Slot Is Subscribed
**Summary**: A new tag in the slot is subscribed on the next iteration.

**Description**:
- The Orchestrator adds `ns=2;s=Level` to slot 1; the pod iterates.

**Expected Outcome**:
- Changes of the new tag and of the existing one are persisted.

### 2.2.2 Tag Removed from the Slot Is No Longer Read
**Summary**: A tag removed from the slot stops being persisted.

**Description**:
- Slot 1 reads `Temperature` and `Level`; `Level` is removed; the pod iterates.

**Expected Outcome**:
- `Temperature` keeps being persisted; a new `Level` value is not.

### 2.2.3 Name Added to a Tag Receives Its Readings
**Summary**: A new name on an existing tag starts receiving the readings.

**Description**:
- A second name (`raw_dryer`/`dryer_temperature`/model 50) is added to `Temperature`.

**Expected Outcome**:
- The next change is persisted in both collections.

### 2.2.4 Server URL Change Reconnects to the New Server
**Summary**: Changing the connection settings of a server re-initializes its reader.

**Description**:
- Slot 1 points `plant-a` to a second server (same tags) and the pod iterates.

**Expected Outcome**:
- Values of the new server are persisted; the session on the old server is closed.
- Changes on the old server are no longer persisted.

### 2.3.1 Unreachable Server Does Not Stop the Others
**Summary**: A server that cannot be reached is dropped; the other servers of the slot keep reading.

**Description**:
- Slot 1 holds `plant-a` and `plant-b`, whose URL has nothing listening.

**Expected Outcome**:
- Notification `OPC_CONNECTION_ERROR_plant-b`; the pod only has the `plant-a` reader.
- `plant-a` values are persisted.

### 2.3.2 Tag Without Data Raises a Warning
**Summary**: A tag without data changes for 5 cycles is notified.

**Description**:
- The value does not change for 5 iterations.

**Expected Outcome**:
- One WARNING notification `TAG_ns=2;s=Temperature:temperature_LISTENNING_STOPPED`.

### 2.3.3 Silent Server Is Reconnected After 15 Cycles
**Summary**: A server without any data change for 15 cycles is treated as silently dead.

**Description**:
- The value does not change for 15 iterations, then the pod iterates once more.

**Expected Outcome**:
- Notifications `OPC_LISTENNING_STOPPED__1` and `OPC_CONNECTION_RETRY__1`; the reader is dropped
  (pool connection evicted).
- The next iteration re-initializes the reader with a new connection; changes are persisted again.

### 2.3.4 Server Restart Resumes the Reads
**Summary**: A server that goes down and comes back is read again.

**Description**:
- The server stops; the pod iterates 15 times (reader dropped) and once more while it is down;
  the server starts again on the same port; the pod iterates twice.

**Expected Outcome**:
- While down, the reconnection fails with notification `OPC_CONNECTION_ERROR_plant-a`.
- Once back, the reader is re-initialized and changes are persisted again.

---

## 3. Writes (NATS C2 → OPC UA → Postgres `write_status` C3)
Source: `e2e/test_writes.py`

Common setup: `plant-a` with a writable `Setpoint`, its connector in `connectors`
(`connector_plant_a.json`, `connector_id "1"`) and a `pending` row in `sientia_data.predictions`
(model 2, `2026-10-13T12:00:00-03:00`). Commands come from `write_command.json` and are published
on `opc.write`; the gateway answers through the row's `write_status`.

### 3.1 DataValue Parts: Timestamp and Quality (SIENTIAPDE-2202)
**Summary**: The value is always written; the prediction timestamp (`SourceTimestamp`) and a Good
`StatusCode` are written unless turned off. Each flag comes from the command, else from the env
(`WRITE_TIMESTAMP` / `WRITE_QUALITY`), else `true`.

**Description**:
- `test_scenario_3_1_datavalue_parts`, one case per id. The timestamp is checked on the value the
  server stored; the quality on the DataValue encoding mask of the WriteRequest the gateway sent
  (an omitted StatusCode decodes as Good on the server, so only the wire tells them apart).

| Id | Env | Command flags | SourceTimestamp | StatusCode |
|---|---|---|---|---|
| `3_1_1_defaults_on` | — | — | sent | sent |
| `3_1_2_timestamp_off` | `WRITE_TIMESTAMP=false` | — | not sent | sent |
| `3_1_3_quality_off` | `WRITE_QUALITY=false` | — | sent | not sent |
| `3_1_4_both_off` | both `false` | — | not sent | not sent |
| `3_1_5_command_overrides_env` | `WRITE_TIMESTAMP=true` | `write_timestamp: false` | not sent | sent |
| `3_1_6_command_turns_on_over_env` | both `false` | both `true` | sent | sent |

**Expected Outcome**:
- `write_status = ok`; the server holds `1.25`.
- When sent, the stored `SourceTimestamp` equals the command `timestamp`; otherwise it is empty.

### 3.2.1 Pending Row Becomes Ok
**Summary**: A successful write records `ok`.

**Expected Outcome**:
- `write_status = ok`; the server holds the command value.

### 3.2.2 Null write_status Becomes Ok
**Summary**: Rows without a status (inserted before the column existed) are updated too.

**Expected Outcome**:
- `write_status` goes from NULL to `ok`.

### 3.2.3 Worse Outcome Is Kept
**Summary**: `ok` never replaces a worse outcome (`ok` < `error` < `session_bad`).

**Description**:
- Parametrized: the row starts as `error` or `session_bad`; a successful command arrives.

**Expected Outcome**:
- The OPC write happens, the row keeps its worse status.

### 3.2.4 Failure Replaces Ok
**Summary**: A worse outcome replaces a better one.

**Description**:
- A successful command records `ok`; a second command for the same row fails (conversion).

**Expected Outcome**:
- `write_status = error`.

### 3.2.5 Missing Row Is Not Created
**Summary**: The gateway only updates rows; it never inserts one.

**Description**:
- The command targets model 3, which has no prediction row.

**Expected Outcome**:
- The OPC write happens; no row for model 3 exists afterwards.

### 3.3.1 Unknown Server Is an Error
**Summary**: A `server_id` with no live connection and no connector fails the write.

**Expected Outcome**:
- `write_status = error`; nothing is written to the server.

### 3.3.2 Unusable Connector Is an Error
**Summary**: Only active `opc_ua` connectors with a URL and full security are used.

**Description**:
- Parametrized: connector `inactive`, `not_opc_ua` (`type: modbus`), `no_policy`, `no_url`.

**Expected Outcome**:
- `write_status = error`; nothing is written to the server.

### 3.3.3 Invalid Command Is an Error Without OPC Write
**Summary**: Commands that cannot become a DataValue are rejected before touching the server.

**Description**:
- Parametrized: `conversion` (`"abc"` as `int`), `unknown_data_type` (`decimal128`), `missing_addr`.

**Expected Outcome**:
- `write_status = error`; nothing is written to the server.

### 3.3.4 Bad Timestamp Skips the Write and Keeps the Row
**Summary**: A non-ISO `timestamp` (with the timestamp flag on) fails before the OPC write.

**Description**:
- The command carries `13/10/2026 12:00`; a marker command is used to know it was handled.

**Expected Outcome**:
- Only the marker reaches the server; the row stays `pending` — the status cannot be recorded
  either, since the row is found by that same timestamp.

### 3.3.5 OPC Rejection Is an Error
**Summary**: A write the server refuses is recorded as `error`.

**Description**:
- Parametrized: `not_writable` (read-only `Locked`), `unknown_node` (`ns=2;s=Missing`).

**Expected Outcome**:
- `write_status = error`; `Locked` keeps its value.

### 3.3.6 Malformed JSON Is Dropped and the Consumer Keeps Going
**Summary**: A payload that is not JSON is discarded without stopping the consumer.

**Expected Outcome**:
- The next valid command is written and recorded as `ok`.

### 3.4.1 Write Reuses the Read Connection
**Summary**: A server read by the pod is written through the same pooled connection.

**Description**:
- `plant-a` is only known through slot 1 (no connector in Mongo).

**Expected Outcome**:
- `write_status = ok`; the server has a single session.

### 3.4.2 Write-Only Server Gets Its Own Connection
**Summary**: A server only known as a connector is connected on the first write.

**Expected Outcome**:
- No session before the command; one session after; the pool holds the connector's settings.

### 3.5.1 Minimum Pause Between Writes
**Summary**: `WRITE_MIN_PAUSE_S` spaces consecutive writes.

**Description**:
- `WRITE_MIN_PAUSE_S=0.5`; three commands are published back to back.

**Expected Outcome**:
- The server receives the three writes at least ~0.5 s apart.

### 3.5.2 Queue Group Writes Each Command Once
**Summary**: Pods share the `gateway-writers` queue: each command is handled by a single pod.

**Description**:
- Two pods with the runtime; six commands.

**Expected Outcome**:
- The server receives exactly six writes.

### 3.6.1 NATS Down at Startup Connects Later
**Summary**: The runtime starts without NATS and connects from the main loop once NATS is up.

**Description**:
- `NATS_URL` points to a port with nothing listening; the pod starts and iterates once.
- A NATS server starts on that port; the pod iterates again.

**Expected Outcome**:
- One `GATEWAY_NATS_CONSUMER_START_FAILED` notification for the whole outage (not one per retry).
- After NATS starts, the consumer connects and a published command is recorded as `ok`.

---

## 4. On-demand gRPC ReadTag
Source: `e2e/test_grpc_read_tag.py`

Common setup: `GRPC_ENABLED=true`; slot 1 reads `plant-a`, so the pod's pool holds a connection.

### 4.1.1 Reads a Tag of a Connected Server
**Expected Outcome**:
- `value` (as text), `opc_status_code = Good`, `source_timestamp` / `server_timestamp` in
  `YYYY-MM-DD HH:MM:SS-0300`.
- `persisted = false` and no collection is created: the slot lookup of `ReadTag` is not wired yet.

### 4.1.2 Reads the Current Value
**Expected Outcome**:
- After a server-side change, `ReadTag` returns the new value.

### 4.2.1 Unknown Address Is an Internal Error
**Expected Outcome**:
- gRPC `INTERNAL` whose details carry `BadNodeIdUnknown`.

### 4.2.2 Server Not Connected by the Reads Is an Internal Error
**Summary**: `ReadTag` has no connection settings of its own: it only reads servers the read path
already connected (documents the current limitation).

**Description**:
- `plant-a` only exists as a connector; no slot.

**Expected Outcome**:
- gRPC `INTERNAL`; no session is opened on the server.

### 4.2.3 gRPC Disabled by Default
**Expected Outcome**:
- Without `GRPC_ENABLED`, nothing listens on `GRPC_PORT` (`UNAVAILABLE`/`DEADLINE_EXCEEDED`).

---

## 5. Scheduled Active Reads
Source: `e2e/test_active_read.py`

`GatewayRuntime` builds `ActiveReadEngine` on the pod's pool and sink, but no loop calls
`scheduled_tick` yet; these scenarios call it on the running pod's engine with the parsed slot
(`slot_active_read.json`).

### 5.1.1 Enabled Tag Is Read into Every Name
**Expected Outcome**:
- `Level` (`active_read.enabled`) is persisted in `active_tank` (model 30) and `active_mixer`
  (model 40); `Temperature` (disabled) is not read.

### 5.1.2 cycle_multiple Reads Every Nth Tick
**Description**:
- `cycle_multiple = 3`; six ticks.

**Expected Outcome**:
- Persisted documents after each tick: `0, 0, 1, 1, 1, 2`.

### 5.2.1 Unreachable Server Does Not Stop the Tick
**Expected Outcome**:
- The tick does not raise; the unreachable server's tag is not persisted, the other server's is.

### 5.2.2 Active Read Uses the Slot Connection Settings
**Expected Outcome**:
- The first tick opens the connection with the slot settings; later ticks reuse it (one session);
  each tick persists the current value.

---

## 6. Client Certificates and OPC Security
Source: `e2e/test_certificates.py`

Common setup: `plant-s` exposes only Basic256Sha256 SignAndEncrypt. Client certificates are created
like the platform stores them: cert (DER) and key (PEM) in MinIO bucket `opc-certificates`, plus an
`opc_certificates` document (`kind: client`).

### 6.1.1 Secured Read with Certificate from MinIO
**Expected Outcome**:
- Readings are persisted; the certificate is cached as `<CERT_CACHE_DIR>/7/cert.der` and `key.pem`.
- No connection error is notified.

### 6.1.2 Secured Write Resolved from Connectors and Certificates
**Description**:
- No slot: the connector (`client_certificate_id: 7`) is joined with `opc_certificates`.

**Expected Outcome**:
- `write_status = ok`; the server holds the value.

### 6.2.1 Certificate Missing in MinIO
**Expected Outcome**:
- Notification `OPC_CONNECTION_ERROR_plant-s`; no reader; nothing cached; no readings.

### 6.2.2 Unsupported Security Policy
**Expected Outcome**:
- Notification `OPC_CONNECTION_ERROR_plant-s` whose message names the unsupported mode/policy.

### 6.2.3 No Certificate Cannot Reach a Secured-only Server
**Description**:
- The slot has `certificate: null` (connect without security, D5).

**Expected Outcome**:
- Notification `OPC_CONNECTION_ERROR_plant-s`; no readings.

### 6.2.4 Inactive Certificate Write Fails
**Description**:
- The connector's certificate is `revoked`: the write connects without security.

**Expected Outcome**:
- `write_status = error`; nothing is written to the server.

### 6.3.1 Certificate Rotation Reconnects with the New One
**Description**:
- Slot 1 switches from certificate 7 to certificate 8; the pod iterates.

**Expected Outcome**:
- The pooled connection is replaced and opened with certificate 8 (cached under `8/`).
- Readings continue; the server ends with a single session.

---

## 7. Pod Lifecycle
Source: `e2e/test_lifecycle.py`

### 7.1.1 Shutdown Closes Sessions and Stops Consuming
**Expected Outcome**:
- After `GatewayRuntime.stop` + `Ingestor.shutdown`, the OPC server has no session.
- A command published afterwards is not handled (row stays `pending`); value changes are not persisted.

### 7.1.2 Idle Write Connection Is Closed and Reopened
**Description**:
- `IDLE_CONN_MAX_S=1`; `plant-a` is read (slot), `plant-b` only written (connector `2`).

**Expected Outcome**:
- After the idle time, `idle_sweep` closes the `plant-b` session; the subscribed `plant-a`
  connection is pinned and stays open.
- The next write to `plant-b` reconnects and is recorded as `ok`.

### 7.2.1 Redis Unreachable Fails the Startup
**Expected Outcome**:
- `prepare_ingestor` raises a connection error (`app.py` then stops the process).

### 7.2.2 Restarted Pod Takes Its Slot Back
**Description**:
- `pod-a` (`LEASE_TTL=2`) stops; a new `pod-a` starts after the lease expired.

**Expected Outcome**:
- The restarted pod leases slot `1` and its readings reach MongoDB.