sientia-dataops-opc-ingestor/e2e/scenarios.md

E2E Scenario Documentation — OPC Gateway

Functional reference for the gateway end-to-end scenarios: slot leasing, subscription reads, NATS writes, gRPC reads, active reads, certificates and the pod lifecycle. Tests live under e2e/, are marked with @pytest.mark.integration and run with:

pytest e2e/ -o testpaths=e2e -m integration

Every section below maps 1:1 to an actual test function (or to one pytest.param id of a parametrized test) — this file describes what the suite verifies today. Adding a scenario here without a matching test (or vice versa) is a doc bug; fix the mismatch instead of leaving it for later.

Execution Context

One-time setup: disable the testcontainers reaper

Same as the Laborious and Orchestrator suites. Once per machine:

echo "ryuk.disabled=true" >> ~/.testcontainers.properties

Under Docker Desktop on Linux the reaper ("Ryuk") cannot mount the Docker socket and every test errors at setup (ERROR, not FAILED). Every fixture stops its own containers in teardown; after a killed run, clean up by hand:

docker ps -a --filter "label=org.testcontainers=true" -q | xargs -r docker rm -f

What runs for real

Dependency	How	Scope
MongoDB (mongo:7, user/password)	testcontainers	session
Redis (redis:7-alpine)	testcontainers	session
PostgreSQL (postgres:15, e2e/db_schema.sql)	testcontainers	session
MinIO (bitnamilegacy/minio, client certificates)	testcontainers	session
NATS (nats:2.10-alpine)	testcontainers	session
OPC UA servers	in process (e2e/opc_server.py, asyncua Server)	per test

• Gateway code under ingestor/** is not mocked. A pod (e2e/helpers.Pod) is built from the environment exactly like ingestor/app.py builds it — Ingestor.prepare_ingestor, then GatewayRuntime on the same data manager and connection pool — minus the Prometheus HTTP server. Pod.tick() runs one main loop iteration (Ingestor.loop + GatewayRuntime.idle_sweep) without the POLL_INTERVAL sleep, so every scenario controls exactly how many iterations run.
• The environment (gateway_env in e2e/conftest.py) points every setting at the containers; scenarios override single variables per pod (LEASE_TTL, WRITE_TIMESTAMP, ...).
• State is reset before every test: Redis FLUSHDB, Mongo database dropped, Postgres schema recreated from e2e/db_schema.sql, certificate bucket emptied.
• Inputs live in e2e/scenario_inputs/*.json and are loaded with load_scenario_input; @server:<name> markers resolve to the URL of that test's in-process OPC server.
• Default POLL_INTERVAL=1 and tag frequency=1000: one OPC "cycle" per loop iteration.
• Not covered by the default pytest run (testpaths = ["tests"]) nor by the unit coverage. --strict-markers is on: a test without @pytest.mark.integration is not collected, and a run that collects nothing still exits 0 — check the collected count.

---

1. Slot Leasing and Load Balancing

Source: e2e/test_slot_leasing.py

1.1.1 Single Pod Leases the Slot and Heartbeats

Summary: A lone pod takes the only slot on its first loop iteration.

Description:

• Slot 1 reads plant-a; one pod starts and runs one iteration.

Expected Outcome:

• The pod manages slot 1; lease:opc_tags:1 = "pod-a" with TTL ≤ LEASE_TTL (10 s).
• heartbeat:ingestor:pod-a = 1 with TTL ≤ HEARTBEAT_TTL (20 s).
• Readings of the slot reach MongoDB.

1.1.2 Idle Pod Takes a Slot Created After Startup

Summary: A pod without slots picks up a slot the Orchestrator creates later.

Description:

• The pod starts with Redis empty and runs one iteration (nothing to lease).
• Slot 1 is written; the pod runs another iteration.

Expected Outcome:

• The pod manages slot 1, holds its lease and its readings reach MongoDB.

1.2.1 Two Pods Split Two Slots

Summary: The rebalancing hands one slot to each pod.

Description:

• Slot 1 reads plant-a, slot 2 reads plant-b. pod-a starts alone, then pod-b starts.
• Iterations: pod-a, pod-b, pod-a, pod-b.

Expected Outcome:

• Alone, pod-a takes both slots; pod-b finds them leased.
• pod-a sees a pod without slots and gives slot 2 back (unsubscribing first); pod-b takes it.
• Leases: slot 1 → pod-a, slot 2 → pod-b; each pod only has the reader of its own server.
• New values of both servers reach MongoDB.

1.2.2 Extra Pod Stays Idle

Summary: With more pods than slots, no slot is read twice.

Description:

• Two slots balanced between pod-a and pod-b; pod-c joins and every pod iterates twice.

Expected Outcome:

• pod-c manages nothing; pod-a and pod-b hold disjoint slots covering {1, 2}.
• Leases belong to pod-a and pod-b only.

1.2.3 Failover After Lease Expiry

Summary: A slot left by a dead pod is taken over once its lease expires.

Description:

• pod-a (LEASE_TTL=2) reads slot 1; pod-b runs idle.
• pod-a stops without releasing the lease (crash-like).

Expected Outcome:

• The lease still names pod-a right after the stop, then expires.
• On its next iteration pod-b leases slot 1 and new values reach MongoDB.

1.3.1 Deleted Slot Stops the Reads

Summary: Removing a slot from Redis unsubscribes it.

Description:

• A pod reads slot 1; the slot key is deleted; the pod iterates.

Expected Outcome:

• The pod manages no slot and has no OPC reader.
• A value changed afterwards is not persisted.

1.3.2 Invalid Slot Is Notified and Not Read

Summary: A slot v1 (topics, no security/names) is rejected.

Description:

• Slot 1 holds the legacy v1 shape (slot_v1_invalid.json).

Expected Outcome:

• Notification OPC_SLOT_CONFIG_ERROR_1 in notification_queue.
• The pod manages nothing; no reading is persisted.

1.3.3 Fixed Slot Is Read Once Its Lease Expires

Summary: Fixing an invalid slot takes effect after the lease of the failed acquisition expires.

Description:

• pod-a (LEASE_TTL=2) leases the invalid slot 1 (the lease is kept, the slot is not managed).
• The slot is fixed; the pod iterates before and after the lease expires.

Expected Outcome:

• Before expiry the pod still manages nothing (its own lease blocks the re-acquisition).
• After expiry the pod manages slot 1 and its readings reach MongoDB.

---

2. Subscription Reads (OPC UA → MongoDB)

Source: e2e/test_subscription_reads.py

2.1.1 Value Change Is Persisted

Summary: Each data change becomes one document in the name's collection.

Description:

• Slot 1 subscribes ns=2;s=Temperature of plant-a into raw_plant_a; the value changes.

Expected Outcome:

• The initial value and the new one are persisted.
• Document fields: tag (address), name (tag_name), model_id, value, timestamp (source timestamp, YYYY-MM-DD HH:MM:SS-0300) and inserted_at — nothing else.

2.1.2 One Address Fans Out to Every Name

Summary: An address with several names is written to each collection.

Description:

• ns=2;s=Pressure has two names: raw_furnace/furnace_pressure/model 10 and raw_boiler/boiler_inlet_pressure/model 20 (slot_multi_name.json).

Expected Outcome:

• One document per collection for the change, each with its own name/model_id, the same tag and the same timestamp.

2.1.3 Two Servers in One Slot

Summary: A slot can hold several OPC servers.

Description:

• Slot 1 holds plant-a and plant-b (slot_two_servers.json); both values change.

Expected Outcome:

• Both changes are persisted; the pod has one reader per server.

2.1.4 Values Keep Their OPC Type

Summary: Double, Int32, Boolean and String values are stored with their native type.

Description:

• Four tags of different OPC types change value (slot_data_types.json).

Expected Outcome:

• Each document stores the value with the matching Python/BSON type (float, int, bool, str).

2.2.1 Tag Added to the Slot Is Subscribed

Summary: A new tag in the slot is subscribed on the next iteration.

Description:

• The Orchestrator adds ns=2;s=Level to slot 1; the pod iterates.

Expected Outcome:

• Changes of the new tag and of the existing one are persisted.

2.2.2 Tag Removed from the Slot Is No Longer Read

Summary: A tag removed from the slot stops being persisted.

Description:

• Slot 1 reads Temperature and Level; Level is removed; the pod iterates.

Expected Outcome:

• Temperature keeps being persisted; a new Level value is not.

2.2.3 Name Added to a Tag Receives Its Readings

Summary: A new name on an existing tag starts receiving the readings.

Description:

• A second name (raw_dryer/dryer_temperature/model 50) is added to Temperature.

Expected Outcome:

• The next change is persisted in both collections.

2.2.4 Server URL Change Reconnects to the New Server

Summary: Changing the connection settings of a server re-initializes its reader.

Description:

• Slot 1 points plant-a to a second server (same tags) and the pod iterates.

Expected Outcome:

• Values of the new server are persisted; the session on the old server is closed.
• Changes on the old server are no longer persisted.

2.3.1 Unreachable Server Does Not Stop the Others

Summary: A server that cannot be reached is dropped; the other servers of the slot keep reading.

Description:

• Slot 1 holds plant-a and plant-b, whose URL has nothing listening.

Expected Outcome:

• Notification OPC_CONNECTION_ERROR_plant-b; the pod only has the plant-a reader.
• plant-a values are persisted.

2.3.2 Tag Without Data Raises a Warning

Summary: A tag without data changes for 5 cycles is notified.

Description:

• The value does not change for 5 iterations.

Expected Outcome:

• One WARNING notification TAG_ns=2;s=Temperature:temperature_LISTENNING_STOPPED.

2.3.3 Silent Server Is Reconnected After 15 Cycles

Summary: A server without any data change for 15 cycles is treated as silently dead.

Description:

• The value does not change for 15 iterations, then the pod iterates once more.

Expected Outcome:

• Notifications OPC_LISTENNING_STOPPED__1 and OPC_CONNECTION_RETRY__1; the reader is dropped (pool connection evicted).
• The next iteration re-initializes the reader with a new connection; changes are persisted again.

2.3.4 Server Restart Resumes the Reads

Summary: A server that goes down and comes back is read again.

Description:

• The server stops; the pod iterates 15 times (reader dropped) and once more while it is down; the server starts again on the same port; the pod iterates twice.

Expected Outcome:

• While down, the reconnection fails with notification OPC_CONNECTION_ERROR_plant-a.
• Once back, the reader is re-initialized and changes are persisted again.

---

3. Writes (NATS C2 → OPC UA → Postgres write_status C3)

Source: e2e/test_writes.py

Common setup: plant-a with a writable Setpoint, its connector in connectors (connector_plant_a.json, connector_id "1") and a pending row in sientia_data.predictions (model 2, 2026-10-13T12:00:00-03:00). Commands come from write_command.json and are published on opc.write; the gateway answers through the row's write_status.

3.1 DataValue Parts: Timestamp and Quality (SIENTIAPDE-2202)

Summary: The value is always written; the prediction timestamp (SourceTimestamp) and a Good StatusCode are written unless turned off. Each flag comes from the command, else from the env (WRITE_TIMESTAMP / WRITE_QUALITY), else true.

Description:

• test_scenario_3_1_datavalue_parts, one case per id. The timestamp is checked on the value the server stored; the quality on the DataValue encoding mask of the WriteRequest the gateway sent (an omitted StatusCode decodes as Good on the server, so only the wire tells them apart).

Id	Env	Command flags	SourceTimestamp	StatusCode
3_1_1_defaults_on	—	—	sent	sent
3_1_2_timestamp_off	WRITE_TIMESTAMP=false	—	not sent	sent
3_1_3_quality_off	WRITE_QUALITY=false	—	sent	not sent
3_1_4_both_off	both false	—	not sent	not sent
3_1_5_command_overrides_env	WRITE_TIMESTAMP=true	write_timestamp: false	not sent	sent
3_1_6_command_turns_on_over_env	both false	both true	sent	sent

Expected Outcome:

• write_status = ok; the server holds 1.25.
• When sent, the stored SourceTimestamp equals the command timestamp; otherwise it is empty.

3.2.1 Pending Row Becomes Ok

Summary: A successful write records ok.

Expected Outcome:

• write_status = ok; the server holds the command value.

3.2.2 Null write_status Becomes Ok

Summary: Rows without a status (inserted before the column existed) are updated too.

Expected Outcome:

• write_status goes from NULL to ok.

3.2.3 Worse Outcome Is Kept

Summary: ok never replaces a worse outcome (ok < error < session_bad).

Description:

• Parametrized: the row starts as error or session_bad; a successful command arrives.

Expected Outcome:

• The OPC write happens, the row keeps its worse status.

3.2.4 Failure Replaces Ok

Summary: A worse outcome replaces a better one.

Description:

• A successful command records ok; a second command for the same row fails (conversion).

Expected Outcome:

• write_status = error.

3.2.5 Missing Row Is Not Created

Summary: The gateway only updates rows; it never inserts one.

Description:

• The command targets model 3, which has no prediction row.

Expected Outcome:

• The OPC write happens; no row for model 3 exists afterwards.

3.3.1 Unknown Server Is an Error

Summary: A server_id with no live connection and no connector fails the write.

Expected Outcome:

• write_status = error; nothing is written to the server.

3.3.2 Unusable Connector Is an Error

Summary: Only active opc_ua connectors with a URL and full security are used.

Description:

• Parametrized: connector inactive, not_opc_ua (type: modbus), no_policy, no_url.

Expected Outcome:

• write_status = error; nothing is written to the server.

3.3.3 Invalid Command Is an Error Without OPC Write

Summary: Commands that cannot become a DataValue are rejected before touching the server.

Description:

• Parametrized: conversion ("abc" as int), unknown_data_type (decimal128), missing_addr.

Expected Outcome:

• write_status = error; nothing is written to the server.

3.3.4 Bad Timestamp Skips the Write and Keeps the Row

Summary: A non-ISO timestamp (with the timestamp flag on) fails before the OPC write.

Description:

• The command carries 13/10/2026 12:00; a marker command is used to know it was handled.

Expected Outcome:

• Only the marker reaches the server; the row stays pending — the status cannot be recorded either, since the row is found by that same timestamp.

3.3.5 OPC Rejection Is an Error

Summary: A write the server refuses is recorded as error.

Description:

• Parametrized: not_writable (read-only Locked), unknown_node (ns=2;s=Missing).

Expected Outcome:

• write_status = error; Locked keeps its value.

3.3.6 Malformed JSON Is Dropped and the Consumer Keeps Going

Summary: A payload that is not JSON is discarded without stopping the consumer.

Expected Outcome:

• The next valid command is written and recorded as ok.

3.4.1 Write Reuses the Read Connection

Summary: A server read by the pod is written through the same pooled connection.

Description:

• plant-a is only known through slot 1 (no connector in Mongo).

Expected Outcome:

• write_status = ok; the server has a single session.

3.4.2 Write-Only Server Gets Its Own Connection

Summary: A server only known as a connector is connected on the first write.

Expected Outcome:

• No session before the command; one session after; the pool holds the connector's settings.

3.5.1 Minimum Pause Between Writes

Summary: WRITE_MIN_PAUSE_S spaces consecutive writes.

Description:

• WRITE_MIN_PAUSE_S=0.5; three commands are published back to back.

Expected Outcome:

• The server receives the three writes at least ~0.5 s apart.

3.5.2 Queue Group Writes Each Command Once

Summary: Pods share the gateway-writers queue: each command is handled by a single pod.

Description:

• Two pods with the runtime; six commands.

Expected Outcome:

• The server receives exactly six writes.

3.6.1 NATS Down at Startup Connects Later

Summary: The runtime starts without NATS and connects from the main loop once NATS is up.

Description:

• NATS_URL points to a port with nothing listening; the pod starts and iterates once.
• A NATS server starts on that port; the pod iterates again.

Expected Outcome:

• One GATEWAY_NATS_CONSUMER_START_FAILED notification for the whole outage (not one per retry).
• After NATS starts, the consumer connects and a published command is recorded as ok.

---

4. On-demand gRPC ReadTag

Source: e2e/test_grpc_read_tag.py

Common setup: GRPC_ENABLED=true; slot 1 reads plant-a, so the pod's pool holds a connection.

4.1.1 Reads a Tag of a Connected Server

Expected Outcome:

• value (as text), opc_status_code = Good, source_timestamp / server_timestamp in YYYY-MM-DD HH:MM:SS-0300.
• persisted = false and no collection is created: the slot lookup of ReadTag is not wired yet.

4.1.2 Reads the Current Value

Expected Outcome:

• After a server-side change, ReadTag returns the new value.

4.2.1 Unknown Address Is an Internal Error

Expected Outcome:

• gRPC INTERNAL whose details carry BadNodeIdUnknown.

4.2.2 Server Not Connected by the Reads Is an Internal Error

Summary: ReadTag has no connection settings of its own: it only reads servers the read path already connected (documents the current limitation).

Description:

• plant-a only exists as a connector; no slot.

Expected Outcome:

• gRPC INTERNAL; no session is opened on the server.

4.2.3 gRPC Disabled by Default

Expected Outcome:

• Without GRPC_ENABLED, nothing listens on GRPC_PORT (UNAVAILABLE/DEADLINE_EXCEEDED).

---

5. Scheduled Active Reads

Source: e2e/test_active_read.py

GatewayRuntime builds ActiveReadEngine on the pod's pool and sink, but no loop calls scheduled_tick yet; these scenarios call it on the running pod's engine with the parsed slot (slot_active_read.json).

5.1.1 Enabled Tag Is Read into Every Name

Expected Outcome:

• Level (active_read.enabled) is persisted in active_tank (model 30) and active_mixer (model 40); Temperature (disabled) is not read.

5.1.2 cycle_multiple Reads Every Nth Tick

Description:

• cycle_multiple = 3; six ticks.

Expected Outcome:

• Persisted documents after each tick: 0, 0, 1, 1, 1, 2.

5.2.1 Unreachable Server Does Not Stop the Tick

Expected Outcome:

• The tick does not raise; the unreachable server's tag is not persisted, the other server's is.

5.2.2 Active Read Uses the Slot Connection Settings

Expected Outcome:

• The first tick opens the connection with the slot settings; later ticks reuse it (one session); each tick persists the current value.

---

6. Client Certificates and OPC Security

Source: e2e/test_certificates.py

Common setup: plant-s exposes only Basic256Sha256 SignAndEncrypt. Client certificates are created like the platform stores them: cert (DER) and key (PEM) in MinIO bucket opc-certificates, plus an opc_certificates document (kind: client).

6.1.1 Secured Read with Certificate from MinIO

Expected Outcome:

• Readings are persisted; the certificate is cached as <CERT_CACHE_DIR>/7/cert.der and key.pem.
• No connection error is notified.

6.1.2 Secured Write Resolved from Connectors and Certificates

Description:

• No slot: the connector (client_certificate_id: 7) is joined with opc_certificates.

Expected Outcome:

• write_status = ok; the server holds the value.

6.2.1 Certificate Missing in MinIO

Expected Outcome:

• Notification OPC_CONNECTION_ERROR_plant-s; no reader; nothing cached; no readings.

6.2.2 Unsupported Security Policy

Expected Outcome:

• Notification OPC_CONNECTION_ERROR_plant-s whose message names the unsupported mode/policy.

6.2.3 No Certificate Cannot Reach a Secured-only Server

Description:

• The slot has certificate: null (connect without security, D5).

Expected Outcome:

• Notification OPC_CONNECTION_ERROR_plant-s; no readings.

6.2.4 Inactive Certificate Write Fails

Description:

• The connector's certificate is revoked: the write connects without security.

Expected Outcome:

• write_status = error; nothing is written to the server.

6.3.1 Certificate Rotation Reconnects with the New One

Description:

• Slot 1 switches from certificate 7 to certificate 8; the pod iterates.

Expected Outcome:

• The pooled connection is replaced and opened with certificate 8 (cached under 8/).
• Readings continue; the server ends with a single session.

---

7. Pod Lifecycle

Source: e2e/test_lifecycle.py

7.1.1 Shutdown Closes Sessions and Stops Consuming

Expected Outcome:

• After GatewayRuntime.stop + Ingestor.shutdown, the OPC server has no session.
• A command published afterwards is not handled (row stays pending); value changes are not persisted.

7.1.2 Idle Write Connection Is Closed and Reopened

Description:

• IDLE_CONN_MAX_S=1; plant-a is read (slot), plant-b only written (connector 2).

Expected Outcome:

• After the idle time, idle_sweep closes the plant-b session; the subscribed plant-a connection is pinned and stays open.
• The next write to plant-b reconnects and is recorded as ok.

7.2.1 Redis Unreachable Fails the Startup

Expected Outcome:

• prepare_ingestor raises a connection error (app.py then stops the process).

7.2.2 Restarted Pod Takes Its Slot Back

Description:

• pod-a (LEASE_TTL=2) stops; a new pod-a starts after the lease expired.

Expected Outcome:

• The restarted pod leases slot 1 and its readings reach MongoDB.