gitea_admin/sientia-dataops-opc-ingestor
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Compare commits

base: gitea_admin:2.1.0
Branches Tags
gitea_admin:main gitea_admin:2.1.0 gitea_admin:release/SIENTIAPDE-1973 gitea_admin:test-SIENTIAPDE-2202-gw gitea_admin:test-SIENTIAPDE-1975-gw-r6 gitea_admin:hotfix-gateway-sqlalchemy-21 gitea_admin:test-SIENTIAPDE-1975-write-path gitea_admin:2.2.0-rc1 gitea_admin:test-SIENTIAPDE-1974-release-slot-keyerror gitea_admin:2.1.0-rc1 gitea_admin:2.0.0-rc3 gitea_admin:2.0.0-rc2 gitea_admin:2.0.0-rc1 gitea_admin:1.4.4 gitea_admin:1.4.2 gitea_admin:fix/SIENTIAPDE-2072
..
compare: gitea_admin:fix/SIENTIAPDE-2072
Branches Tags
gitea_admin:2.1.0 gitea_admin:main gitea_admin:release/SIENTIAPDE-1973 gitea_admin:test-SIENTIAPDE-2202-gw gitea_admin:test-SIENTIAPDE-1975-gw-r6 gitea_admin:hotfix-gateway-sqlalchemy-21 gitea_admin:test-SIENTIAPDE-1975-write-path gitea_admin:2.2.0-rc1 gitea_admin:test-SIENTIAPDE-1974-release-slot-keyerror gitea_admin:2.1.0-rc1 gitea_admin:2.0.0-rc3 gitea_admin:2.0.0-rc2 gitea_admin:2.0.0-rc1 gitea_admin:1.4.4 gitea_admin:1.4.2 gitea_admin:fix/SIENTIAPDE-2072

1 Commits
2.1.0 ... fix/SIENTI

Author SHA1 Message Date
gitea_admin
c27477b4d0 Upload files to "certs" 2026-09-01 13:11:41 +00:00
105 changed files with 1721 additions and 9228 deletions
Expand all files Collapse all files

28
.env.example
View File

@@ -15,31 +15,3 @@ MONGODB_USERNAME="mongo_user"
MONGODB_PASSWORD="mongo_password"
MONGODB_DATABASE="sientia"
# OPC Gateway
NATS_URL="nats://localhost:4222"
NATS_WRITE_SUBJECT="opc.write"
NATS_QUEUE="gateway-writers"
GRPC_ENABLED="false"
GRPC_PORT="50051"
CERT_CACHE_DIR="/tmp/opc-certs"
WRITE_TIMESTAMP="true"
WRITE_QUALITY="true"
WRITE_MIN_PAUSE_S="0.0"
IDLE_CONN_MAX_S="300.0"
# OPC Gateway - MinIO (client cert storage, read by GatewayRuntime/CertProvider)
MINIO_ENDPOINT="localhost:9000"
MINIO_ACCESS_KEY="minioadmin"
MINIO_SECRET_KEY="minioadmin"
MINIO_SECURE="false"
# OPC Gateway - Postgres (write_status updates, read by GatewayRuntime/DataSink
# via sientia_do.utils.connectors_config.build_postgres_config)
POSTGRES_HOST="localhost"
POSTGRES_PORT="5432"
POSTGRES_USER="sientia"
POSTGRES_PASSWORD="sientia"
POSTGRES_DBNAME="sientia"
POSTGRES_MIN_CONNECTIONS="5"
POSTGRES_MAX_CONNECTIONS="20"

34
.github/workflows/deploy.yml vendored
View File

@@ -1,34 +0,0 @@
name: Deploy DataOps Worker
on:
workflow_dispatch:
inputs:
chart_ref:
description: >
sientia-helm-chart branch to bump GITHUB_BRANCH on AND deploy — the literal
deploy target. Validated via git ls-remote; a branch that does not exist in
sientia-helm-chart is a hard error, not a silent fallback to main.
required: true
type: string
permissions:
contents: write
id-token: write
pull-requests: read
jobs:
deploy:
name: Deploy
uses: Aignosi/github_workflow_templates/.github/workflows/reusable-deploy-dataops.yml@main
with:
branch_name: ${{ github.ref_name }}
chart_ref: ${{ inputs.chart_ref }}
project_name: 'ingestor'
chart_name: 'opc-ingestor'
env_list_path: '[".env"]'
update_version_in: '["pyproject"]'
create_tarball: ${{ github.event_name == 'pull_request' }}
mirror: true
bump_chart: true
deploy: true
secrets: inherit

1
.github/workflows/quality-gate.yml vendored
View File

@@ -5,6 +5,7 @@ on:
branches:
- main
- 'release/**'
- 'feature/**'
pull_request:
branches:
- main

16
.github/workflows/release.yml vendored Normal file
View File

@@ -0,0 +1,16 @@
name: Create Release on Merge to Main
on:
pull_request:
types: [closed]
branches:
- main
jobs:
release:
if: github.event.pull_request.merged == true
uses: Aignosi/github_workflow_templates/.github/workflows/dataops-module-release.yml@main
permissions: write-all
with:
project_name: 'ingestor'
secrets: inherit

17
.github/workflows/version-bump.yml vendored
View File

@@ -1,17 +0,0 @@
name: Version Bump
permissions:
contents: write
on:
pull_request:
types: [closed]
branches: [main, 'release/**', 'feature/**']
jobs:
bump:
if: github.event.pull_request.merged == true
uses: Aignosi/github_workflow_templates/.github/workflows/reusable-version-bump.yml@main
with:
update_version_in: '["pyproject"]'
secrets: inherit

11
.gitignore vendored
View File

@@ -181,13 +181,4 @@ git_log
.git/
.ruff_cache/
.mypy_cache/
# Local planning/validation notes (not committed, not team docs)
docs/superpowers/
docs/e2e-test-guide.md
docs/deploy-notes.md
docs/HANDOFF.md
openspec/
.claude/
.mypy_cache/

13
CLAUDE.md
View File

@@ -17,9 +17,9 @@ pip install -r requirements.txt -r requirements-dev.txt
./run_local.sh # loads .env, runs python -m ingestor.app
./run_coverage.sh # pytest --cov=ingestor --cov-report=html, opens report
pytest # unit tests (testpaths = tests)
pytest # all tests
pytest tests/unit # unit only
pytest e2e/ -o testpaths=e2e -m integration # E2E (Docker: testcontainers + in-process OPC servers)
pytest tests/functional # functional only (spins up real-ish clients, see conftest.py)
pytest tests/unit/test_ingestor.py::TestClass::test_name # single test
pytest --cov=ingestor --cov-report=html # coverage report -> htmlcov/
@@ -40,8 +40,7 @@ app.py (asyncio loop, signals, Prometheus server)
-> Ingestor (ingestor.py) — lifecycle: prepare_ingestor() -> loop() -> shutdown()
-> IngestorManager (managers/ingestor_manager.py) — central coordinator
-> ResourceManager (managers/resource_manager.py) — Redis slot leases, heartbeats, active-ingestor registry
-> ServerReader (managers/server_reader.py), one per OPC server — SubscriptionEngine over the shared OpcConnectionPool
-> OpcConnectionPool (connection/pool.py) + CertProvider (connection/certs.py) — pooled connections, cert from MinIO
-> OpcManager (managers/opc_manager.py), one instance per OPC server — connection, subscriptions, datachange callbacks
-> DataManager (managers/data_manager.py) — MongoDB persistence + optional Kafka publish
```
@@ -57,9 +56,7 @@ app.py (asyncio loop, signals, Prometheus server)
Any change to slot/lease counts should be traced through `manage_leases`/`manage_no_slots` — that's where the rebalancing math lives, not in `ResourceManager` itself (which just does Redis primitives).
**ServerReader** (SIENTIAPDE-1974) reads one OPC server through a `SubscriptionEngine`: the connection comes from the shared `OpcConnectionPool` (also used by NATS writes and gRPC). Slots are slot v2 only, parsed by `io/slots.py` (`parse_slot`): each server carries `security` (`mode`, `policy`) and `certificate` (certificate_id, application_uri, MinIO `bucket`/`object_name` of cert and private_key, from `opc_certificates` via the Orchestrator; `None` = connect without security, D5). The pool applies security only with a certificate (`connection/security.py`: `is_secured`, `apply_security` maps mode/policy to asyncua, unknown → error), and `CertProvider` downloads each certificate into `<CERT_CACHE_DIR>/<certificate_id>/` (cache, lock and `invalidate` per certificate) — no certificate files in the repo/image, no default certificate. A change of `security`/`certificate` in the slot re-initializes the reader (pool connection evicted). `datachange_notification` fans each value out to every `names` entry via `DataSink`. Subscribed connections are pinned in the pool so the idle sweep never drops them. Tracks `non_receive_count` to detect silently-dead subscriptions (`check_cycles` / `check_opc_listenning`); a lost server is shut down (pool connection evicted) and re-initialized on the next loop.
**Writes** (SIENTIAPDE-1975): `NatsWriteConsumer` → `WriteEngine` → pool, then `DataSink.update_write_status`. Commands (C2) carry only `server_id`; `ServerConfigResolver` (`connection/servers.py`) resolves the connection: live pooled connection → `connectors` aggregation with a `$lookup` on `opc_certificates`, in the slot format. The certificate rule (active + client, `connection/certificates.py`) mirrors the Orchestrator's `tag_database.resolve_certificate` — keep both in sync (SIENTIAPDE-1974 D1; the Mongo fallback goes away when writes are routed to the pod that owns the connection). `write_status` keeps the worst outcome (`ok` < `error` < `session_bad`); the gateway is the only one that updates it (the Laborious inserts the row as `pending` and never records OPC outcomes). The NATS first connect is bounded (it runs on the main loop); reconnection after it is unbounded.
**OpcManager** owns one OPC UA `Client` connection (asyncua), handles security policy setup (cert/key-based, `SecurityPolicyBasic256`), subscriptions per tag group, and `datachange_notification` callbacks that push into `data_queue` for `DataManager` to consume. Tracks `non_receive_count` to detect silently-dead subscriptions (`check_cycles`).
**Metrics** (`ingestor/metrics.py`): all Prometheus metrics defined here, always labeled by `pod_id`. Emitted via `SientiaMonitoring.emit_metric(...)` (from `sientia_do`), not the prometheus client directly — new metrics should follow that pattern for consistency with logging/notifications.
@@ -72,6 +69,6 @@ Env vars documented in README ("Configuration" section) and `.env.example`. Nota
## Testing notes
- `tests/unit/managers/` mirrors `ingestor/managers/` 1:1.
- `e2e/` is the end-to-end suite (Laborious/Orchestrator pattern): real MongoDB/Redis/Postgres/MinIO/NATS via testcontainers, in-process asyncua servers, pods built like `app.py` (`e2e/helpers.Pod`). Scenarios are numbered by topic in `e2e/scenarios.md`, 1:1 with `test_scenario_<n>_<n>_<n>_*` — keep both in sync. Marked `integration`, not collected by the default `pytest`.
- `tests/functional/` exercises more end-to-end paths against `conftest.py` fixtures.
- Coverage config (`pyproject.toml` / `.coveragerc`) excludes `ingestor/app.py` (thin entrypoint, signal wiring) — don't chase coverage there.
- `ruff` complexity cap is `max-complexity = 15` (mccabe) — factor out branches in manager methods before that.

211
README.md
View File

@@ -49,64 +49,6 @@ The OPC Ingestor uses a modular, manager-based architecture designed for scalabi
- **Data Manager**: Manages data persistence (MongoDB)
- **Resource Manager**: Coordinates resource allocation and instance coordination via Redis
## 🚪 OPC Gateway (SIENTIAPDE-1972)
Alongside the passive ingestion pipeline above, this service also runs an OPC
Gateway for on-demand and command-driven interactions with OPC UA servers:
- **NATS write consumer**: consumes write commands from `opc.write` (see
`NATS_WRITE_SUBJECT`) and executes them against the target OPC server
(`ingestor/adapters/nats_consumer.py`, `ingestor/engines/write.py`). See
[Write commands](#write-commands-c2-and-write_status-c3) below.
- **On-demand gRPC `ReadTag`**: a gRPC server (`ingestor/adapters/grpc_server.py`,
`ingestor/proto/opc_gateway.proto`) for synchronous, single-tag reads.
- **Scheduled active reads**: a polling read engine (`ingestor/engines/active_read.py`)
for tags that need periodic reads outside the subscription pipeline;
orchestrator-side wiring lands in SIENTIAPDE-1974.
- **Certificate management**: each OPC connector has its own client certificate
(`opc_certificates`), named in the slot and downloaded from MinIO
(`ingestor/connection/certs.py`) rather than shipped in the image.
See [Gateway environment variables](#gateway-environment-variables) and the
`# OPC Gateway` blocks in `.env.example` for the new environment variables.
### Write commands (C2) and `write_status` (C3)
The Laborious publishes one command per tag (SIENTIAPDE-1975):
```json
{"server_id": "1", "addr": "ns=2;s=Pred", "value": 1.25, "data_type": "float",
"model_id": "2", "model_name": "sin-approx", "schema": "sientia_data",
"table_name": "predictions", "timestamp": "2026-10-13T12:00:00-03:00"}
```
- **Written `DataValue`** (SIENTIAPDE-2202): the value, plus the prediction
`timestamp` as `SourceTimestamp` and a Good `StatusCode`. Timestamp and quality can be
turned off separately; the value is always written. Each flag is taken from the
command (`write_timestamp` / `write_quality`) when present, else from the env
(`WRITE_TIMESTAMP` / `WRITE_QUALITY`), else `true`. The envs apply to every server of
the gateway (per-server config is a future SIENTIAPDE-2276 card).
- **Server connection**: the command carries only `server_id`. The connection is
resolved by `ServerConfigResolver` (`ingestor/connection/servers.py`): the live
pooled connection opened by the reads, else the `connectors` document (`type: opc_ua`,
`status: Active`) joined with `opc_certificates`, with the orchestrator's rule for the
read slots (`ingestor/connection/certificates.py`), in the slot format (`url`,
`security`, `certificate`). Lookups are cached for 60 s. An unknown or incomplete
server is a failed write (`server_not_found`).
- **Outcome**: `UPDATE ... SET write_status` on the prediction row (by `timestamp` +
`model_id`), keeping the worst outcome: `ok` < `error` < `session_bad`, and
`pending`/NULL are replaced by any answer, so the order of the answers does not
matter. The gateway is the only one that updates the column: the Laborious inserts the
row as `pending` (merged with its PI Web API outcome) and never records OPC outcomes.
The UPDATE never raises; its outcome (`updated`, `kept`, `row_not_found`,
`invalid`, `error`) is logged and counted in `opc_write_status_update_total`.
The reason of a failed write is logged by `WriteEngine`.
- **NATS connection**: the first connection gives up after 5 s (it runs on the main
loop: nats-py's defaults would block it for minutes while NATS is down); once
connected, it reconnects forever. A connection closed for good is started again
by `GatewayRuntime.idle_sweep`.
## 📋 Prerequisites
- Python 3.11+
@@ -160,9 +102,9 @@ The Laborious publishes one command per tag (SIENTIAPDE-1975):
# Ensure services are accessible on localhost with appropriate ports
```
6. **Register the OPC UA client certificate** (only if connecting to a secured OPC UA server)
6. **Generate an OPC UA client certificate** (only if connecting to a secured OPC UA server)
See [OPC Client Certificate](#-opc-client-certificate) below.
See [Certificate Generation](#-certificate-generation) below.
## Usage
@@ -230,44 +172,37 @@ pytest --cov=ingestor --cov-report=html
The feeder creates sample OPC tag configurations in Redis that the ingestor can discover and manage.
## 🔐 OPC Client Certificate
## 🔐 Certificate Generation
Each OPC connector has its own X.509 client certificate (SIENTIAPDE-1974): the
connector's `security.client_certificate_id` points to an `opc_certificates` document
(`kind: client`, `status: active`). The Orchestrator joins them and writes, for each
server of the slot, its `security` (`mode`, `policy`) and its `certificate`:
OPC UA servers that require secure connections (e.g. KEPServer with `Basic256` policy) validate the ingestor via an X.509 client certificate. Use `scripts/generate-and-push-opc-cert.sh` to generate that certificate and push it to the Gitea repo the pod clones on boot.
```json
"certificate": {
"certificate_id": 1,
"application_uri": "urn:sientia:opc-ingestor",
"cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-cert.pem"},
"private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-key.pem"}
}
```bash
GITEA_PASSWORD='<senha-do-gitea>' ./scripts/generate-and-push-opc-cert.sh
```
The certificate is **never** read from the repository or the image. On connect, the
`CertProvider` (`ingestor/connection/certs.py`) downloads both objects from MinIO into
`CERT_CACHE_DIR/<certificate_id>/` (default `/tmp/opc-certs`), and caches them per
certificate. When the slot names other objects for the same certificate (it was
replaced), the server config changes, the connection is re-opened and the files are
downloaded again. A failed secured connection also forces a new download.
Environment variables (all optional except `GITEA_PASSWORD`):
- **Security only with a certificate.** With a `certificate`, the connection uses the
slot's `security.mode` (`Sign` / `SignAndEncrypt`) and `security.policy`
(`Basic256Sha256`, `Aes128Sha256RsaOaep`, `Aes256Sha256RsaPss`, `Basic256`,
`Basic128Rsa15`); an unknown one fails the connection. Without a certificate
(`null`), or with `mode: None`, it connects without security, and a server that
requires a certificate refuses the connection (the error shows in the log and the
connection metrics). There is no default certificate.
- The certificate's `application_uri` (its SAN URI) is sent as the client application
URI (KEPServer rejects a mismatch with `BadCertificateUriInvalid`).
- The server must trust the certificate (e.g. KEPServer: *OPC UA Configuration Manager
→ Trusted Clients → Trust*, then restart its runtime).
| Variable | Description | Default |
|----------|-------------|---------|
| `GITEA_URL` | Gitea in-cluster URL | `https://git.sientia.ai` |
| `GITEA_USER` | Gitea user | `gitea_admin` |
| `GITEA_PASSWORD` | Gitea password (**required**) | — |
| `GITEA_REPO` | Repo the ingestor pod clones | `gitea_admin/sientia-dataops-opc-ingestor` |
| `BRANCH` | Branch to push the cert to | `main` |
| `APP_URI` | SAN URI of the cert — **must match** the `uri` field of the corresponding document in the `OPC_servers` Mongo collection, since the ingestor uses it as `application_uri` (KEPServer rejects with `BadCertificateUriInvalid` on mismatch) | `urn:sientia:opc-ingestor` |
| `CN` | Certificate common name | `sientia-opc-ingestor` |
| `DAYS` | Certificate validity (days) | `3650` |
| `FORCE` | Set to `1` to overwrite an existing cert already pushed to the repo | unset |
MinIO is only contacted when a certificate is first needed (one client per bucket), so
a MinIO outage only affects secured servers, which are retried on the next main-loop
iteration.
What the script does:
1. Generates a self-signed RSA-2048 X.509 cert + key with `openssl`, with the SAN URI, `keyUsage`, and `extendedKeyUsage` extensions OPC UA / KEPServer require.
2. Clones the ingestor's Gitea repo, copies the cert/key (`.pem`) and a `.der` copy (for manual import) into `certs/`, commits, and pushes.
3. Prints the paths the pod will see after cloning (`/app/certs/opc-ingestor-cert.pem`, `/app/certs/opc-ingestor-key.pem`) and the next manual steps.
After pushing, you still need to:
- Set `cert_path` / `private_key_path` (and `uri`) on the server's document in the `OPC_servers` collection so `OpcManager` picks them up (see [OPC Server Configuration](#opc-server-configuration)).
- Restart the ingestor deployment so it re-clones the repo: `kubectl -n sientia rollout restart deployment/opc-ingestor`.
- On first connection, the server rejects the new cert — trust it in the OPC UA server's console (e.g. KEPServer: *OPC UA Configuration Manager → Trusted Clients → Trust*) and restart its runtime.
## 🧪 Testing
@@ -287,14 +222,11 @@ pytest --cov=ingestor
pytest --cov=ingestor --cov-report=html
```
### End-to-end Tests
Every dependency runs for real (MongoDB, Redis, PostgreSQL, MinIO and NATS via testcontainers;
OPC UA servers in process). Requires Docker; not part of the default `pytest` run.
### Functional Tests
```bash
pytest e2e/ -o testpaths=e2e -m integration
# Run functional tests
pytest tests/functional/
```
Numbered scenarios (slot leasing, reads, writes, gRPC, active reads, certificates, lifecycle):
[`e2e/scenarios.md`](e2e/scenarios.md). Setup and layout: [`e2e/README.md`](e2e/README.md).
## 📊 Monitoring and Metrics
@@ -321,15 +253,6 @@ The OPC Ingestor exposes comprehensive Prometheus metrics for monitoring:
- `ingestor_tag_written_count`: Data write operations
- `redis_operations_total`: Redis operation count
### OPC Gateway Metrics
- `opc_write_total`: Active write attempts (NATS-driven), by result
- `opc_write_latency_seconds`: Active write latency
- `opc_write_status_by_model`: Last write outcome per model (1 = ok, 0 = failed; label `status`)
- `opc_write_status_update_total`: `write_status` UPDATEs, by result
(`updated`, `kept`, `row_not_found`, `invalid`, `error`)
- `opc_active_read_total`: Scheduled/on-demand read attempts, by result
- `opc_error_by_type_total`: OPC errors by type (connection, conversion, session, ...)
## ⚙️ Configuration
### Environment Variables
@@ -350,73 +273,41 @@ The OPC Ingestor exposes comprehensive Prometheus metrics for monitoring:
| `MONGODB_PASSWORD` | MongoDB password | `sientia` | No |
| `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
#### Gateway environment variables
Read by `GatewayConfig` (`ingestor/config.py`). Booleans accept `true`/`1`/`yes`
(case-insensitive); anything else is `false`.
| Variable | Description | Default |
|----------|-------------|---------|
| `NATS_URL` | NATS server for write commands (C2) | `nats://localhost:4222` |
| `NATS_WRITE_SUBJECT` | Subject of the write commands | `opc.write` |
| `NATS_QUEUE` | Queue group shared by the gateway pods | `gateway-writers` |
| `GRPC_ENABLED` | Starts the on-demand `ReadTag` gRPC server | `false` |
| `GRPC_PORT` | gRPC server port | `50051` |
| `CERT_CACHE_DIR` | Local cache of the client certificates downloaded from MinIO | `/tmp/opc-certs` |
| `WRITE_TIMESTAMP` | Writes the prediction `timestamp` as `SourceTimestamp` when the command has no `write_timestamp` | `true` |
| `WRITE_QUALITY` | Writes a Good `StatusCode` when the command has no `write_quality` | `true` |
| `WRITE_MIN_PAUSE_S` | Minimum pause between two OPC writes (seconds) | `0.0` |
| `IDLE_CONN_MAX_S` | Idle time before an unpinned pool connection is closed (seconds) | `300.0` |
`WRITE_TIMESTAMP` / `WRITE_QUALITY` are independent and apply to every OPC server of
the gateway; the value is always written. See [Write commands](#write-commands-c2-and-write_status-c3)
for the precedence (command → env → `true`). MinIO (`MINIO_*`) and Postgres
(`POSTGRES_*`) settings are listed in `.env.example`.
### OPC Server Configuration
OPC servers are configured by the Orchestrator through Redis, one key per slot
(`slot:opc_tags:<n>`). The gateway reads the **slot v2** structure (contract C1):
one entry per address, with one `names` item per collection / tag name / model, so
the same address can feed several models under different names. Each server carries
its `security` and its `certificate` (`null` = no security, see
[OPC Client Certificate](#-opc-client-certificate)):
OPC servers are configured through Redis with the following structure. `cert_path` / `private_key_path` are only required for secured connections (see [Certificate Generation](#-certificate-generation)):
```json
{
"slot:opc_tags:1": {
"server1": {
"name": "server1",
"server_id": "kepserver",
"server_id": "1",
"url": "opc.tcp://localhost:4841",
"subscription_period_ms": 7500,
"security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256"},
"certificate": {
"certificate_id": 1,
"application_uri": "urn:sientia:opc-ingestor",
"cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-cert.pem"},
"private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-key.pem"}
},
"server_uri": "http://opcua-server.simulator",
"cert_path": "/app/certs/opc-ingestor-cert.pem",
"private_key_path": "/app/certs/opc-ingestor-key.pem",
"tags": {
"ns=2;i=2": {
"frequency": "15000",
"active_read": {"enabled": false},
"names": [
{"collection": "raw_scouter-opcua-orchestrated-pipeline", "tag_name": "Counter", "model_name": "m1", "model_id": "1"},
{"collection": "raw_scouter-basic-sum-model", "tag_name": "Contador", "model_name": "m2", "model_id": "2"}
]
}
"aggr_func": "avg",
"data_range": [
-100,
100
],
"frequency": "15000",
"server_id": "1",
"tag_address": "ns=2;i=2",
"tag_name": "Counter",
"topics": [
"raw_scouter-opcua-orchestrated-pipeline",
"raw_scouter-basic-sum-model"
]
}
}
}
}
```
Only the slot v2 is accepted (`ingestor/io/slots.py`): a slot without `security`, a
tag without `names`, or a name without `collection`, `tag_name` or `model_id` is
rejected and notified (`OPC_SLOT_CONFIG_ERROR_<slot>`), and the last good config keeps
being read. The gateway and the Orchestrator of SIENTIAPDE-1974 are deployed together.
## 🔧 Development
### Project Structure
@@ -426,13 +317,13 @@ sientia-dataops-opc-ingestor/
│ ├── managers/ # Component managers
│ │ ├── data_manager.py # Data persistence and export
│ │ ├── ingestor_manager.py # Main coordination
│ │ ├── resource_manager.py # Resource coordination
│ │ └── server_reader.py # OPC UA server reads (SubscriptionEngine + pool)
│ │ ├── opc_manager.py # OPC UA server management
│ │ └── resource_manager.py # Resource coordination
│ ├── app.py # Main application entry point
│ ├── ingestor.py # Core ingestor logic
│ └── metrics.py # Prometheus metrics definitions
├── simulator/ # OPC simulation and testing tools
├── scripts/ # Ops scripts (e2e seed)
├── scripts/ # Ops scripts (e.g. OPC UA cert generation)
├── tests/ # Test suite
└── requirements.txt # Python dependencies
```

BIN
certs/opc-ingestor-cert.der Normal file
View File

Binary file not shown.

23
certs/opc-ingestor-cert.pem Normal file
View File

@@ -0,0 +1,23 @@
-----BEGIN CERTIFICATE-----
MIID0zCCArugAwIBAgIUW3zWE3LXrXXSwmWnqDZWlx1GIrowDQYJKoZIhvcNAQEL
BQAwQzEdMBsGA1UEAwwUc2llbnRpYS1vcGMtaW5nZXN0b3IxEDAOBgNVBAoMB0Fp
Z25vc2kxEDAOBgNVBAsMB1NpZW50aWEwHhcNMjYwNzE1MjAwMDI3WhcNMzYwNzEy
MjAwMDI3WjBDMR0wGwYDVQQDDBRzaWVudGlhLW9wYy1pbmdlc3RvcjEQMA4GA1UE
CgwHQWlnbm9zaTEQMA4GA1UECwwHU2llbnRpYTCCASIwDQYJKoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBALUoKgKJTlfpWqraCoi4f0GgAtmgQcIu+VD+7Y89PSNKx+bk
HCoe67nEHnFHXEhtKVxprQ1/001bQjyrKjZOIjd6G2WvYfYP0FROa67P+3zzQ1sY
/mkQBvV6LdZOy6tk9Amk8QYVbVOXmacg8+0elAxeM2swl+02W9yuTcOpTGNLA7bW
vTZrFL9DqPcghdeA6ZlD9AuUGaYvj7C/ShR+ELTHblUQO+f4xxpEXDv6w/8UaUaW
xZMJtWbBO4fX0DIvMtCBi28oxGD2obbTVXHbgKDzuwqPh5qP4IPJJQEg4M8z6ttf
2/OfFOc1O8XcFRK9vji4TBJFbs5wTDE6cPogFJcCAwEAAaOBvjCBuzAdBgNVHQ4E
FgQU64naOPKzlGirtomkUtclQOIjtMAwHwYDVR0jBBgwFoAU64naOPKzlGirtomk
UtclQOIjtMAwOQYDVR0RBDIwMIYYdXJuOnNpZW50aWE6b3BjLWluZ2VzdG9yghRz
aWVudGlhLW9wYy1pbmdlc3RvcjAOBgNVHQ8BAf8EBAMCBPAwIAYDVR0lAQH/BBYw
FAYIKwYBBQUHAwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEL
BQADggEBAGM4GXMc0EzsF2D02HOV0ZW1Igr5ya+561K0Cpg1SmMyQMFdCSk3s53I
OrB/hcUNQ/xtg3MeVNQSGeif/dijNWCVKN8o7dGvZU0gjm7j/ue2gIj53TjONw8K
QdF8vxvSDWgRHYFtPB3DlmWrHZaIeYD7Q60tnG1aaaM4pyRF0vH90PJL+XA9Tz4f
9Q5+TzyqMPs87+fPpWtI5YwAhpLYfRU/nbpiTEqYUn8L97oFCyn4fjOa7cN5Wil3
kiJgexHb0GimUiiKJeKmXDGgwZdw/xol//oB2DjhYPNcaPXTI4cqVtnTjuPwSkSS
ZnxDPAwl0ssqzil895bYcq3AHG/jrro=
-----END CERTIFICATE-----

28
certs/opc-ingestor-key.pem Normal file
View File

@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC1KCoCiU5X6Vqq
2gqIuH9BoALZoEHCLvlQ/u2PPT0jSsfm5BwqHuu5xB5xR1xIbSlcaa0Nf9NNW0I8
qyo2TiI3ehtlr2H2D9BUTmuuz/t880NbGP5pEAb1ei3WTsurZPQJpPEGFW1Tl5mn
IPPtHpQMXjNrMJftNlvcrk3DqUxjSwO21r02axS/Q6j3IIXXgOmZQ/QLlBmmL4+w
v0oUfhC0x25VEDvn+McaRFw7+sP/FGlGlsWTCbVmwTuH19AyLzLQgYtvKMRg9qG2
01Vx24Cg87sKj4eaj+CDySUBIODPM+rbX9vznxTnNTvF3BUSvb44uEwSRW7OcEwx
OnD6IBSXAgMBAAECggEAJUF7UkY2rHG7ScPJwtBeq3pi26rPm0XZ1ER5SqoHdsic
T+m9uRPe2rdwo09PtT1t8ilxg4M0S4ZnGt8AsWZkNxCPrdvhw58A8zWlg7AWQl4t
DgEX4TeAlt9nqdlOnZMHdSxMvTmO4FAfM3XddHiDbf+5h2PY+tnhDoNyTrj0kr6z
8JCD35FuOdU1zOBmum7T4eJDK1ol7XZ+UdrPraeM/jqDKaAnutUkRaDCpxUdaR7m
RKG9XoqC0mw6t/esPbSbA4gUDpojgle69QXx98M/EhTzE3Q4bSGsjT9G5q3/6rHB
RWgC2dWsTJOOI2y8Eym2MFLtf0MksZ4cgBtI0zW9AQKBgQDjjnVHSXC4LZ/Lbzdd
bAE0lzTjpoNMKLyL+spWzbnY6kYhq79XC2lb8C1aor9P9Gkq+uUSqyt0yZ0nEjB4
baCiKHEwe9qmBqyLEc84d6rt0aYzOxzzG/UGzMyvl2lhpJmofAxjgw4vaG2IVMog
qaQymfIPNCXIo0UdUNzDWnaTQQKBgQDLzPks/MtMcCVCReJT3LYGD4D4NDAxv0/q
+V0fLngKP3DmeLRbvqWyTQ6bCM6SH6HfOJYLLyK39DBnf0uPbolPH/iBuRqkl7Ez
YxcM4EJMR3U+aQH8hH24RJupr16OT3T+bZXK1pdjzM9SqwXiVI1rq/kRokfXT3hC
GBkhZrUp1wKBgQCJv5rO5G4Pj87DXzesncy9AStLA2lHmQFJ2y7RDQgLfwWlLEpP
buEAfot16j7VL6baFbbFbkiIEePH2wLsY7BxePnL162sAy4/oTBHhmk6PUbWjFoh
8DKLQz5ErOtl6DX7/9ugAZ18pUJo3KreytJSr04QVgjpQ6x66pA40XImAQKBgHLF
4J5BT4ygq5sDGE76QVhwasVdPrcFctB11se0t82tpLtmxsvUNiSP2OXes9b/EX5Q
s89goEDQuGHRApCoQ5PYAW+s+yAufZDfASAzO6MRqymWoZPAWbzohnGtXabvgIe4
118AMwn1UdzQi+PGkk5M71vK9IKBMsymHaD94pDJAoGBANAbRSmWgJfZNsYiVV7u
kQ6L5Y5aN2/LfHlqNbWFYot00olsUWPz2WUEvKXI2e3L6lmkGJFNREOlAsOlFwSH
ZBOSX+9Fq6vg8CUHi8obDgWAnIelTCCzHWC3Ep/ATLZQct82wgYZxIYTt4yCtMQo
R5c2DqKOYl8ZAjr3Zah5uwFI
-----END PRIVATE KEY-----

48
e2e/README.md
View File

@@ -1,48 +0,0 @@
# OPC Gateway end-to-end tests
End-to-end tests run every external dependency for real (MongoDB, Redis, PostgreSQL, MinIO and
NATS via testcontainers; OPC UA servers in process with asyncua) and build each gateway pod from
the environment exactly like `ingestor/app.py` does.
## Requirements
- Docker (for testcontainers), with the reaper disabled once per machine:
`echo "ryuk.disabled=true" >> ~/.testcontainers.properties` (see [scenarios.md](scenarios.md)).
- Python dev dependencies: `pip install -r requirements.txt -r requirements-dev.txt`
## Run locally
```bash
source ./venv/bin/activate
pytest e2e/ -o testpaths=e2e -m integration
```
One topic, or stop on first failure:
```bash
pytest e2e/test_writes.py -o testpaths=e2e -m integration
pytest e2e/ -o testpaths=e2e -m integration -x
```
The default `pytest` run only collects `tests/` (`testpaths`), so the E2E suite never runs there.
## Layout
| Path | Content |
|---|---|
| `conftest.py` | Containers (session), environment + state reset (per test), factories: `opc_servers`, `pods`, `put_slot`, `publish_command`, `client_certificate`, `opc_wire` |
| `helpers.py` | `Pod` (one gateway pod, `tick()` = one main loop iteration), `load_scenario_input`, `wait_until`/`stays_false`, Mongo/Postgres/Redis queries, DataValue mask |
| `opc_server.py` | In-process OPC UA server (`ns=2;s=<name>` variables, restart, secured mode, received writes) |
| `db_schema.sql` | `sientia_data.predictions` DDL, mirrored from production |
| `scenario_inputs/` | Slots (v2), connector and write command used by the scenarios |
| `test_*.py` | One file per topic, `test_scenario_<topic>_<group>_<n>_*` |
## Coverage (separate from unit tests)
```bash
COVERAGE_FILE=.coverage.e2e pytest e2e/ -o testpaths=e2e -m integration --cov=ingestor --cov-branch
```
## Scenario catalog
See [scenarios.md](scenarios.md) for the numbered scenarios and which test implements each one.

355
e2e/conftest.py
View File

@@ -1,355 +0,0 @@
"""Pytest configuration and fixtures for the gateway E2E tests.
Every external dependency runs for real: MongoDB, Redis, PostgreSQL, MinIO and NATS
via testcontainers (session-scoped), OPC UA servers in process (``e2e/opc_server.py``,
function-scoped). Gateway code under ``ingestor/**`` is not mocked: pods are built from
the environment exactly like ``ingestor/app.py`` builds them (``e2e/helpers.Pod``).
State is reset before every test.
"""
import io
import json
import os
from pathlib import Path
import nats
import pytest
import pytest_asyncio
from asyncua import ua
from asyncua.client import ua_client
from minio import Minio
from pymongo import MongoClient
from redis import Redis
from sqlalchemy import create_engine, text
from testcontainers.community.minio import MinioContainer
from testcontainers.community.mongodb import MongoDbContainer
from testcontainers.community.postgres import PostgresContainer
from testcontainers.community.redis import RedisContainer
from testcontainers.core.container import DockerContainer
from testcontainers.core.wait_strategies import LogMessageWaitStrategy
from e2e.helpers import CERT_BUCKET, Pod
from e2e.opc_server import OpcTestServer, free_port, make_certificate
DB_SCHEMA_SQL_PATH = Path(__file__).parent / 'db_schema.sql'
MONGO_USER = MONGO_PASSWORD = 'sientia'
MONGO_DATABASE = 'sientia_e2e'
WRITE_SUBJECT = 'opc.write'
# --------------------------------------------------------------------------------------
# Session infrastructure
# --------------------------------------------------------------------------------------
@pytest.fixture(scope='session')
def mongo_container():
"""MongoDB with root credentials: the gateway always connects as user:password@url."""
mongo = MongoDbContainer('mongo:7', username=MONGO_USER, password=MONGO_PASSWORD)
mongo.start()
yield mongo
mongo.stop()
@pytest.fixture(scope='session')
def redis_container():
redis = RedisContainer('redis:7-alpine')
redis.start()
yield redis
redis.stop()
@pytest.fixture(scope='session')
def postgres_container():
postgres = PostgresContainer(
'postgres:15', username='sientia', password='sientia', dbname='sientia'
)
postgres.start()
yield postgres
postgres.stop()
@pytest.fixture(scope='session')
def minio_container():
"""
MinIO holding the OPC client certificates. Same Bitnami "legacy" image as the
Laborious E2E suite and production: Docker Hub no longer serves ``minio/minio``.
"""
minio = MinioContainer(image='docker.io/bitnamilegacy/minio:2025.7.23-debian-12-r3')
minio.with_env('MINIO_ROOT_USER', minio.access_key)
minio.with_env('MINIO_ROOT_PASSWORD', minio.secret_key)
minio.with_kwargs(user='root')
minio.start()
yield minio
minio.stop()
NATS_IMAGE = 'nats:2.10-alpine'
def nats_server_container(host_port: int | None = None) -> DockerContainer:
"""NATS server; ``host_port`` pins the published port (to start it after the gateway)."""
container = DockerContainer(NATS_IMAGE).waiting_for(LogMessageWaitStrategy('Server is ready'))
if host_port is None:
return container.with_exposed_ports(4222)
return container.with_bind_ports(4222, host_port)
@pytest.fixture(scope='session')
def nats_container():
nats_server = nats_server_container()
nats_server.start()
yield nats_server
nats_server.stop()
@pytest.fixture(scope='session')
def mongo_client(mongo_container):
port = mongo_container.get_exposed_port(27017)
client = MongoClient(f'mongodb://{MONGO_USER}:{MONGO_PASSWORD}@localhost:{port}')
yield client
client.close()
@pytest.fixture(scope='session')
def postgres_engine(postgres_container):
engine = create_engine(postgres_container.get_connection_url())
yield engine
engine.dispose()
@pytest.fixture(scope='session')
def redis_client(redis_container):
client = Redis(
host=redis_container.get_container_host_ip(),
port=int(redis_container.get_exposed_port(6379)),
decode_responses=True,
)
yield client
client.close()
@pytest.fixture(scope='session')
def minio_client(minio_container):
return minio_container.get_client()
@pytest.fixture(scope='session')
def nats_url(nats_container) -> str:
host = nats_container.get_container_host_ip()
return f'nats://{host}:{nats_container.get_exposed_port(4222)}'
# --------------------------------------------------------------------------------------
# Per-test environment and state reset
# --------------------------------------------------------------------------------------
@pytest.fixture(autouse=True)
def gateway_env(
monkeypatch,
tmp_path,
mongo_container,
redis_container,
postgres_container,
minio_container,
nats_url,
):
"""Point every gateway setting at the containers, with the app's defaults otherwise."""
minio_cfg = minio_container.get_config()
env = {
'HOSTNAME': 'pod-a',
'REDIS_HOST': redis_container.get_container_host_ip(),
'REDIS_PORT': str(redis_container.get_exposed_port(6379)),
'LEASE_TTL': '10',
'HEARTBEAT_TTL': '20',
'POLL_INTERVAL': '1',
'MONGODB_URL': f'localhost:{mongo_container.get_exposed_port(27017)}',
'MONGODB_USERNAME': MONGO_USER,
'MONGODB_PASSWORD': MONGO_PASSWORD,
'MONGODB_DATABASE': MONGO_DATABASE,
'POSTGRES_HOST': postgres_container.get_container_host_ip(),
'POSTGRES_PORT': str(postgres_container.get_exposed_port(5432)),
'POSTGRES_USER': 'sientia',
'POSTGRES_PASSWORD': 'sientia',
'POSTGRES_DBNAME': 'sientia',
'POSTGRES_MIN_CONNECTIONS': '1',
'POSTGRES_MAX_CONNECTIONS': '2',
'MINIO_ENDPOINT': minio_cfg['endpoint'],
'MINIO_ACCESS_KEY': minio_cfg['access_key'],
'MINIO_SECRET_KEY': minio_cfg['secret_key'],
'MINIO_SECURE': 'false',
'NATS_URL': nats_url,
'NATS_WRITE_SUBJECT': WRITE_SUBJECT,
'NATS_QUEUE': 'gateway-writers',
'GRPC_ENABLED': 'false',
'GRPC_PORT': str(free_port()),
'CERT_CACHE_DIR': str(tmp_path / 'cert-cache'),
'WRITE_MIN_PAUSE_S': '0',
'IDLE_CONN_MAX_S': '300',
}
for name, value in env.items():
monkeypatch.setenv(name, value)
for name in ('REDIS_USERNAME', 'REDIS_PASSWORD', 'WRITE_TIMESTAMP', 'WRITE_QUALITY'):
monkeypatch.delenv(name, raising=False)
return env
@pytest.fixture(autouse=True)
def reset_state(redis_client, mongo_client, postgres_engine, minio_client):
"""Known-empty baseline: Redis flushed, Mongo database dropped, Postgres schema
recreated from ``e2e/db_schema.sql``, certificate bucket emptied."""
redis_client.flushdb()
mongo_client.drop_database(MONGO_DATABASE)
with postgres_engine.begin() as conn:
conn.execute(text('DROP SCHEMA IF EXISTS sientia_data CASCADE'))
conn.exec_driver_sql(DB_SCHEMA_SQL_PATH.read_text(encoding='utf-8'))
if minio_client.bucket_exists(CERT_BUCKET):
for obj in minio_client.list_objects(CERT_BUCKET, recursive=True):
minio_client.remove_object(CERT_BUCKET, obj.object_name)
else:
minio_client.make_bucket(CERT_BUCKET)
yield
@pytest.fixture
def mongo_db(mongo_client):
return mongo_client[MONGO_DATABASE]
# --------------------------------------------------------------------------------------
# Factories: OPC servers, pods, slots, certificates, NATS commands
# --------------------------------------------------------------------------------------
@pytest_asyncio.fixture
async def opc_servers(tmp_path):
"""
Factory of started in-process OPC servers.
``await opc_servers('plant-a', {'Temperature': 20.0}, secured=False)`` declares the
variables (``{name: value}`` or ``{name: (value, data_type, writable)}``) and starts it.
"""
started: dict[str, OpcTestServer] = {}
async def start(name: str, variables: dict, secured: bool = False) -> OpcTestServer:
server = OpcTestServer(name, cert_dir=tmp_path / 'server-certs', secured=secured)
for var, spec in variables.items():
value, data_type, writable = spec if isinstance(spec, tuple) else (spec, 'float', True)
server.add_variable(var, value, data_type, writable)
await server.start()
started[name] = server
return server
yield start
for server in started.values():
await server.stop()
@pytest_asyncio.fixture
async def pods(monkeypatch):
"""
Factory of started gateway pods: ``await pods('pod-a', runtime=True, **env)``.
Extra keyword arguments are environment overrides applied before the pod is built
(e.g. ``WRITE_TIMESTAMP='false'``). Every pod is stopped at teardown.
"""
started: list[Pod] = []
async def start(name: str = 'pod-a', runtime: bool = True, **env: str) -> Pod:
for key, value in env.items():
monkeypatch.setenv(key, value)
pod = Pod(name, runtime=runtime)
started.append(pod)
return await pod.start()
yield start
for pod in reversed(started):
await pod.stop()
@pytest.fixture
def put_slot(redis_client):
"""Write a slot v2 to Redis (``slot:opc_tags:<n>``), as the Orchestrator does."""
def put(number: int, slot: dict) -> None:
redis_client.set(f'slot:opc_tags:{number}', json.dumps(slot))
return put
@pytest_asyncio.fixture
async def publish_command(nats_url):
"""Publish C2 write commands (dicts) or raw payloads (bytes) on the write subject."""
client = await nats.connect(nats_url)
async def publish(payload) -> None:
data = payload if isinstance(payload, bytes) else json.dumps(payload).encode()
await client.publish(WRITE_SUBJECT, data)
await client.flush()
yield publish
await client.drain()
@pytest_asyncio.fixture
async def client_certificate(tmp_path, minio_client: Minio, mongo_db):
"""
Create an OPC client certificate the way the platform stores it: cert (DER) and
private key (PEM) uploaded to MinIO, and an ``opc_certificates`` document (active,
kind client). Returns the certificate in the slot format (contract C1).
"""
async def create(
certificate_id: int,
upload: bool = True,
status: str = 'active',
app_uri: str | None = None,
) -> dict:
app_uri = app_uri or f'urn:sientia:e2e:client:{certificate_id}'
cert, key = await make_certificate(
tmp_path / 'client-certs', f'client-{certificate_id}', app_uri, server=False
)
objects = {
'cert': {'bucket': CERT_BUCKET, 'object_name': f'{certificate_id}/client.der'},
'private_key': {'bucket': CERT_BUCKET, 'object_name': f'{certificate_id}/client.pem'},
}
if upload:
for path, field in ((cert, 'cert'), (key, 'private_key')):
data = path.read_bytes()
minio_client.put_object(
CERT_BUCKET, objects[field]['object_name'], io.BytesIO(data), len(data)
)
mongo_db['opc_certificates'].insert_one(
{
'certificate_id': certificate_id,
'kind': 'client',
'status': status,
'application_uri': app_uri,
**objects,
}
)
return {'certificate_id': certificate_id, 'application_uri': app_uri, **objects}
return create
@pytest.fixture
def cert_cache_dir() -> Path:
return Path(os.environ['CERT_CACHE_DIR'])
@pytest.fixture
def opc_wire(monkeypatch):
"""Serialized WriteRequests, exactly as the gateway's OPC client sends them."""
sent: list[bytes] = []
original = ua_client.struct_to_binary
def capture(obj):
data = original(obj)
if isinstance(obj, ua.WriteRequest):
sent.append(data)
return data
monkeypatch.setattr(ua_client, 'struct_to_binary', capture)
return sent

33
e2e/db_schema.sql
View File

@@ -1,33 +0,0 @@
-- =============================================================================
-- E2E test database schema for the ``sientia_data`` namespace.
--
-- Mirrors the production DDL one-to-one so any change in production can be
-- pasted directly into this file. The conftest fixture drops the schema and
-- loads this SQL into the testcontainers Postgres before every test.
--
-- The gateway only touches ``sientia_data.predictions``: it updates
-- ``write_status`` (contract C3, SIENTIAPDE-1975) on the row the Laborious
-- inserted. Same DDL as ``sientia-dataops-laborious/e2e/db_schema.sql``.
-- =============================================================================
CREATE SCHEMA IF NOT EXISTS sientia_data;
-- -----------------------------------------------------------------------------
-- sientia_data.predictions
-- -----------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS sientia_data.predictions (
model_id int4 NOT NULL,
prediction numeric NULL,
prediction_confidence numeric NOT NULL,
response_time numeric NOT NULL,
prediction_status text NOT NULL,
"timestamp" timestamptz NOT NULL,
created_at timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
"comments" text NULL,
metadata_id varchar(24) NULL,
write_status text NULL,
CONSTRAINT unique_model_id_timestamp
UNIQUE (model_id, "timestamp"),
CONSTRAINT chk_predictions_write_status
CHECK (write_status IN ('pending', 'ok', 'error', 'session_bad'))
);

243
e2e/helpers.py
View File

@@ -1,243 +0,0 @@
"""Shared helpers for the gateway E2E tests (OPC UA + Redis + MongoDB + Postgres + MinIO + NATS)."""
import asyncio
import copy
import inspect
import json
import os
import time
from collections.abc import Awaitable, Callable
from pathlib import Path
from typing import Any
from asyncua import ua
from asyncua.ua.ua_binary import Primitives, nodeid_to_binary
from pymongo.database import Database
from sqlalchemy import text
from sqlalchemy.engine import Engine
from ingestor.config import GatewayConfig
from ingestor.gateway_runtime import GatewayRuntime
from ingestor.ingestor import Ingestor
SCENARIO_INPUTS_DIR = Path(__file__).parent / 'scenario_inputs'
# Collections the gateway reads (connectors, opc_certificates) or writes
# (readings, notification_queue). Dropped before every test with the whole database.
NOTIFICATIONS = 'notification_queue'
CERT_BUCKET = 'opc-certificates'
PREDICTIONS_TABLE = 'predictions'
def _resolve_markers(value: Any, servers: dict) -> Any:
"""Replace ``@server:<name>`` with the URL of that in-process OPC server."""
if isinstance(value, dict):
return {k: _resolve_markers(v, servers) for k, v in value.items()}
if isinstance(value, list):
return [_resolve_markers(v, servers) for v in value]
if isinstance(value, str) and value.startswith('@server:'):
return servers[value.removeprefix('@server:')].url
return value
def load_scenario_input(name: str, servers: dict | None = None, **overrides: Any) -> Any:
"""
Load a JSON scenario input from ``e2e/scenario_inputs``.
Args:
name: File name (e.g. ``slot_single_server.json``).
servers: ``{name: OpcTestServer}`` used to resolve ``@server:<name>`` markers.
**overrides: Top-level keys replaced in the loaded dict.
Return:
Any: The resolved JSON content.
"""
data = json.loads((SCENARIO_INPUTS_DIR / name).read_text(encoding='utf-8'))
data = _resolve_markers(data, servers or {})
if overrides:
data = {**data, **copy.deepcopy(overrides)}
return data
async def wait_until(
predicate: Callable[[], Any | Awaitable[Any]], timeout: float = 15.0, interval: float = 0.1
) -> Any:
"""
Poll ``predicate`` (sync or async) until it returns a truthy value.
Sleeps on the event loop between attempts, so OPC subscription callbacks and NATS
handlers keep running while the test waits.
Raises:
AssertionError: When the predicate is still falsy after ``timeout`` seconds.
"""
deadline = time.monotonic() + timeout
while True:
result = predicate()
if inspect.isawaitable(result):
result = await result
if result:
return result
if time.monotonic() > deadline:
raise AssertionError(f'condition not met within {timeout}s')
await asyncio.sleep(interval)
async def stays_false(
predicate: Callable[[], Any | Awaitable[Any]], duration: float = 1.5, interval: float = 0.1
) -> bool:
"""True when ``predicate`` never becomes truthy during ``duration`` seconds."""
deadline = time.monotonic() + duration
while time.monotonic() < deadline:
result = predicate()
if inspect.isawaitable(result):
result = await result
if result:
return False
await asyncio.sleep(interval)
return True
class Pod:
"""
One gateway pod, wired exactly as ``ingestor/app.py`` does (minus the Prometheus
HTTP server and the sleep between iterations): ``Ingestor.prepare_ingestor``, then
``GatewayRuntime`` on its data manager and connection pool. ``tick`` runs one main
loop iteration (``Ingestor.loop`` + ``GatewayRuntime.idle_sweep``).
Every component reads its settings from the environment at construction time, so
the caller sets the env (``HOSTNAME`` included) before ``start``.
"""
def __init__(self, name: str, runtime: bool = True):
self.name = name
self.with_runtime = runtime
self.ingestor: Ingestor | None = None
self.runtime: GatewayRuntime | None = None
async def start(self) -> 'Pod':
os.environ['HOSTNAME'] = self.name
self.ingestor = Ingestor()
await self.ingestor.prepare_ingestor()
if self.with_runtime:
manager = self.ingestor.ingestor_manager
assert manager is not None
self.runtime = GatewayRuntime(
config=GatewayConfig.from_env(),
data_manager=manager.data_manager,
pool=manager.pool,
logger=self.ingestor.logger,
notification_handler=self.ingestor.notification_handler,
metrics_controller=self.ingestor.metrics_controller,
metadata=self.ingestor.metadata,
)
await self.runtime.start()
consumer = self.runtime.nats_consumer
if consumer.nc is not None:
# PING/PONG: the server has processed the SUB, so a command published
# right after start is not lost for lack of a subscriber.
await consumer.nc.flush()
return self
@property
def manager(self):
assert self.ingestor is not None and self.ingestor.ingestor_manager is not None
return self.ingestor.ingestor_manager
@property
def slots(self) -> set[str]:
"""Slots this pod currently manages."""
return set(self.manager.managed_tags)
async def tick(self, times: int = 1) -> None:
assert self.ingestor is not None
for _ in range(times):
await self.ingestor.loop()
if self.runtime is not None:
await self.runtime.idle_sweep()
async def stop(self) -> None:
if self.runtime is not None:
await self.runtime.stop()
self.runtime = None
if self.ingestor is not None:
await self.ingestor.shutdown()
self.ingestor.notification_handler.shutdown()
self.ingestor = None
def redis_json(redis_client, key: str) -> Any:
"""Value stored by the gateway's RedisRepository (JSON-encoded), or None."""
raw = redis_client.get(key)
return None if raw is None else json.loads(raw)
def readings(db: Database, collection: str, **query: Any) -> list[dict]:
"""Documents the read path persisted into ``collection`` (oldest first)."""
return list(db[collection].find(query, {'_id': 0}).sort('inserted_at', 1))
def notifications(db: Database, prefix: str = '') -> list[dict]:
"""Notifications sent by the gateway whose ``notification_id`` starts with ``prefix``."""
return [
n
for n in db[NOTIFICATIONS].find({}, {'_id': 0})
if str(n.get('notification_id', '')).startswith(prefix)
]
def insert_prediction(engine: Engine, model_id: int, timestamp: str, write_status: str | None):
"""One ``sientia_data.predictions`` row, as the Laborious inserts it before publishing C2."""
with engine.begin() as conn:
conn.execute(
text(
'INSERT INTO sientia_data.predictions (model_id, prediction, '
'prediction_confidence, response_time, prediction_status, "timestamp", '
'write_status) VALUES (:model_id, 1.5, 0.9, 0.1, :status, :ts, :write_status)'
),
{
'model_id': model_id,
'status': 'success',
'ts': timestamp,
'write_status': write_status,
},
)
def write_status(engine: Engine, model_id: int, timestamp: str) -> str | None:
"""``write_status`` of a prediction row (``'<missing>'`` when the row does not exist)."""
with engine.begin() as conn:
row = conn.execute(
text(
'SELECT write_status FROM sientia_data.predictions '
'WHERE model_id = :model_id AND "timestamp" = :ts'
),
{'model_id': model_id, 'ts': timestamp},
).first()
return '<missing>' if row is None else row[0]
def prediction_count(engine: Engine) -> int:
with engine.begin() as conn:
return conn.execute(text('SELECT count(*) FROM sientia_data.predictions')).scalar_one()
# DataValue encoding mask bits (OPC UA Part 6, 5.2.2.17).
STATUS_CODE_BIT = 0x02
SOURCE_TIMESTAMP_BIT = 0x04
def datavalue_mask(request: bytes, addr: str) -> int:
"""
Encoding mask of the DataValue written to ``addr`` in a serialized WriteRequest.
An omitted StatusCode decodes as Good on the server (OPC UA default), so whether
the gateway sent the quality is only visible on the wire. WriteValue = NodeId,
AttributeId, IndexRange, DataValue: the byte right after the first three is the mask.
"""
prefix = (
nodeid_to_binary(ua.NodeId.from_string(addr))
+ Primitives.UInt32.pack(ua.AttributeIds.Value)
+ Primitives.String.pack(None)
)
return request[request.index(prefix) + len(prefix)]

134
e2e/opc_server.py
View File

@@ -1,134 +0,0 @@
"""In-process OPC UA servers for the E2E suite (asyncua `Server`).
Each `OpcTestServer` listens on its own localhost port and exposes string NodeIds
under namespace 2 (`ns=2;s=<name>`), so a slot can address them the same way the
Orchestrator addresses a real server. Writes received from the gateway are recorded
(decoded `WriteValue`s) through the server's PreWrite callback.
"""
import socket
import time
from pathlib import Path
from asyncua import Server, ua
from asyncua.common.callback import CallbackType
from asyncua.crypto.cert_gen import setup_self_signed_certificate
from cryptography.x509.oid import ExtendedKeyUsageOID
NAMESPACE_URI = 'urn:sientia:e2e'
_VARIANT_TYPES = {
'float': ua.VariantType.Float,
'double': ua.VariantType.Double,
'int': ua.VariantType.Int32,
'bool': ua.VariantType.Boolean,
'string': ua.VariantType.String,
}
def free_port() -> int:
with socket.socket() as s:
s.bind(('127.0.0.1', 0))
return s.getsockname()[1]
def address(name: str) -> str:
"""NodeId string of a variable created by `OpcTestServer.add_variable`."""
return f'ns=2;s={name}'
async def make_certificate(
folder: Path, name: str, app_uri: str, server: bool
) -> tuple[Path, Path]:
"""Self-signed application certificate (DER) + private key (PEM) with `app_uri` as
the SAN URI, as asyncua expects from both ends of a secure channel."""
folder.mkdir(parents=True, exist_ok=True)
cert, key = folder / f'{name}.der', folder / f'{name}.pem'
usage = ExtendedKeyUsageOID.SERVER_AUTH if server else ExtendedKeyUsageOID.CLIENT_AUTH
await setup_self_signed_certificate(
key, cert, app_uri, 'localhost', [usage], {'commonName': name}
)
return cert, key
class OpcTestServer:
"""One asyncua server. `secured=True` exposes only Basic256Sha256 SignAndEncrypt."""
def __init__(self, name: str, cert_dir: Path | None = None, secured: bool = False):
self.name = name
self.port = free_port()
self.secured = secured
self.cert_dir = cert_dir
self.server: Server | None = None
self.variables: dict[str, tuple[object, str, bool]] = {}
self.writes: list[ua.WriteValue] = []
self.write_times: list[float] = [] # time.monotonic() of each received write
@property
def url(self) -> str:
return f'opc.tcp://127.0.0.1:{self.port}'
def add_variable(
self, name: str, value: object, data_type: str = 'float', writable: bool = True
) -> str:
"""Declare a variable (created on `start`, and again on every restart)."""
self.variables[name] = (value, data_type, writable)
return address(name)
async def start(self) -> None:
server = Server()
await server.init()
server.set_endpoint(self.url)
server.set_server_name(self.name)
if self.secured:
assert self.cert_dir is not None, 'a secured server needs a cert_dir'
cert, key = await make_certificate(
self.cert_dir, f'server-{self.name}', f'urn:sientia:e2e:{self.name}', True
)
await server.load_certificate(str(cert))
await server.load_private_key(str(key))
server.set_security_policy([ua.SecurityPolicyType.Basic256Sha256_SignAndEncrypt])
idx = await server.register_namespace(NAMESPACE_URI)
assert idx == 2, f'unexpected namespace index {idx}'
for name, (value, data_type, writable) in self.variables.items():
node = await server.nodes.objects.add_variable(
address(name), name, value, _VARIANT_TYPES[data_type]
)
if writable:
await node.set_writable()
server.subscribe_server_callback(CallbackType.PreWrite, self._record_write)
await server.start()
self.server = server
async def stop(self) -> None:
if self.server is not None:
await self.server.stop()
self.server = None
async def restart(self) -> None:
"""Stop and start again on the same port (values reset to their initial ones)."""
await self.stop()
await self.start()
async def set(self, name: str, value: object) -> None:
"""Change a value server-side (raises a data change for subscribed clients)."""
assert self.server is not None
_, data_type, _ = self.variables[name]
node = self.server.get_node(address(name))
await node.write_value(ua.Variant(value, _VARIANT_TYPES[data_type]))
async def read(self, name: str) -> ua.DataValue:
assert self.server is not None
return await self.server.get_node(address(name)).read_data_value()
def session_count(self) -> int:
"""Client sessions currently open on this server."""
assert self.server is not None
return len(self.server.iserver._external_sessions)
async def _record_write(self, event, _dispatcher) -> None:
if not event.is_external: # the server updates its own nodes (ServerStatus, ...)
return
for value in event.request_params.NodesToWrite:
self.writes.append(value)
self.write_times.append(time.monotonic())

10
e2e/scenario_inputs/connector_plant_a.json
View File

@@ -1,10 +0,0 @@
{
"connector_id": "1",
"type": "opc_ua",
"name": "plant-a",
"url": "@server:plant-a",
"status": "Active",
"subscription_period_ms": 100,
"security": {"mode": "None", "policy": "None", "client_certificate_id": null},
"legacy_opc_server_id": "1"
}

25
e2e/scenario_inputs/slot_active_read.json
View File

@@ -1,25 +0,0 @@
{
"plant-a": {
"name": "plant-a",
"server_id": "1",
"url": "@server:plant-a",
"subscription_period_ms": 100,
"security": {"mode": "None", "policy": "None"},
"certificate": null,
"tags": {
"ns=2;s=Level": {
"frequency": 1000,
"active_read": {"enabled": true, "cycle_multiple": 1},
"names": [
{"collection": "active_tank", "tag_name": "tank_level", "model_id": "30"},
{"collection": "active_mixer", "tag_name": "mixer_inlet_level", "model_id": "40"}
]
},
"ns=2;s=Temperature": {
"frequency": 1000,
"active_read": {"enabled": false},
"names": [{"collection": "active_tank", "tag_name": "tank_temperature", "model_id": "30"}]
}
}
}
}

28
e2e/scenario_inputs/slot_data_types.json
View File

@@ -1,28 +0,0 @@
{
"plant-a": {
"name": "plant-a",
"server_id": "1",
"url": "@server:plant-a",
"subscription_period_ms": 100,
"security": {"mode": "None", "policy": "None"},
"certificate": null,
"tags": {
"ns=2;s=Temperature": {
"frequency": 1000,
"names": [{"collection": "raw_types", "tag_name": "temperature", "model_id": "10"}]
},
"ns=2;s=Counter": {
"frequency": 1000,
"names": [{"collection": "raw_types", "tag_name": "counter", "model_id": "10"}]
},
"ns=2;s=Running": {
"frequency": 1000,
"names": [{"collection": "raw_types", "tag_name": "running", "model_id": "10"}]
},
"ns=2;s=Batch": {
"frequency": 1000,
"names": [{"collection": "raw_types", "tag_name": "batch", "model_id": "10"}]
}
}
}
}

20
e2e/scenario_inputs/slot_multi_name.json
View File

@@ -1,20 +0,0 @@
{
"plant-a": {
"name": "plant-a",
"server_id": "1",
"url": "@server:plant-a",
"subscription_period_ms": 100,
"security": {"mode": "None", "policy": "None"},
"certificate": null,
"tags": {
"ns=2;s=Pressure": {
"frequency": 1000,
"active_read": {"enabled": false},
"names": [
{"collection": "raw_furnace", "tag_name": "furnace_pressure", "model_id": "10", "model_name": "furnace"},
{"collection": "raw_boiler", "tag_name": "boiler_inlet_pressure", "model_id": "20", "model_name": "boiler"}
]
}
}
}
}

19
e2e/scenario_inputs/slot_secured.json
View File

@@ -1,19 +0,0 @@
{
"plant-s": {
"name": "plant-s",
"server_id": "7",
"url": "@server:plant-s",
"subscription_period_ms": 100,
"security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256"},
"certificate": null,
"tags": {
"ns=2;s=Temperature": {
"frequency": 1000,
"active_read": {"enabled": false},
"names": [
{"collection": "raw_secured", "tag_name": "secured_temperature", "model_id": "70"}
]
}
}
}
}

19
e2e/scenario_inputs/slot_single_server.json
View File

@@ -1,19 +0,0 @@
{
"plant-a": {
"name": "plant-a",
"server_id": "1",
"url": "@server:plant-a",
"subscription_period_ms": 100,
"security": {"mode": "None", "policy": "None"},
"certificate": null,
"tags": {
"ns=2;s=Temperature": {
"frequency": 1000,
"active_read": {"enabled": false},
"names": [
{"collection": "raw_plant_a", "tag_name": "temperature", "model_id": "10", "model_name": "furnace"}
]
}
}
}
}

36
e2e/scenario_inputs/slot_two_servers.json
View File

@@ -1,36 +0,0 @@
{
"plant-a": {
"name": "plant-a",
"server_id": "1",
"url": "@server:plant-a",
"subscription_period_ms": 100,
"security": {"mode": "None", "policy": "None"},
"certificate": null,
"tags": {
"ns=2;s=Temperature": {
"frequency": 1000,
"active_read": {"enabled": false},
"names": [
{"collection": "raw_plant_a", "tag_name": "temperature", "model_id": "10", "model_name": "furnace"}
]
}
}
},
"plant-b": {
"name": "plant-b",
"server_id": "2",
"url": "@server:plant-b",
"subscription_period_ms": 100,
"security": {"mode": "None", "policy": "None"},
"certificate": null,
"tags": {
"ns=2;s=Flow": {
"frequency": 1000,
"active_read": {"enabled": false},
"names": [
{"collection": "raw_plant_b", "tag_name": "flow", "model_id": "20", "model_name": "boiler"}
]
}
}
}
}

10
e2e/scenario_inputs/slot_v1_invalid.json
View File

@@ -1,10 +0,0 @@
{
"plant-a": {
"name": "plant-a",
"url": "@server:plant-a",
"server_uri": "http://opcua-server.simulator",
"tags": {
"ns=2;s=Temperature": {"tag_name": "Temperature", "frequency": 1000, "topics": ["raw_plant_a"]}
}
}
}

12
e2e/scenario_inputs/write_command.json
View File

@@ -1,12 +0,0 @@
{
"server_id": "1",
"addr": "ns=2;s=Setpoint",
"value": 1.25,
"data_type": "float",
"tag_type": "prediction",
"model_id": "2",
"model_name": "sin-approx",
"schema": "sientia_data",
"table_name": "predictions",
"timestamp": "2026-10-13T12:00:00-03:00"
}

584
e2e/scenarios.md
View File

@@ -1,584 +0,0 @@
# E2E Scenario Documentation — OPC Gateway
Functional reference for the gateway end-to-end scenarios: slot leasing, subscription reads,
NATS writes, gRPC reads, active reads, certificates and the pod lifecycle. Tests live under
`e2e/`, are marked with `@pytest.mark.integration` and run with:
```bash
pytest e2e/ -o testpaths=e2e -m integration
```
**Every section below maps 1:1 to an actual test function** (or to one `pytest.param` id of a
parametrized test) — this file describes what the suite verifies today. Adding a scenario here
without a matching test (or vice versa) is a doc bug; fix the mismatch instead of leaving it
for later.
## Execution Context
### One-time setup: disable the testcontainers reaper
Same as the Laborious and Orchestrator suites. Once per machine:
```bash
echo "ryuk.disabled=true" >> ~/.testcontainers.properties
```
Under Docker Desktop on Linux the reaper ("Ryuk") cannot mount the Docker socket and every
test errors at setup (`ERROR`, not `FAILED`). Every fixture stops its own containers in
teardown; after a killed run, clean up by hand:
```bash
docker ps -a --filter "label=org.testcontainers=true" -q | xargs -r docker rm -f
```
### What runs for real
| Dependency | How | Scope |
|---|---|---|
| MongoDB (`mongo:7`, user/password) | testcontainers | session |
| Redis (`redis:7-alpine`) | testcontainers | session |
| PostgreSQL (`postgres:15`, `e2e/db_schema.sql`) | testcontainers | session |
| MinIO (`bitnamilegacy/minio`, client certificates) | testcontainers | session |
| NATS (`nats:2.10-alpine`) | testcontainers | session |
| OPC UA servers | in process (`e2e/opc_server.py`, asyncua `Server`) | per test |
- Gateway code under `ingestor/**` is not mocked. A pod (`e2e/helpers.Pod`) is built from the
environment exactly like `ingestor/app.py` builds it — `Ingestor.prepare_ingestor`, then
`GatewayRuntime` on the same data manager and connection pool — minus the Prometheus HTTP
server. `Pod.tick()` runs one main loop iteration (`Ingestor.loop` + `GatewayRuntime.idle_sweep`)
without the `POLL_INTERVAL` sleep, so every scenario controls exactly how many iterations run.
- The environment (`gateway_env` in `e2e/conftest.py`) points every setting at the containers;
scenarios override single variables per pod (`LEASE_TTL`, `WRITE_TIMESTAMP`, ...).
- State is reset before every test: Redis `FLUSHDB`, Mongo database dropped, Postgres schema
recreated from `e2e/db_schema.sql`, certificate bucket emptied.
- Inputs live in `e2e/scenario_inputs/*.json` and are loaded with `load_scenario_input`;
`@server:<name>` markers resolve to the URL of that test's in-process OPC server.
- Default `POLL_INTERVAL=1` and tag `frequency=1000`: one OPC "cycle" per loop iteration.
- Not covered by the default `pytest` run (`testpaths = ["tests"]`) nor by the unit coverage.
`--strict-markers` is on: a test without `@pytest.mark.integration` is not collected, and a run
that collects nothing still exits 0 — check the collected count.
---
## 1. Slot Leasing and Load Balancing
Source: `e2e/test_slot_leasing.py`
### 1.1.1 Single Pod Leases the Slot and Heartbeats
**Summary**: A lone pod takes the only slot on its first loop iteration.
**Description**:
- Slot 1 reads `plant-a`; one pod starts and runs one iteration.
**Expected Outcome**:
- The pod manages slot `1`; `lease:opc_tags:1 = "pod-a"` with TTL ≤ `LEASE_TTL` (10 s).
- `heartbeat:ingestor:pod-a = 1` with TTL ≤ `HEARTBEAT_TTL` (20 s).
- Readings of the slot reach MongoDB.
### 1.1.2 Idle Pod Takes a Slot Created After Startup
**Summary**: A pod without slots picks up a slot the Orchestrator creates later.
**Description**:
- The pod starts with Redis empty and runs one iteration (nothing to lease).
- Slot 1 is written; the pod runs another iteration.
**Expected Outcome**:
- The pod manages slot `1`, holds its lease and its readings reach MongoDB.
### 1.2.1 Two Pods Split Two Slots
**Summary**: The rebalancing hands one slot to each pod.
**Description**:
- Slot 1 reads `plant-a`, slot 2 reads `plant-b`. `pod-a` starts alone, then `pod-b` starts.
- Iterations: `pod-a`, `pod-b`, `pod-a`, `pod-b`.
**Expected Outcome**:
- Alone, `pod-a` takes both slots; `pod-b` finds them leased.
- `pod-a` sees a pod without slots and gives slot `2` back (unsubscribing first); `pod-b` takes it.
- Leases: slot 1 → `pod-a`, slot 2 → `pod-b`; each pod only has the reader of its own server.
- New values of both servers reach MongoDB.
### 1.2.2 Extra Pod Stays Idle
**Summary**: With more pods than slots, no slot is read twice.
**Description**:
- Two slots balanced between `pod-a` and `pod-b`; `pod-c` joins and every pod iterates twice.
**Expected Outcome**:
- `pod-c` manages nothing; `pod-a` and `pod-b` hold disjoint slots covering `{1, 2}`.
- Leases belong to `pod-a` and `pod-b` only.
### 1.2.3 Failover After Lease Expiry
**Summary**: A slot left by a dead pod is taken over once its lease expires.
**Description**:
- `pod-a` (`LEASE_TTL=2`) reads slot 1; `pod-b` runs idle.
- `pod-a` stops without releasing the lease (crash-like).
**Expected Outcome**:
- The lease still names `pod-a` right after the stop, then expires.
- On its next iteration `pod-b` leases slot `1` and new values reach MongoDB.
### 1.3.1 Deleted Slot Stops the Reads
**Summary**: Removing a slot from Redis unsubscribes it.
**Description**:
- A pod reads slot 1; the slot key is deleted; the pod iterates.
**Expected Outcome**:
- The pod manages no slot and has no OPC reader.
- A value changed afterwards is not persisted.
### 1.3.2 Invalid Slot Is Notified and Not Read
**Summary**: A slot v1 (`topics`, no `security`/`names`) is rejected.
**Description**:
- Slot 1 holds the legacy v1 shape (`slot_v1_invalid.json`).
**Expected Outcome**:
- Notification `OPC_SLOT_CONFIG_ERROR_1` in `notification_queue`.
- The pod manages nothing; no reading is persisted.
### 1.3.3 Fixed Slot Is Read Once Its Lease Expires
**Summary**: Fixing an invalid slot takes effect after the lease of the failed acquisition expires.
**Description**:
- `pod-a` (`LEASE_TTL=2`) leases the invalid slot 1 (the lease is kept, the slot is not managed).
- The slot is fixed; the pod iterates before and after the lease expires.
**Expected Outcome**:
- Before expiry the pod still manages nothing (its own lease blocks the re-acquisition).
- After expiry the pod manages slot `1` and its readings reach MongoDB.
---
## 2. Subscription Reads (OPC UA → MongoDB)
Source: `e2e/test_subscription_reads.py`
### 2.1.1 Value Change Is Persisted
**Summary**: Each data change becomes one document in the name's collection.
**Description**:
- Slot 1 subscribes `ns=2;s=Temperature` of `plant-a` into `raw_plant_a`; the value changes.
**Expected Outcome**:
- The initial value and the new one are persisted.
- Document fields: `tag` (address), `name` (`tag_name`), `model_id`, `value`, `timestamp`
(source timestamp, `YYYY-MM-DD HH:MM:SS-0300`) and `inserted_at` — nothing else.
### 2.1.2 One Address Fans Out to Every Name
**Summary**: An address with several `names` is written to each collection.
**Description**:
- `ns=2;s=Pressure` has two names: `raw_furnace`/`furnace_pressure`/model 10 and
`raw_boiler`/`boiler_inlet_pressure`/model 20 (`slot_multi_name.json`).
**Expected Outcome**:
- One document per collection for the change, each with its own `name`/`model_id`, the same
`tag` and the same `timestamp`.
### 2.1.3 Two Servers in One Slot
**Summary**: A slot can hold several OPC servers.
**Description**:
- Slot 1 holds `plant-a` and `plant-b` (`slot_two_servers.json`); both values change.
**Expected Outcome**:
- Both changes are persisted; the pod has one reader per server.
### 2.1.4 Values Keep Their OPC Type
**Summary**: Double, Int32, Boolean and String values are stored with their native type.
**Description**:
- Four tags of different OPC types change value (`slot_data_types.json`).
**Expected Outcome**:
- Each document stores the value with the matching Python/BSON type (`float`, `int`, `bool`, `str`).
### 2.2.1 Tag Added to the Slot Is Subscribed
**Summary**: A new tag in the slot is subscribed on the next iteration.
**Description**:
- The Orchestrator adds `ns=2;s=Level` to slot 1; the pod iterates.
**Expected Outcome**:
- Changes of the new tag and of the existing one are persisted.
### 2.2.2 Tag Removed from the Slot Is No Longer Read
**Summary**: A tag removed from the slot stops being persisted.
**Description**:
- Slot 1 reads `Temperature` and `Level`; `Level` is removed; the pod iterates.
**Expected Outcome**:
- `Temperature` keeps being persisted; a new `Level` value is not.
### 2.2.3 Name Added to a Tag Receives Its Readings
**Summary**: A new name on an existing tag starts receiving the readings.
**Description**:
- A second name (`raw_dryer`/`dryer_temperature`/model 50) is added to `Temperature`.
**Expected Outcome**:
- The next change is persisted in both collections.
### 2.2.4 Server URL Change Reconnects to the New Server
**Summary**: Changing the connection settings of a server re-initializes its reader.
**Description**:
- Slot 1 points `plant-a` to a second server (same tags) and the pod iterates.
**Expected Outcome**:
- Values of the new server are persisted; the session on the old server is closed.
- Changes on the old server are no longer persisted.
### 2.3.1 Unreachable Server Does Not Stop the Others
**Summary**: A server that cannot be reached is dropped; the other servers of the slot keep reading.
**Description**:
- Slot 1 holds `plant-a` and `plant-b`, whose URL has nothing listening.
**Expected Outcome**:
- Notification `OPC_CONNECTION_ERROR_plant-b`; the pod only has the `plant-a` reader.
- `plant-a` values are persisted.
### 2.3.2 Tag Without Data Raises a Warning
**Summary**: A tag without data changes for 5 cycles is notified.
**Description**:
- The value does not change for 5 iterations.
**Expected Outcome**:
- One WARNING notification `TAG_ns=2;s=Temperature:temperature_LISTENNING_STOPPED`.
### 2.3.3 Silent Server Is Reconnected After 15 Cycles
**Summary**: A server without any data change for 15 cycles is treated as silently dead.
**Description**:
- The value does not change for 15 iterations, then the pod iterates once more.
**Expected Outcome**:
- Notifications `OPC_LISTENNING_STOPPED__1` and `OPC_CONNECTION_RETRY__1`; the reader is dropped
(pool connection evicted).
- The next iteration re-initializes the reader with a new connection; changes are persisted again.
### 2.3.4 Server Restart Resumes the Reads
**Summary**: A server that goes down and comes back is read again.
**Description**:
- The server stops; the pod iterates 15 times (reader dropped) and once more while it is down;
the server starts again on the same port; the pod iterates twice.
**Expected Outcome**:
- While down, the reconnection fails with notification `OPC_CONNECTION_ERROR_plant-a`.
- Once back, the reader is re-initialized and changes are persisted again.
---
## 3. Writes (NATS C2 → OPC UA → Postgres `write_status` C3)
Source: `e2e/test_writes.py`
Common setup: `plant-a` with a writable `Setpoint`, its connector in `connectors`
(`connector_plant_a.json`, `connector_id "1"`) and a `pending` row in `sientia_data.predictions`
(model 2, `2026-10-13T12:00:00-03:00`). Commands come from `write_command.json` and are published
on `opc.write`; the gateway answers through the row's `write_status`.
### 3.1 DataValue Parts: Timestamp and Quality (SIENTIAPDE-2202)
**Summary**: The value is always written; the prediction timestamp (`SourceTimestamp`) and a Good
`StatusCode` are written unless turned off. Each flag comes from the command, else from the env
(`WRITE_TIMESTAMP` / `WRITE_QUALITY`), else `true`.
**Description**:
- `test_scenario_3_1_datavalue_parts`, one case per id. The timestamp is checked on the value the
server stored; the quality on the DataValue encoding mask of the WriteRequest the gateway sent
(an omitted StatusCode decodes as Good on the server, so only the wire tells them apart).
| Id | Env | Command flags | SourceTimestamp | StatusCode |
|---|---|---|---|---|
| `3_1_1_defaults_on` | — | — | sent | sent |
| `3_1_2_timestamp_off` | `WRITE_TIMESTAMP=false` | — | not sent | sent |
| `3_1_3_quality_off` | `WRITE_QUALITY=false` | — | sent | not sent |
| `3_1_4_both_off` | both `false` | — | not sent | not sent |
| `3_1_5_command_overrides_env` | `WRITE_TIMESTAMP=true` | `write_timestamp: false` | not sent | sent |
| `3_1_6_command_turns_on_over_env` | both `false` | both `true` | sent | sent |
**Expected Outcome**:
- `write_status = ok`; the server holds `1.25`.
- When sent, the stored `SourceTimestamp` equals the command `timestamp`; otherwise it is empty.
### 3.2.1 Pending Row Becomes Ok
**Summary**: A successful write records `ok`.
**Expected Outcome**:
- `write_status = ok`; the server holds the command value.
### 3.2.2 Null write_status Becomes Ok
**Summary**: Rows without a status (inserted before the column existed) are updated too.
**Expected Outcome**:
- `write_status` goes from NULL to `ok`.
### 3.2.3 Worse Outcome Is Kept
**Summary**: `ok` never replaces a worse outcome (`ok` < `error` < `session_bad`).
**Description**:
- Parametrized: the row starts as `error` or `session_bad`; a successful command arrives.
**Expected Outcome**:
- The OPC write happens, the row keeps its worse status.
### 3.2.4 Failure Replaces Ok
**Summary**: A worse outcome replaces a better one.
**Description**:
- A successful command records `ok`; a second command for the same row fails (conversion).
**Expected Outcome**:
- `write_status = error`.
### 3.2.5 Missing Row Is Not Created
**Summary**: The gateway only updates rows; it never inserts one.
**Description**:
- The command targets model 3, which has no prediction row.
**Expected Outcome**:
- The OPC write happens; no row for model 3 exists afterwards.
### 3.3.1 Unknown Server Is an Error
**Summary**: A `server_id` with no live connection and no connector fails the write.
**Expected Outcome**:
- `write_status = error`; nothing is written to the server.
### 3.3.2 Unusable Connector Is an Error
**Summary**: Only active `opc_ua` connectors with a URL and full security are used.
**Description**:
- Parametrized: connector `inactive`, `not_opc_ua` (`type: modbus`), `no_policy`, `no_url`.
**Expected Outcome**:
- `write_status = error`; nothing is written to the server.
### 3.3.3 Invalid Command Is an Error Without OPC Write
**Summary**: Commands that cannot become a DataValue are rejected before touching the server.
**Description**:
- Parametrized: `conversion` (`"abc"` as `int`), `unknown_data_type` (`decimal128`), `missing_addr`.
**Expected Outcome**:
- `write_status = error`; nothing is written to the server.
### 3.3.4 Bad Timestamp Skips the Write and Keeps the Row
**Summary**: A non-ISO `timestamp` (with the timestamp flag on) fails before the OPC write.
**Description**:
- The command carries `13/10/2026 12:00`; a marker command is used to know it was handled.
**Expected Outcome**:
- Only the marker reaches the server; the row stays `pending` — the status cannot be recorded
either, since the row is found by that same timestamp.
### 3.3.5 OPC Rejection Is an Error
**Summary**: A write the server refuses is recorded as `error`.
**Description**:
- Parametrized: `not_writable` (read-only `Locked`), `unknown_node` (`ns=2;s=Missing`).
**Expected Outcome**:
- `write_status = error`; `Locked` keeps its value.
### 3.3.6 Malformed JSON Is Dropped and the Consumer Keeps Going
**Summary**: A payload that is not JSON is discarded without stopping the consumer.
**Expected Outcome**:
- The next valid command is written and recorded as `ok`.
### 3.4.1 Write Reuses the Read Connection
**Summary**: A server read by the pod is written through the same pooled connection.
**Description**:
- `plant-a` is only known through slot 1 (no connector in Mongo).
**Expected Outcome**:
- `write_status = ok`; the server has a single session.
### 3.4.2 Write-Only Server Gets Its Own Connection
**Summary**: A server only known as a connector is connected on the first write.
**Expected Outcome**:
- No session before the command; one session after; the pool holds the connector's settings.
### 3.5.1 Minimum Pause Between Writes
**Summary**: `WRITE_MIN_PAUSE_S` spaces consecutive writes.
**Description**:
- `WRITE_MIN_PAUSE_S=0.5`; three commands are published back to back.
**Expected Outcome**:
- The server receives the three writes at least ~0.5 s apart.
### 3.5.2 Queue Group Writes Each Command Once
**Summary**: Pods share the `gateway-writers` queue: each command is handled by a single pod.
**Description**:
- Two pods with the runtime; six commands.
**Expected Outcome**:
- The server receives exactly six writes.
### 3.6.1 NATS Down at Startup Connects Later
**Summary**: The runtime starts without NATS and connects from the main loop once NATS is up.
**Description**:
- `NATS_URL` points to a port with nothing listening; the pod starts and iterates once.
- A NATS server starts on that port; the pod iterates again.
**Expected Outcome**:
- One `GATEWAY_NATS_CONSUMER_START_FAILED` notification for the whole outage (not one per retry).
- After NATS starts, the consumer connects and a published command is recorded as `ok`.
---
## 4. On-demand gRPC ReadTag
Source: `e2e/test_grpc_read_tag.py`
Common setup: `GRPC_ENABLED=true`; slot 1 reads `plant-a`, so the pod's pool holds a connection.
### 4.1.1 Reads a Tag of a Connected Server
**Expected Outcome**:
- `value` (as text), `opc_status_code = Good`, `source_timestamp` / `server_timestamp` in
`YYYY-MM-DD HH:MM:SS-0300`.
- `persisted = false` and no collection is created: the slot lookup of `ReadTag` is not wired yet.
### 4.1.2 Reads the Current Value
**Expected Outcome**:
- After a server-side change, `ReadTag` returns the new value.
### 4.2.1 Unknown Address Is an Internal Error
**Expected Outcome**:
- gRPC `INTERNAL` whose details carry `BadNodeIdUnknown`.
### 4.2.2 Server Not Connected by the Reads Is an Internal Error
**Summary**: `ReadTag` has no connection settings of its own: it only reads servers the read path
already connected (documents the current limitation).
**Description**:
- `plant-a` only exists as a connector; no slot.
**Expected Outcome**:
- gRPC `INTERNAL`; no session is opened on the server.
### 4.2.3 gRPC Disabled by Default
**Expected Outcome**:
- Without `GRPC_ENABLED`, nothing listens on `GRPC_PORT` (`UNAVAILABLE`/`DEADLINE_EXCEEDED`).
---
## 5. Scheduled Active Reads
Source: `e2e/test_active_read.py`
`GatewayRuntime` builds `ActiveReadEngine` on the pod's pool and sink, but no loop calls
`scheduled_tick` yet; these scenarios call it on the running pod's engine with the parsed slot
(`slot_active_read.json`).
### 5.1.1 Enabled Tag Is Read into Every Name
**Expected Outcome**:
- `Level` (`active_read.enabled`) is persisted in `active_tank` (model 30) and `active_mixer`
(model 40); `Temperature` (disabled) is not read.
### 5.1.2 cycle_multiple Reads Every Nth Tick
**Description**:
- `cycle_multiple = 3`; six ticks.
**Expected Outcome**:
- Persisted documents after each tick: `0, 0, 1, 1, 1, 2`.
### 5.2.1 Unreachable Server Does Not Stop the Tick
**Expected Outcome**:
- The tick does not raise; the unreachable server's tag is not persisted, the other server's is.
### 5.2.2 Active Read Uses the Slot Connection Settings
**Expected Outcome**:
- The first tick opens the connection with the slot settings; later ticks reuse it (one session);
each tick persists the current value.
---
## 6. Client Certificates and OPC Security
Source: `e2e/test_certificates.py`
Common setup: `plant-s` exposes only Basic256Sha256 SignAndEncrypt. Client certificates are created
like the platform stores them: cert (DER) and key (PEM) in MinIO bucket `opc-certificates`, plus an
`opc_certificates` document (`kind: client`).
### 6.1.1 Secured Read with Certificate from MinIO
**Expected Outcome**:
- Readings are persisted; the certificate is cached as `<CERT_CACHE_DIR>/7/cert.der` and `key.pem`.
- No connection error is notified.
### 6.1.2 Secured Write Resolved from Connectors and Certificates
**Description**:
- No slot: the connector (`client_certificate_id: 7`) is joined with `opc_certificates`.
**Expected Outcome**:
- `write_status = ok`; the server holds the value.
### 6.2.1 Certificate Missing in MinIO
**Expected Outcome**:
- Notification `OPC_CONNECTION_ERROR_plant-s`; no reader; nothing cached; no readings.
### 6.2.2 Unsupported Security Policy
**Expected Outcome**:
- Notification `OPC_CONNECTION_ERROR_plant-s` whose message names the unsupported mode/policy.
### 6.2.3 No Certificate Cannot Reach a Secured-only Server
**Description**:
- The slot has `certificate: null` (connect without security, D5).
**Expected Outcome**:
- Notification `OPC_CONNECTION_ERROR_plant-s`; no readings.
### 6.2.4 Inactive Certificate Write Fails
**Description**:
- The connector's certificate is `revoked`: the write connects without security.
**Expected Outcome**:
- `write_status = error`; nothing is written to the server.
### 6.3.1 Certificate Rotation Reconnects with the New One
**Description**:
- Slot 1 switches from certificate 7 to certificate 8; the pod iterates.
**Expected Outcome**:
- The pooled connection is replaced and opened with certificate 8 (cached under `8/`).
- Readings continue; the server ends with a single session.
---
## 7. Pod Lifecycle
Source: `e2e/test_lifecycle.py`
### 7.1.1 Shutdown Closes Sessions and Stops Consuming
**Expected Outcome**:
- After `GatewayRuntime.stop` + `Ingestor.shutdown`, the OPC server has no session.
- A command published afterwards is not handled (row stays `pending`); value changes are not persisted.
### 7.1.2 Idle Write Connection Is Closed and Reopened
**Description**:
- `IDLE_CONN_MAX_S=1`; `plant-a` is read (slot), `plant-b` only written (connector `2`).
**Expected Outcome**:
- After the idle time, `idle_sweep` closes the `plant-b` session; the subscribed `plant-a`
connection is pinned and stays open.
- The next write to `plant-b` reconnects and is recorded as `ok`.
### 7.2.1 Redis Unreachable Fails the Startup
**Expected Outcome**:
- `prepare_ingestor` raises a connection error (`app.py` then stops the process).
### 7.2.2 Restarted Pod Takes Its Slot Back
**Description**:
- `pod-a` (`LEASE_TTL=2`) stops; a new `pod-a` starts after the lease expired.
**Expected Outcome**:
- The restarted pod leases slot `1` and its readings reach MongoDB.

84
e2e/test_active_read.py
View File

@@ -1,84 +0,0 @@
"""E2E scenarios — 5. Scheduled active reads. See ``e2e/scenarios.md``.
``GatewayRuntime`` builds the ``ActiveReadEngine`` with the pod's pool and sink, but no
loop calls ``scheduled_tick`` yet (its wiring is pending in ``gateway_runtime.py``), so
these scenarios call it on the running pod's engine with the parsed slot.
"""
import pytest
from e2e.helpers import load_scenario_input, readings, stays_false, wait_until
from e2e.opc_server import free_port
from ingestor.io.slots import parse_slot
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
async def test_scenario_5_1_1_enabled_tag_is_read_into_every_name(opc_servers, pods, mongo_db):
server = await opc_servers('plant-a', {'Level': 2.5, 'Temperature': 20.0})
slot = parse_slot(load_scenario_input('slot_active_read.json', {'plant-a': server}))
pod = await pods('pod-a')
await pod.runtime.active_read.scheduled_tick(slot)
[tank] = readings(mongo_db, 'active_tank', name='tank_level')
[mixer] = readings(mongo_db, 'active_mixer', name='mixer_inlet_level')
assert tank['value'] == mixer['value'] == 2.5
assert (tank['model_id'], mixer['model_id']) == ('30', '40')
assert tank['tag'] == 'ns=2;s=Level'
assert readings(mongo_db, 'active_tank', name='tank_temperature') == [] # not enabled
async def test_scenario_5_1_2_cycle_multiple_reads_every_nth_tick(opc_servers, pods, mongo_db):
server = await opc_servers('plant-a', {'Level': 2.5, 'Temperature': 20.0})
raw = load_scenario_input('slot_active_read.json', {'plant-a': server})
raw['plant-a']['tags']['ns=2;s=Level']['active_read']['cycle_multiple'] = 3
slot = parse_slot(raw)
pod = await pods('pod-a')
counts = []
for _ in range(6):
await pod.runtime.active_read.scheduled_tick(slot)
counts.append(len(readings(mongo_db, 'active_tank', name='tank_level')))
assert counts == [0, 0, 1, 1, 1, 2]
async def test_scenario_5_2_1_unreachable_server_does_not_stop_the_tick(
opc_servers, pods, mongo_db
):
server = await opc_servers('plant-a', {'Level': 2.5, 'Temperature': 20.0})
raw = load_scenario_input('slot_active_read.json', {'plant-a': server})
down = {**raw['plant-a'], 'name': 'plant-down', 'server_id': '9'}
down['url'] = f'opc.tcp://127.0.0.1:{free_port()}'
down['tags'] = {
'ns=2;s=Level': {
**raw['plant-a']['tags']['ns=2;s=Level'],
'names': [{'collection': 'active_down', 'tag_name': 'down_level', 'model_id': '90'}],
}
}
slot = parse_slot({'plant-down': down, 'plant-a': raw['plant-a']})
pod = await pods('pod-a')
await pod.runtime.active_read.scheduled_tick(slot) # does not raise
assert readings(mongo_db, 'active_down') == []
await wait_until(lambda: readings(mongo_db, 'active_tank', name='tank_level'))
async def test_scenario_5_2_2_active_read_uses_the_slot_connection_settings(
opc_servers, pods, mongo_db
):
server = await opc_servers('plant-a', {'Level': 2.5, 'Temperature': 20.0})
slot = parse_slot(load_scenario_input('slot_active_read.json', {'plant-a': server}))
pod = await pods('pod-a')
assert pod.manager.pool.config_for('1') is None
await pod.runtime.active_read.scheduled_tick(slot)
assert pod.manager.pool.config_for('1')['url'] == server.url
assert server.session_count() == 1
await server.set('Level', 3.0)
await pod.runtime.active_read.scheduled_tick(slot)
assert await stays_false(lambda: server.session_count() != 1, duration=0.5)
assert [d['value'] for d in readings(mongo_db, 'active_tank', name='tank_level')] == [2.5, 3.0]

165
e2e/test_certificates.py
View File

@@ -1,165 +0,0 @@
"""E2E scenarios — 6. Client certificates and OPC security (MinIO + opc_certificates). See ``e2e/scenarios.md``."""
import pytest
import pytest_asyncio
from e2e.helpers import (
insert_prediction,
load_scenario_input,
notifications,
readings,
stays_false,
wait_until,
write_status,
)
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
TIMESTAMP = '2026-10-13T12:00:00-03:00'
SECURED = {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}
@pytest_asyncio.fixture
async def secured_server(opc_servers):
"""plant-s: only Basic256Sha256 SignAndEncrypt endpoints, accepts any client certificate."""
return await opc_servers('plant-s', {'Temperature': 20.0, 'Setpoint': 0.0}, secured=True)
def secured_slot(server, certificate, **security):
slot = load_scenario_input('slot_secured.json', {'plant-s': server})
slot['plant-s']['certificate'] = certificate
slot['plant-s']['security'] = {**SECURED, **security}
return slot
def secured_connector(server, certificate_id):
return {
'connector_id': '7',
'type': 'opc_ua',
'name': 'plant-s',
'url': server.url,
'status': 'Active',
'subscription_period_ms': 100,
'security': {**SECURED, 'client_certificate_id': certificate_id},
}
def write_command():
return load_scenario_input(
'write_command.json', server_id='7', model_id='2', timestamp=TIMESTAMP
)
# 6.1 Secured connections ---------------------------------------------------------------
async def test_scenario_6_1_1_secured_read_with_certificate_from_minio(
secured_server, client_certificate, pods, put_slot, mongo_db, cert_cache_dir
):
certificate = await client_certificate(7)
put_slot(1, secured_slot(secured_server, certificate))
pod = await pods('pod-a')
await pod.tick()
await secured_server.set('Temperature', 21.0)
await wait_until(lambda: readings(mongo_db, 'raw_secured', value=21.0))
assert sorted(p.name for p in (cert_cache_dir / '7').iterdir()) == ['cert.der', 'key.pem']
assert notifications(mongo_db, 'OPC_CONNECTION_ERROR') == []
async def test_scenario_6_1_2_secured_write_resolved_from_connectors_and_certificates(
secured_server, client_certificate, pods, mongo_db, postgres_engine, publish_command
):
await client_certificate(7)
mongo_db['connectors'].insert_one(secured_connector(secured_server, 7))
insert_prediction(postgres_engine, 2, TIMESTAMP, 'pending')
await pods('pod-a')
await publish_command(write_command())
await wait_until(lambda: write_status(postgres_engine, 2, TIMESTAMP) == 'ok')
assert (await secured_server.read('Setpoint')).Value.Value == 1.25
# 6.2 Failures --------------------------------------------------------------------------
async def test_scenario_6_2_1_certificate_missing_in_minio(
secured_server, client_certificate, pods, put_slot, mongo_db, cert_cache_dir
):
certificate = await client_certificate(7, upload=False)
put_slot(1, secured_slot(secured_server, certificate))
pod = await pods('pod-a')
await pod.tick()
assert notifications(mongo_db, 'OPC_CONNECTION_ERROR_plant-s')
assert pod.manager.opc_managers == {}
assert not list((cert_cache_dir / '7').glob('*.der'))
assert await stays_false(lambda: readings(mongo_db, 'raw_secured'))
async def test_scenario_6_2_2_unsupported_security_policy(
secured_server, client_certificate, pods, put_slot, mongo_db
):
certificate = await client_certificate(7)
put_slot(1, secured_slot(secured_server, certificate, policy='Basic999'))
pod = await pods('pod-a')
await pod.tick()
[error] = notifications(mongo_db, 'OPC_CONNECTION_ERROR_plant-s')
assert 'Unsupported OPC security' in error['message']
assert pod.manager.opc_managers == {}
async def test_scenario_6_2_3_no_certificate_cannot_reach_a_secured_only_server(
secured_server, pods, put_slot, mongo_db
):
put_slot(1, secured_slot(secured_server, None))
pod = await pods('pod-a')
await pod.tick()
assert notifications(mongo_db, 'OPC_CONNECTION_ERROR_plant-s')
assert await stays_false(lambda: readings(mongo_db, 'raw_secured'))
async def test_scenario_6_2_4_inactive_certificate_write_fails(
secured_server, client_certificate, pods, mongo_db, postgres_engine, publish_command
):
await client_certificate(7, status='revoked')
mongo_db['connectors'].insert_one(secured_connector(secured_server, 7))
insert_prediction(postgres_engine, 2, TIMESTAMP, 'pending')
await pods('pod-a')
await publish_command(write_command())
await wait_until(lambda: write_status(postgres_engine, 2, TIMESTAMP) == 'error')
assert secured_server.writes == []
# 6.3 Rotation --------------------------------------------------------------------------
async def test_scenario_6_3_1_certificate_rotation_reconnects_with_the_new_one(
secured_server, client_certificate, pods, put_slot, mongo_db, cert_cache_dir
):
first = await client_certificate(7)
put_slot(1, secured_slot(secured_server, first))
pod = await pods('pod-a')
await pod.tick()
await wait_until(lambda: readings(mongo_db, 'raw_secured'))
first_client = pod.manager.pool._connections['7'][0]
second = await client_certificate(8)
put_slot(1, secured_slot(secured_server, second))
await pod.tick()
assert pod.manager.pool._connections['7'][0] is not first_client
assert pod.manager.pool.config_for('7')['certificate']['certificate_id'] == 8
assert (cert_cache_dir / '8' / 'cert.der').is_file()
await secured_server.set('Temperature', 22.0)
await wait_until(lambda: readings(mongo_db, 'raw_secured', value=22.0))
await wait_until(lambda: secured_server.session_count() == 1)

113
e2e/test_grpc_read_tag.py
View File

@@ -1,113 +0,0 @@
"""E2E scenarios — 4. On-demand gRPC ReadTag. See ``e2e/scenarios.md``."""
import os
import re
import grpc
import pytest
import pytest_asyncio
from e2e.helpers import load_scenario_input, readings, wait_until
from ingestor.proto import opc_gateway_pb2, opc_gateway_pb2_grpc
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
FORMATTED_TIMESTAMP = re.compile(r'^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}-0300$')
@pytest_asyncio.fixture
async def grpc_stub():
"""Client of the gateway's gRPC server (GRPC_PORT of the test environment)."""
channel = grpc.aio.insecure_channel(f'127.0.0.1:{os.environ["GRPC_PORT"]}')
yield opc_gateway_pb2_grpc.OpcGatewayStub(channel)
await channel.close()
async def _reading_pod(opc_servers, pods, put_slot, mongo_db):
server = await opc_servers('plant-a', {'Temperature': 20.0, 'Level': 3.25})
put_slot(1, load_scenario_input('slot_single_server.json', {'plant-a': server}))
pod = await pods('pod-a', GRPC_ENABLED='true')
await pod.tick()
await wait_until(lambda: readings(mongo_db, 'raw_plant_a'))
return server, pod
async def test_scenario_4_1_1_reads_a_tag_of_a_connected_server(
opc_servers, pods, put_slot, mongo_db, grpc_stub
):
await _reading_pod(opc_servers, pods, put_slot, mongo_db)
persisted_before = mongo_db.list_collection_names()
response = await grpc_stub.ReadTag(
opc_gateway_pb2.ReadTagRequest(server_id='1', addr='ns=2;s=Level', data_type='float')
)
assert response.value == '3.25'
assert response.opc_status_code == 'Good'
assert FORMATTED_TIMESTAMP.match(response.source_timestamp), response.source_timestamp
assert FORMATTED_TIMESTAMP.match(response.server_timestamp), response.server_timestamp
assert response.persisted is False # no slot lookup wired yet: nothing is written
assert mongo_db.list_collection_names() == persisted_before
async def test_scenario_4_1_2_reads_the_current_value(
opc_servers, pods, put_slot, mongo_db, grpc_stub
):
server, _ = await _reading_pod(opc_servers, pods, put_slot, mongo_db)
await server.set('Level', 4.5)
response = await grpc_stub.ReadTag(
opc_gateway_pb2.ReadTagRequest(server_id='1', addr='ns=2;s=Level', data_type='float')
)
assert response.value == '4.5'
async def test_scenario_4_2_1_unknown_address_is_internal_error(
opc_servers, pods, put_slot, mongo_db, grpc_stub
):
await _reading_pod(opc_servers, pods, put_slot, mongo_db)
with pytest.raises(grpc.aio.AioRpcError) as error:
await grpc_stub.ReadTag(
opc_gateway_pb2.ReadTagRequest(server_id='1', addr='ns=2;s=Missing', data_type='float')
)
assert error.value.code() == grpc.StatusCode.INTERNAL
assert 'BadNodeIdUnknown' in error.value.details()
async def test_scenario_4_2_2_server_not_connected_by_the_reads_is_internal_error(
opc_servers, pods, mongo_db, grpc_stub
):
server = await opc_servers('plant-a', {'Level': 3.25})
mongo_db['connectors'].insert_one(
load_scenario_input('connector_plant_a.json', {'plant-a': server})
)
await pods('pod-a', GRPC_ENABLED='true') # no slot: the pool has no connection to plant-a
with pytest.raises(grpc.aio.AioRpcError) as error:
await grpc_stub.ReadTag(
opc_gateway_pb2.ReadTagRequest(server_id='1', addr='ns=2;s=Level', data_type='float')
)
assert error.value.code() == grpc.StatusCode.INTERNAL
assert server.session_count() == 0
async def test_scenario_4_2_3_grpc_disabled_by_default(
opc_servers, pods, put_slot, mongo_db, grpc_stub
):
server = await opc_servers('plant-a', {'Temperature': 20.0})
put_slot(1, load_scenario_input('slot_single_server.json', {'plant-a': server}))
pod = await pods('pod-a')
await pod.tick()
with pytest.raises(grpc.aio.AioRpcError) as error:
await grpc_stub.ReadTag(
opc_gateway_pb2.ReadTagRequest(server_id='1', addr='ns=2;s=Temperature'),
timeout=2,
)
assert error.value.code() in (grpc.StatusCode.UNAVAILABLE, grpc.StatusCode.DEADLINE_EXCEEDED)
assert pod.runtime.grpc_server is None

93
e2e/test_lifecycle.py
View File

@@ -1,93 +0,0 @@
"""E2E scenarios — 7. Pod lifecycle: shutdown, idle connections, startup failures. See ``e2e/scenarios.md``."""
import asyncio
import pytest
from e2e.helpers import (
Pod,
insert_prediction,
load_scenario_input,
readings,
stays_false,
wait_until,
write_status,
)
from e2e.opc_server import free_port
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
TIMESTAMP = '2026-10-13T12:00:00-03:00'
async def test_scenario_7_1_1_shutdown_closes_sessions_and_stops_consuming(
opc_servers, pods, put_slot, mongo_db, postgres_engine, publish_command
):
server = await opc_servers('plant-a', {'Temperature': 20.0, 'Setpoint': 0.0})
put_slot(1, load_scenario_input('slot_single_server.json', {'plant-a': server}))
insert_prediction(postgres_engine, 2, TIMESTAMP, 'pending')
pod = await pods('pod-a')
await pod.tick()
await wait_until(lambda: readings(mongo_db, 'raw_plant_a'))
assert server.session_count() == 1
await pod.stop()
await wait_until(lambda: server.session_count() == 0)
await publish_command(load_scenario_input('write_command.json'))
assert await stays_false(lambda: write_status(postgres_engine, 2, TIMESTAMP) != 'pending')
await server.set('Temperature', 50.0)
assert await stays_false(lambda: readings(mongo_db, 'raw_plant_a', value=50.0))
async def test_scenario_7_1_2_idle_write_connection_is_closed_and_reopened(
opc_servers, pods, put_slot, mongo_db, postgres_engine, publish_command
):
reader = await opc_servers('plant-a', {'Temperature': 20.0})
writer = await opc_servers('plant-b', {'Setpoint': 0.0})
put_slot(1, load_scenario_input('slot_single_server.json', {'plant-a': reader}))
connector = load_scenario_input('connector_plant_a.json', {'plant-a': writer})
mongo_db['connectors'].insert_one({**connector, 'connector_id': '2', 'name': 'plant-b'})
insert_prediction(postgres_engine, 2, TIMESTAMP, 'pending')
pod = await pods('pod-a', IDLE_CONN_MAX_S='1')
await pod.tick()
await publish_command(load_scenario_input('write_command.json', server_id='2'))
await wait_until(lambda: write_status(postgres_engine, 2, TIMESTAMP) == 'ok')
assert writer.session_count() == 1
await asyncio.sleep(1.5)
await pod.tick() # idle_sweep
await wait_until(lambda: writer.session_count() == 0)
assert reader.session_count() == 1 # subscribed: pinned, never idle
insert_prediction(postgres_engine, 3, TIMESTAMP, 'pending')
await publish_command(load_scenario_input('write_command.json', server_id='2', model_id='3'))
await wait_until(lambda: write_status(postgres_engine, 3, TIMESTAMP) == 'ok')
assert writer.session_count() == 1
async def test_scenario_7_2_1_redis_unreachable_fails_the_startup(monkeypatch):
monkeypatch.setenv('REDIS_PORT', str(free_port()))
with pytest.raises(Exception, match='(?i)connect'):
await Pod('pod-a').start()
async def test_scenario_7_2_2_restarted_pod_takes_its_slot_back(
opc_servers, pods, put_slot, mongo_db, redis_client
):
server = await opc_servers('plant-a', {'Temperature': 20.0})
put_slot(1, load_scenario_input('slot_single_server.json', {'plant-a': server}))
pod = await pods('pod-a', LEASE_TTL='2', HEARTBEAT_TTL='2')
await pod.tick()
await pod.stop()
await wait_until(lambda: redis_client.get('lease:opc_tags:1') is None, timeout=5)
restarted = await pods('pod-a', LEASE_TTL='2', HEARTBEAT_TTL='2')
await restarted.tick()
assert restarted.slots == {'1'}
await server.set('Temperature', 31.0)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=31.0))

193
e2e/test_slot_leasing.py
View File

@@ -1,193 +0,0 @@
"""E2E scenarios — 1. Slot leasing and load balancing (Redis). See ``e2e/scenarios.md``."""
import asyncio
import pytest
from e2e.helpers import (
load_scenario_input,
notifications,
readings,
redis_json,
stays_false,
wait_until,
)
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
async def _two_slots(opc_servers, put_slot):
"""Slot 1 reads plant-a, slot 2 reads plant-b."""
plant_a = await opc_servers('plant-a', {'Temperature': 20.0})
plant_b = await opc_servers('plant-b', {'Flow': 3.0})
both = load_scenario_input('slot_two_servers.json', {'plant-a': plant_a, 'plant-b': plant_b})
put_slot(1, {'plant-a': both['plant-a']})
put_slot(2, {'plant-b': both['plant-b']})
return plant_a, plant_b
# 1.1 Acquisition ----------------------------------------------------------------------
async def test_scenario_1_1_1_single_pod_leases_the_slot_and_heartbeats(
opc_servers, pods, put_slot, redis_client, mongo_db
):
server = await opc_servers('plant-a', {'Temperature': 20.0})
put_slot(1, load_scenario_input('slot_single_server.json', {'plant-a': server}))
pod = await pods('pod-a')
await pod.tick()
assert pod.slots == {'1'}
assert redis_json(redis_client, 'lease:opc_tags:1') == 'pod-a'
assert 0 < redis_client.ttl('lease:opc_tags:1') <= 10
assert redis_json(redis_client, 'heartbeat:ingestor:pod-a') == 1
assert 0 < redis_client.ttl('heartbeat:ingestor:pod-a') <= 20
await wait_until(lambda: readings(mongo_db, 'raw_plant_a'))
async def test_scenario_1_1_2_idle_pod_takes_a_slot_created_after_startup(
opc_servers, pods, put_slot, redis_client, mongo_db
):
server = await opc_servers('plant-a', {'Temperature': 20.0})
pod = await pods('pod-a')
await pod.tick()
assert pod.slots == set()
put_slot(1, load_scenario_input('slot_single_server.json', {'plant-a': server}))
await pod.tick()
assert pod.slots == {'1'}
assert redis_json(redis_client, 'lease:opc_tags:1') == 'pod-a'
await wait_until(lambda: readings(mongo_db, 'raw_plant_a'))
# 1.2 Balancing and failover ------------------------------------------------------------
async def test_scenario_1_2_1_two_pods_split_two_slots(
opc_servers, pods, put_slot, redis_client, mongo_db
):
plant_a, plant_b = await _two_slots(opc_servers, put_slot)
pod_a = await pods('pod-a')
await pod_a.tick()
assert pod_a.slots == {'1', '2'} # alone: takes every slot
pod_b = await pods('pod-b')
await pod_b.tick()
assert pod_b.slots == set() # every slot is leased
await pod_a.tick() # pod-a now sees a pod without slot and gives one back
assert pod_a.slots == {'1'}
await pod_b.tick()
assert pod_b.slots == {'2'}
assert redis_json(redis_client, 'lease:opc_tags:1') == 'pod-a'
assert redis_json(redis_client, 'lease:opc_tags:2') == 'pod-b'
assert set(pod_a.manager.opc_managers) == {'plant-a'}
assert set(pod_b.manager.opc_managers) == {'plant-b'}
await plant_a.set('Temperature', 25.0)
await plant_b.set('Flow', 4.5)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=25.0))
await wait_until(lambda: readings(mongo_db, 'raw_plant_b', value=4.5))
async def test_scenario_1_2_2_extra_pod_stays_idle(opc_servers, pods, put_slot, redis_client):
await _two_slots(opc_servers, put_slot)
pod_a, pod_b = await pods('pod-a'), await pods('pod-b')
await pod_a.tick()
await pod_b.tick()
await pod_a.tick()
await pod_b.tick()
pod_c = await pods('pod-c')
for _ in range(2):
for pod in (pod_a, pod_b, pod_c):
await pod.tick()
assert pod_c.slots == set()
assert pod_a.slots | pod_b.slots == {'1', '2'}
assert pod_a.slots.isdisjoint(pod_b.slots)
assert {
redis_json(redis_client, 'lease:opc_tags:1'),
redis_json(redis_client, 'lease:opc_tags:2'),
} == {
'pod-a',
'pod-b',
}
async def test_scenario_1_2_3_failover_after_lease_expiry(
opc_servers, pods, put_slot, redis_client, mongo_db
):
server = await opc_servers('plant-a', {'Temperature': 20.0})
put_slot(1, load_scenario_input('slot_single_server.json', {'plant-a': server}))
pod_a = await pods('pod-a', LEASE_TTL='2', HEARTBEAT_TTL='2')
await pod_a.tick()
pod_b = await pods('pod-b')
await pod_b.tick()
assert (pod_a.slots, pod_b.slots) == ({'1'}, set())
await pod_a.stop() # crash-like: the lease is not released, it expires
assert redis_json(redis_client, 'lease:opc_tags:1') == 'pod-a'
await wait_until(lambda: redis_json(redis_client, 'lease:opc_tags:1') is None, timeout=5)
await pod_b.tick()
assert pod_b.slots == {'1'}
assert redis_json(redis_client, 'lease:opc_tags:1') == 'pod-b'
await server.set('Temperature', 30.0)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=30.0))
# 1.3 Slot changes and invalid slots ----------------------------------------------------
async def test_scenario_1_3_1_deleted_slot_stops_the_reads(
opc_servers, pods, put_slot, redis_client, mongo_db
):
server = await opc_servers('plant-a', {'Temperature': 20.0})
put_slot(1, load_scenario_input('slot_single_server.json', {'plant-a': server}))
pod = await pods('pod-a')
await pod.tick()
await wait_until(lambda: readings(mongo_db, 'raw_plant_a'))
redis_client.delete('slot:opc_tags:1')
await pod.tick()
assert pod.slots == set()
assert pod.manager.opc_managers == {}
await server.set('Temperature', 99.0)
assert await stays_false(lambda: readings(mongo_db, 'raw_plant_a', value=99.0))
async def test_scenario_1_3_2_invalid_slot_is_notified_and_not_read(
opc_servers, pods, put_slot, mongo_db
):
server = await opc_servers('plant-a', {'Temperature': 20.0})
put_slot(1, load_scenario_input('slot_v1_invalid.json', {'plant-a': server}))
pod = await pods('pod-a')
await pod.tick()
assert pod.slots == set()
assert notifications(mongo_db, 'OPC_SLOT_CONFIG_ERROR_1')
assert await stays_false(lambda: readings(mongo_db, 'raw_plant_a'))
async def test_scenario_1_3_3_fixed_slot_is_read_once_its_lease_expires(
opc_servers, pods, put_slot, mongo_db
):
server = await opc_servers('plant-a', {'Temperature': 20.0})
put_slot(1, load_scenario_input('slot_v1_invalid.json', {'plant-a': server}))
pod = await pods('pod-a', LEASE_TTL='2')
await pod.tick()
assert pod.slots == set()
put_slot(1, load_scenario_input('slot_single_server.json', {'plant-a': server}))
await pod.tick()
assert pod.slots == set() # still leased by the failed acquisition
await asyncio.sleep(2.5)
await pod.tick()
assert pod.slots == {'1'}
await wait_until(lambda: readings(mongo_db, 'raw_plant_a'))

266
e2e/test_subscription_reads.py
View File

@@ -1,266 +0,0 @@
"""E2E scenarios — 2. Subscription reads (OPC UA → MongoDB). See ``e2e/scenarios.md``."""
import re
import pytest
from e2e.helpers import load_scenario_input, notifications, readings, stays_false, wait_until
from e2e.opc_server import free_port
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
# DATETIME_FORMAT_WITH_TZ in OPC_TIMEZONE (sientia_do), e.g. 2026-10-29 19:48:51-0300
READING_TIMESTAMP = re.compile(r'^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}-0300$')
async def _read_slot(opc_servers, pods, put_slot, slot_file, variables):
server = await opc_servers('plant-a', variables)
slot = load_scenario_input(slot_file, {'plant-a': server})
put_slot(1, slot)
pod = await pods('pod-a')
await pod.tick()
return server, slot, pod
# 2.1 Persistence -----------------------------------------------------------------------
async def test_scenario_2_1_1_value_change_is_persisted(opc_servers, pods, put_slot, mongo_db):
server, _, _ = await _read_slot(
opc_servers, pods, put_slot, 'slot_single_server.json', {'Temperature': 20.0}
)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=20.0)) # initial value
await server.set('Temperature', 21.5)
[doc] = await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=21.5))
assert doc['tag'] == 'ns=2;s=Temperature'
assert doc['name'] == 'temperature'
assert doc['model_id'] == '10'
assert READING_TIMESTAMP.match(doc['timestamp']), doc['timestamp']
assert doc['inserted_at'] is not None
assert set(doc) == {'tag', 'name', 'model_id', 'timestamp', 'value', 'inserted_at'}
async def test_scenario_2_1_2_one_address_fans_out_to_every_name(
opc_servers, pods, put_slot, mongo_db
):
server, _, _ = await _read_slot(
opc_servers, pods, put_slot, 'slot_multi_name.json', {'Pressure': 1.0}
)
await server.set('Pressure', 2.5)
[furnace] = await wait_until(lambda: readings(mongo_db, 'raw_furnace', value=2.5))
[boiler] = await wait_until(lambda: readings(mongo_db, 'raw_boiler', value=2.5))
assert (furnace['name'], furnace['model_id']) == ('furnace_pressure', '10')
assert (boiler['name'], boiler['model_id']) == ('boiler_inlet_pressure', '20')
assert furnace['tag'] == boiler['tag'] == 'ns=2;s=Pressure'
assert furnace['timestamp'] == boiler['timestamp']
async def test_scenario_2_1_3_two_servers_in_one_slot(opc_servers, pods, put_slot, mongo_db):
plant_a = await opc_servers('plant-a', {'Temperature': 20.0})
plant_b = await opc_servers('plant-b', {'Flow': 3.0})
put_slot(
1, load_scenario_input('slot_two_servers.json', {'plant-a': plant_a, 'plant-b': plant_b})
)
pod = await pods('pod-a')
await pod.tick()
await plant_a.set('Temperature', 22.0)
await plant_b.set('Flow', 3.5)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=22.0))
await wait_until(lambda: readings(mongo_db, 'raw_plant_b', value=3.5))
assert set(pod.manager.opc_managers) == {'plant-a', 'plant-b'}
async def test_scenario_2_1_4_values_keep_their_opc_type(opc_servers, pods, put_slot, mongo_db):
variables = {
'Temperature': (20.0, 'double', True),
'Counter': (1, 'int', True),
'Running': (False, 'bool', True),
'Batch': ('B-001', 'string', True),
}
server, _, _ = await _read_slot(opc_servers, pods, put_slot, 'slot_data_types.json', variables)
await server.set('Temperature', 20.25)
await server.set('Counter', 7)
await server.set('Running', True)
await server.set('Batch', 'B-002')
expected = {'temperature': 20.25, 'counter': 7, 'running': True, 'batch': 'B-002'}
for name, value in expected.items():
[doc] = await wait_until(
lambda n=name, v=value: readings(mongo_db, 'raw_types', name=n, value=v)
)
assert type(doc['value']) is type(value)
# 2.2 Slot changes ----------------------------------------------------------------------
async def test_scenario_2_2_1_tag_added_to_the_slot_is_subscribed(
opc_servers, pods, put_slot, mongo_db
):
server, slot, pod = await _read_slot(
opc_servers, pods, put_slot, 'slot_single_server.json', {'Temperature': 20.0, 'Level': 1.0}
)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', name='temperature'))
slot['plant-a']['tags']['ns=2;s=Level'] = {
'frequency': 1000,
'names': [{'collection': 'raw_plant_a', 'tag_name': 'level', 'model_id': '10'}],
}
put_slot(1, slot)
await pod.tick()
await server.set('Level', 1.75)
await server.set('Temperature', 23.0)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', name='level', value=1.75))
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', name='temperature', value=23.0))
async def test_scenario_2_2_2_tag_removed_from_the_slot_is_no_longer_read(
opc_servers, pods, put_slot, mongo_db
):
server = await opc_servers('plant-a', {'Temperature': 20.0, 'Level': 1.0})
slot = load_scenario_input('slot_single_server.json', {'plant-a': server})
slot['plant-a']['tags']['ns=2;s=Level'] = {
'frequency': 1000,
'names': [{'collection': 'raw_plant_a', 'tag_name': 'level', 'model_id': '10'}],
}
put_slot(1, slot)
pod = await pods('pod-a')
await pod.tick()
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', name='level'))
del slot['plant-a']['tags']['ns=2;s=Level']
put_slot(1, slot)
await pod.tick()
await server.set('Level', 9.0)
await server.set('Temperature', 24.0)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', name='temperature', value=24.0))
assert await stays_false(lambda: readings(mongo_db, 'raw_plant_a', name='level', value=9.0))
async def test_scenario_2_2_3_name_added_to_a_tag_receives_its_readings(
opc_servers, pods, put_slot, mongo_db
):
server, slot, pod = await _read_slot(
opc_servers, pods, put_slot, 'slot_single_server.json', {'Temperature': 20.0}
)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a'))
slot['plant-a']['tags']['ns=2;s=Temperature']['names'].append(
{'collection': 'raw_dryer', 'tag_name': 'dryer_temperature', 'model_id': '50'}
)
put_slot(1, slot)
await pod.tick()
await server.set('Temperature', 26.0)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=26.0))
[doc] = await wait_until(lambda: readings(mongo_db, 'raw_dryer', value=26.0))
assert (doc['name'], doc['model_id']) == ('dryer_temperature', '50')
async def test_scenario_2_2_4_server_url_change_reconnects_to_the_new_server(
opc_servers, pods, put_slot, mongo_db
):
old = await opc_servers('plant-a', {'Temperature': 20.0})
new = await opc_servers('plant-a-new', {'Temperature': 40.0})
slot = load_scenario_input('slot_single_server.json', {'plant-a': old})
put_slot(1, slot)
pod = await pods('pod-a')
await pod.tick()
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=20.0))
slot['plant-a']['url'] = new.url
put_slot(1, slot)
await pod.tick()
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=40.0))
await new.set('Temperature', 41.0)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=41.0))
await wait_until(lambda: old.session_count() == 0)
await old.set('Temperature', 21.0)
assert await stays_false(lambda: readings(mongo_db, 'raw_plant_a', value=21.0))
# 2.3 Failures and resilience -----------------------------------------------------------
async def test_scenario_2_3_1_unreachable_server_does_not_stop_the_others(
opc_servers, pods, put_slot, mongo_db
):
plant_a = await opc_servers('plant-a', {'Temperature': 20.0})
plant_b = await opc_servers('plant-b', {'Flow': 3.0})
slot = load_scenario_input('slot_two_servers.json', {'plant-a': plant_a, 'plant-b': plant_b})
slot['plant-b']['url'] = f'opc.tcp://127.0.0.1:{free_port()}' # nothing listens there
put_slot(1, slot)
pod = await pods('pod-a')
await pod.tick()
assert notifications(mongo_db, 'OPC_CONNECTION_ERROR_plant-b')
assert set(pod.manager.opc_managers) == {'plant-a'}
await plant_a.set('Temperature', 27.0)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=27.0))
async def test_scenario_2_3_2_tag_without_data_raises_a_warning(
opc_servers, pods, put_slot, mongo_db
):
_, _, pod = await _read_slot(
opc_servers, pods, put_slot, 'slot_single_server.json', {'Temperature': 20.0}
)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a'))
await pod.tick(5) # POLL_INTERVAL 1 s / frequency 1000 ms: one cycle per iteration
[warning] = notifications(mongo_db, 'TAG_ns=2;s=Temperature:temperature_LISTENNING_STOPPED')
assert warning['level'] in ('WARNING', 'warning')
async def test_scenario_2_3_3_silent_server_is_reconnected_after_15_cycles(
opc_servers, pods, put_slot, mongo_db
):
server, _, pod = await _read_slot(
opc_servers, pods, put_slot, 'slot_single_server.json', {'Temperature': 20.0}
)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a'))
first_client = pod.manager.pool._connections['1'][0]
await pod.tick(15)
assert notifications(mongo_db, 'OPC_LISTENNING_STOPPED__1')
assert notifications(mongo_db, 'OPC_CONNECTION_RETRY__1')
assert 'plant-a' not in pod.manager.opc_managers
await pod.tick()
assert 'plant-a' in pod.manager.opc_managers
assert pod.manager.pool._connections['1'][0] is not first_client
await server.set('Temperature', 28.0)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=28.0))
async def test_scenario_2_3_4_server_restart_resumes_the_reads(
opc_servers, pods, put_slot, mongo_db
):
server, _, pod = await _read_slot(
opc_servers, pods, put_slot, 'slot_single_server.json', {'Temperature': 20.0}
)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a'))
await server.stop()
await pod.tick(15) # no data: the server is dropped
assert 'plant-a' not in pod.manager.opc_managers
await pod.tick() # still down: reconnection fails and is notified
assert notifications(mongo_db, 'OPC_CONNECTION_ERROR_plant-a')
await server.start()
await pod.tick(2)
assert 'plant-a' in pod.manager.opc_managers
await server.set('Temperature', 29.0)
await wait_until(lambda: readings(mongo_db, 'raw_plant_a', value=29.0))

386
e2e/test_writes.py
View File

@@ -1,386 +0,0 @@
"""E2E scenarios — 3. Writes (NATS C2 → OPC UA → Postgres write_status C3). See ``e2e/scenarios.md``."""
import json
from datetime import datetime
import nats
import pytest
import pytest_asyncio
from e2e.conftest import WRITE_SUBJECT, nats_server_container
from e2e.helpers import (
SOURCE_TIMESTAMP_BIT,
STATUS_CODE_BIT,
datavalue_mask,
insert_prediction,
load_scenario_input,
notifications,
prediction_count,
stays_false,
wait_until,
write_status,
)
from e2e.opc_server import free_port
pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
TIMESTAMP = '2026-10-13T12:00:00-03:00'
MODEL_ID = 2
MARKER_MODEL_ID = 99
@pytest_asyncio.fixture
async def target(opc_servers, mongo_db, postgres_engine):
"""plant-a with a writable Setpoint and a read-only Locked, its connector in Mongo
and a pending prediction row (model 2)."""
server = await opc_servers(
'plant-a',
{
'Setpoint': 0.0,
'Counter': (0, 'int', True),
'Locked': (5.0, 'float', False),
'Marker': 0.0,
},
)
mongo_db['connectors'].insert_one(
load_scenario_input('connector_plant_a.json', {'plant-a': server})
)
insert_prediction(postgres_engine, MODEL_ID, TIMESTAMP, 'pending')
return server
def command(**overrides) -> dict:
return load_scenario_input('write_command.json', **overrides)
async def processed(publish_command, postgres_engine) -> None:
"""Wait until every command published so far was handled: NATS delivers the
messages of a subscription in order, so once a marker command (another row, the
``Marker`` variable) is recorded, the previous ones are done."""
insert_prediction(postgres_engine, MARKER_MODEL_ID, TIMESTAMP, 'pending')
await publish_command(command(model_id=str(MARKER_MODEL_ID), addr='ns=2;s=Marker'))
await wait_until(lambda: write_status(postgres_engine, MARKER_MODEL_ID, TIMESTAMP) == 'ok')
async def wait_status(postgres_engine, expected: str, model_id: int = MODEL_ID) -> None:
await wait_until(lambda: write_status(postgres_engine, model_id, TIMESTAMP) == expected)
# 3.1 DataValue parts: timestamp and quality (SIENTIAPDE-2202) ------------------------
DATAVALUE_CASES = [
pytest.param({}, {}, True, True, id='3_1_1_defaults_on'),
pytest.param({'WRITE_TIMESTAMP': 'false'}, {}, False, True, id='3_1_2_timestamp_off'),
pytest.param({'WRITE_QUALITY': 'false'}, {}, True, False, id='3_1_3_quality_off'),
pytest.param(
{'WRITE_TIMESTAMP': 'false', 'WRITE_QUALITY': 'false'},
{},
False,
False,
id='3_1_4_both_off',
),
pytest.param(
{'WRITE_TIMESTAMP': 'true'},
{'write_timestamp': False},
False,
True,
id='3_1_5_command_overrides_env',
),
pytest.param(
{'WRITE_TIMESTAMP': 'false', 'WRITE_QUALITY': 'false'},
{'write_timestamp': True, 'write_quality': True},
True,
True,
id='3_1_6_command_turns_on_over_env',
),
]
@pytest.mark.parametrize('env,flags,expect_timestamp,expect_quality', DATAVALUE_CASES)
async def test_scenario_3_1_datavalue_parts(
target,
pods,
publish_command,
postgres_engine,
opc_wire,
env,
flags,
expect_timestamp,
expect_quality,
):
await pods('pod-a', **env)
await publish_command(command(**flags))
await wait_status(postgres_engine, 'ok')
stored = await target.read('Setpoint')
assert stored.Value.Value == 1.25
if expect_timestamp:
assert stored.SourceTimestamp == datetime.fromisoformat(TIMESTAMP)
else:
assert stored.SourceTimestamp is None
[request] = opc_wire
mask = datavalue_mask(request, 'ns=2;s=Setpoint')
assert bool(mask & SOURCE_TIMESTAMP_BIT) is expect_timestamp
assert bool(mask & STATUS_CODE_BIT) is expect_quality
# 3.2 write_status (C3) -----------------------------------------------------------------
async def test_scenario_3_2_1_pending_row_becomes_ok(
target, pods, publish_command, postgres_engine
):
await pods('pod-a')
await publish_command(command(value=3.5))
await wait_status(postgres_engine, 'ok')
assert (await target.read('Setpoint')).Value.Value == 3.5
async def test_scenario_3_2_2_null_write_status_becomes_ok(
opc_servers, mongo_db, pods, publish_command, postgres_engine
):
server = await opc_servers('plant-a', {'Setpoint': 0.0})
mongo_db['connectors'].insert_one(
load_scenario_input('connector_plant_a.json', {'plant-a': server})
)
insert_prediction(postgres_engine, MODEL_ID, TIMESTAMP, None) # row from before the column
await pods('pod-a')
await publish_command(command())
await wait_status(postgres_engine, 'ok')
@pytest.mark.parametrize('worse', ['error', 'session_bad'])
async def test_scenario_3_2_3_worse_outcome_is_kept(
opc_servers, mongo_db, pods, publish_command, postgres_engine, worse
):
server = await opc_servers('plant-a', {'Setpoint': 0.0, 'Marker': 0.0})
mongo_db['connectors'].insert_one(
load_scenario_input('connector_plant_a.json', {'plant-a': server})
)
insert_prediction(postgres_engine, MODEL_ID, TIMESTAMP, worse)
await pods('pod-a')
await publish_command(command(value=4.0))
await processed(publish_command, postgres_engine)
assert (await server.read('Setpoint')).Value.Value == 4.0 # the OPC write happened
assert write_status(postgres_engine, MODEL_ID, TIMESTAMP) == worse
async def test_scenario_3_2_4_failure_replaces_ok(target, pods, publish_command, postgres_engine):
await pods('pod-a')
await publish_command(command())
await wait_status(postgres_engine, 'ok')
await publish_command(command(value='not-a-number'))
await wait_status(postgres_engine, 'error')
async def test_scenario_3_2_5_missing_row_is_not_created(
target, pods, publish_command, postgres_engine
):
await pods('pod-a')
await publish_command(command(model_id='3', value=6.0))
await processed(publish_command, postgres_engine)
assert (await target.read('Setpoint')).Value.Value == 6.0
assert write_status(postgres_engine, 3, TIMESTAMP) == '<missing>'
assert prediction_count(postgres_engine) == 2 # model 2 + marker
# 3.3 Failed commands -------------------------------------------------------------------
async def test_scenario_3_3_1_unknown_server_is_an_error(
target, pods, publish_command, postgres_engine
):
await pods('pod-a')
await publish_command(command(server_id='404'))
await wait_status(postgres_engine, 'error')
assert target.writes == []
@pytest.mark.parametrize(
'connector_change',
[
pytest.param({'status': 'Inactive'}, id='inactive'),
pytest.param({'type': 'modbus'}, id='not_opc_ua'),
pytest.param({'security': {'mode': 'None'}}, id='no_policy'),
pytest.param({'url': ''}, id='no_url'),
],
)
async def test_scenario_3_3_2_unusable_connector_is_an_error(
target, mongo_db, pods, publish_command, postgres_engine, connector_change
):
mongo_db['connectors'].update_one({'connector_id': '1'}, {'$set': connector_change})
await pods('pod-a')
await publish_command(command())
await wait_status(postgres_engine, 'error')
assert target.writes == []
@pytest.mark.parametrize(
'overrides',
[
pytest.param(
{'value': 'abc', 'data_type': 'int', 'addr': 'ns=2;s=Counter'}, id='conversion'
),
pytest.param({'data_type': 'decimal128'}, id='unknown_data_type'),
pytest.param({'addr': None}, id='missing_addr'),
],
)
async def test_scenario_3_3_3_invalid_command_is_an_error_without_opc_write(
target, pods, publish_command, postgres_engine, overrides
):
await pods('pod-a')
cmd = command(**overrides)
if overrides.get('addr', '') is None:
del cmd['addr']
await publish_command(cmd)
await wait_status(postgres_engine, 'error')
assert target.writes == []
async def test_scenario_3_3_4_bad_timestamp_skips_the_write_and_keeps_the_row(
target, pods, publish_command, postgres_engine
):
await pods('pod-a')
await publish_command(command(timestamp='13/10/2026 12:00'))
await processed(publish_command, postgres_engine)
assert [w.NodeId.Identifier for w in target.writes] == ['Marker']
assert write_status(postgres_engine, MODEL_ID, TIMESTAMP) == 'pending'
@pytest.mark.parametrize(
'addr',
[
pytest.param('ns=2;s=Locked', id='not_writable'),
pytest.param('ns=2;s=Missing', id='unknown_node'),
],
)
async def test_scenario_3_3_5_opc_rejection_is_an_error(
target, pods, publish_command, postgres_engine, addr
):
await pods('pod-a')
await publish_command(command(addr=addr))
await wait_status(postgres_engine, 'error')
assert (await target.read('Locked')).Value.Value == 5.0
async def test_scenario_3_3_6_malformed_json_is_dropped_and_the_consumer_keeps_going(
target, pods, publish_command, postgres_engine
):
await pods('pod-a')
await publish_command(b'{"server_id": "1", "value": ')
await publish_command(command(value=7.0))
await wait_status(postgres_engine, 'ok')
assert (await target.read('Setpoint')).Value.Value == 7.0
# 3.4 Connections -----------------------------------------------------------------------
async def test_scenario_3_4_1_write_reuses_the_read_connection(
opc_servers, pods, put_slot, publish_command, postgres_engine, mongo_db
):
server = await opc_servers('plant-a', {'Temperature': 20.0, 'Setpoint': 0.0})
put_slot(1, load_scenario_input('slot_single_server.json', {'plant-a': server}))
insert_prediction(postgres_engine, MODEL_ID, TIMESTAMP, 'pending')
pod = await pods('pod-a')
await pod.tick()
assert mongo_db['connectors'].count_documents({}) == 0 # only the slot knows plant-a
await publish_command(command())
await wait_status(postgres_engine, 'ok')
assert server.session_count() == 1
async def test_scenario_3_4_2_write_only_server_gets_its_own_connection(
target, pods, publish_command, postgres_engine
):
pod = await pods('pod-a')
assert target.session_count() == 0
await publish_command(command())
await wait_status(postgres_engine, 'ok')
assert target.session_count() == 1
assert pod.manager.pool.config_for('1')['url'] == target.url
# 3.5 Throughput and scaling ------------------------------------------------------------
async def test_scenario_3_5_1_min_pause_between_writes(
target, pods, publish_command, postgres_engine
):
await pods('pod-a', WRITE_MIN_PAUSE_S='0.5')
for value in (1.0, 2.0, 3.0):
await publish_command(command(value=value))
await wait_until(lambda: len(target.writes) == 3)
gaps = [b - a for a, b in zip(target.write_times, target.write_times[1:], strict=False)]
assert all(gap >= 0.45 for gap in gaps), gaps
async def test_scenario_3_5_2_queue_group_writes_each_command_once(
target, pods, publish_command, postgres_engine
):
await pods('pod-a')
await pods('pod-b')
for value in range(6):
await publish_command(command(value=float(value)))
await wait_until(lambda: len(target.writes) == 6)
assert await stays_false(lambda: len(target.writes) > 6)
# 3.6 NATS availability -----------------------------------------------------------------
async def test_scenario_3_6_1_nats_down_at_startup_connects_later(
target, pods, postgres_engine, mongo_db
):
port = free_port()
url = f'nats://127.0.0.1:{port}'
pod = await pods('pod-a', NATS_URL=url) # nothing listens yet
await pod.tick() # retried by idle_sweep, notified once per outage
assert len(notifications(mongo_db, 'GATEWAY_NATS_CONSUMER_START_FAILED')) == 1
nats_server = nats_server_container(host_port=port)
nats_server.start()
try:
await pod.tick()
assert pod.runtime._nats_started
client = await nats.connect(url)
cmd = load_scenario_input('write_command.json')
await client.publish(WRITE_SUBJECT, json.dumps(cmd).encode())
await client.flush()
await wait_status(postgres_engine, 'ok')
await client.drain()
finally:
await pod.stop()
nats_server.stop()

0
ingestor/adapters/__init__.py
View File

70
ingestor/adapters/grpc_server.py
View File

@@ -1,70 +0,0 @@
"""gRPC input adapter for on-demand tag reads (contract C4).
Thin adapter: unpack the request, call ActiveReadEngine.read_now, map the
result onto ReadTagResponse. Internal-only for now -- no auth/TLS, per the
feature owner (no consumer exists yet, likely a frontend later).
"""
import grpc
from sientia_do.observability.logger import Logger
from ingestor.engines.active_read import ActiveReadEngine
from ingestor.proto import opc_gateway_pb2, opc_gateway_pb2_grpc
class ReadTagServicer(opc_gateway_pb2_grpc.OpcGatewayServicer):
def __init__(self, active_read: ActiveReadEngine, logger: Logger, slot_lookup=None):
self.active_read = active_read
self.logger = logger
# ponytail: slot_lookup wiring (address -> names from the current slot)
# lands at the SIENTIAPDE-1974 integration -- `names` only exist in slot v2,
# which the Orchestrator produces after 1974. Until then ReadTag never
# persists (persisted=False); optional callable keeps this servicer usable now.
self.slot_lookup = slot_lookup
async def ReadTag(self, request, context):
names = None
if self.slot_lookup is not None:
names = self.slot_lookup(request.server_id, request.addr)
try:
result = await self.active_read.read_now(request.server_id, request.addr, names=names)
except Exception as e:
self.logger.error(f'ReadTag failed for {request.server_id}:{request.addr}: {e}')
await context.abort(grpc.StatusCode.INTERNAL, f'read failed: {e}')
return
return opc_gateway_pb2.ReadTagResponse(
value=str(result['value']),
opc_status_code=result['opc_status_code'] or '',
source_timestamp=result['source_timestamp'] or '',
server_timestamp=result['server_timestamp'] or '',
persisted=result['persisted'],
)
class GrpcServer:
def __init__(
self,
active_read: ActiveReadEngine,
logger: Logger,
port: int = 50051,
slot_lookup=None,
):
self.active_read = active_read
self.logger = logger
self.port = port
self.slot_lookup = slot_lookup
self.server: grpc.aio.Server | None = None
async def start(self) -> None:
self.server = grpc.aio.server()
opc_gateway_pb2_grpc.add_OpcGatewayServicer_to_server(
ReadTagServicer(self.active_read, self.logger, self.slot_lookup), self.server
)
self.server.add_insecure_port(f'[::]:{self.port}')
await self.server.start()
self.logger.info(f'GrpcServer started on port {self.port}')
async def stop(self) -> None:
if self.server is None:
return
await self.server.stop(grace=5)

124
ingestor/adapters/nats_consumer.py
View File

@@ -1,124 +0,0 @@
"""NATS input adapter for OPC write commands (contract C2).
Thin adapter: decode the message JSON and hand the dict to WriteEngine,
which is already raise-safe against malformed commands. No metrics of its
own — the engine records write metrics; a malformed-JSON drop is logged
here. Once connected, nats-py reconnects by itself, forever.
"""
import asyncio
import json
import math
import traceback
from nats.aio.client import Client
from nats.aio.subscription import Subscription
from sientia_do.observability.logger import Logger
from ingestor.engines.write import WriteEngine
class NatsWriteConsumer:
def __init__(
self,
nats_url: str,
write_engine: WriteEngine,
logger: Logger,
subject: str = 'opc.write',
queue: str = 'gateway-writers',
connect_timeout_s: float = 5.0,
):
self.nats_url = nats_url
self.write_engine = write_engine
self.logger = logger
self.subject = subject
self.queue = queue
self.connect_timeout_s = connect_timeout_s
self.nc: Client | None = None
self.sub: Subscription | None = None
@property
def closed(self) -> bool:
"""True when there is no connection, or it was closed for good."""
return self.nc is None or self.nc.is_closed
async def start(self) -> None:
"""Connect and subscribe, giving up after ``connect_timeout_s``.
It runs from the main loop (startup and ``GatewayRuntime.idle_sweep``), so it
must never block it: nats-py's defaults retry the first connection 60 times,
about 2 minutes, longer than the heartbeat TTL, and with
``max_reconnect_attempts=-1`` they would retry it forever. Once connected,
reconnection is unbounded, so an outage never closes the connection for good.
Raises:
Exception: When the connection or the subscription failed; nothing is left
open in that case.
"""
nc = Client()
try:
await asyncio.wait_for(
nc.connect(
servers=self.nats_url,
connect_timeout=math.ceil(self.connect_timeout_s),
max_reconnect_attempts=-1,
name='opc-gateway',
# One log line per event instead of nats-py's traceback per attempt.
error_cb=self._on_error,
disconnected_cb=self._on_disconnected,
reconnected_cb=self._on_reconnected,
),
timeout=self.connect_timeout_s,
)
self.sub = await nc.subscribe(self.subject, queue=self.queue, cb=self._handle)
except (Exception, asyncio.CancelledError):
await self._discard(nc)
raise
self.nc = nc
self.logger.info(f'NatsWriteConsumer started: subject={self.subject} queue={self.queue}')
async def _handle(self, msg) -> None:
try:
payload = json.loads(msg.data)
except ValueError as e: # includes JSONDecodeError and UnicodeDecodeError
preview = msg.data[:200]
self.logger.error(f'NatsWriteConsumer dropped malformed JSON ({e}): {preview!r}')
return
try:
await self.write_engine.write(payload)
except Exception as e:
# The engine is raise-safe; this only keeps an unexpected bug from going to
# nats-py's error callback without context.
self.logger.error(
f'NatsWriteConsumer: unexpected error handling a write command ({e}):\n'
f'{traceback.format_exc()}'
)
# These stay async with nothing to await: nats-py rejects callbacks that are not
# coroutine functions (InvalidCallbackTypeError on connect).
async def _on_error(self, e: Exception) -> None:
self.logger.warning(f'NATS error: {type(e).__name__} {e}')
async def _on_disconnected(self) -> None:
self.logger.warning('Disconnected from NATS; reconnecting')
async def _on_reconnected(self) -> None:
self.logger.info('Reconnected to NATS')
async def _discard(self, nc: Client) -> None:
try:
await nc.close()
except Exception as e:
self.logger.warning(f'Error closing a failed NATS connection: {e}')
async def stop(self) -> None:
if self.nc is None:
return
if self.nc.is_closed:
# Closed for good already: unsubscribe/drain would raise ConnectionClosedError
# and abort the shutdown sequence.
self.nc = None
return
if self.sub is not None:
await self.sub.unsubscribe()
await self.nc.drain()

38
ingestor/app.py
View File

@@ -7,16 +7,7 @@ from time import time
from prometheus_client import start_http_server
try: # optional: local dev reads .env; in-cluster env comes from Helm (no .env file)
from dotenv import load_dotenv
load_dotenv()
except ImportError:
pass
import ingestor.metrics as metrics
from ingestor.config import GatewayConfig
from ingestor.gateway_runtime import GatewayRuntime
from ingestor.ingestor import Ingestor
exit_signal = Event()
@@ -57,37 +48,10 @@ async def main(): # NOSONAR
exit_signal.set()
ingestor.logger.info('Ingestor prepared. Starting main loop.')
# Gateway write/active-read/gRPC wiring (Task 18). Only built if prepare_ingestor
# got far enough to have an ingestor_manager (its data_manager and connection
# pool are shared) -- skipped entirely if prepare failed before that point. The
# read path (subscriptions) lives in the ingestor_manager and is already running:
# a failure here only disables writes/gRPC, the app continues without the runtime.
runtime: GatewayRuntime | None = None
if ingestor.ingestor_manager is not None:
try:
runtime = GatewayRuntime(
config=GatewayConfig.from_env(),
data_manager=ingestor.ingestor_manager.data_manager,
pool=ingestor.ingestor_manager.pool,
logger=ingestor.logger,
notification_handler=ingestor.notification_handler,
metrics_controller=ingestor.metrics_controller,
metadata=ingestor.metadata,
)
await runtime.start()
except Exception as e:
runtime = None
metrics.APP_ERRORS_TOTAL.labels(pod_id=POD_ID).inc()
ingestor.logger.error(
f'GatewayRuntime failed to initialize, continuing without gateway: {e}'
)
while not exit_signal.is_set(): # NOSONAR
start_time = time() # Start loop timer
try:
await ingestor.loop()
if runtime is not None:
await runtime.idle_sweep()
metrics.APP_LOOP_COUNT.labels(pod_id=POD_ID).inc() # Increment loop counter
# Use asyncio.sleep instead of exit_signal.wait for better async compatibility
@@ -106,8 +70,6 @@ async def main(): # NOSONAR
duration = time() - start_time
metrics.APP_LOOP_DURATION.labels(pod_id=POD_ID).observe(duration)
if runtime is not None:
await runtime.stop()
await ingestor.shutdown()
metrics.APP_UP.labels(pod_id=POD_ID).set(0) # Mark app as DOWN

53
ingestor/config.py
View File

@@ -1,53 +0,0 @@
"""Gateway-specific env config (NATS/gRPC/certs/write flags).
Separate from ingestor.py's inline os.getenv (Redis/Mongo/TTLs) — this module
covers only the new gateway wiring consumed by app.py in Task 18.
"""
from dataclasses import dataclass
from os import getenv
def _env_bool(name: str, default: bool) -> bool:
value = getenv(name)
if value is None:
return default
return value.strip().lower() in ('true', '1', 'yes')
@dataclass
class GatewayConfig:
nats_url: str = 'nats://localhost:4222'
nats_write_subject: str = 'opc.write'
nats_queue: str = 'gateway-writers'
grpc_enabled: bool = False
grpc_port: int = 50051
# The certificate bucket and objects come from the slot / the write lookup
# (opc_certificates, SIENTIAPDE-1974); only the local cache dir is configured here.
cert_cache_dir: str = '/tmp/opc-certs' # noqa: S108 # nosec B108 - default is spec'd, overridable via env
write_timestamp: bool = True
write_quality: bool = True
write_min_pause_s: float = 0.0
idle_conn_max_s: float = 300.0
@classmethod
def certs_from_env(cls) -> 'GatewayConfig':
"""Only the certificate settings — what the read path needs. Plain strings,
no number parsing, so a malformed write/gRPC env var (GRPC_PORT,
WRITE_MIN_PAUSE_S, ...) can only disable the runtime, never the reads."""
return cls(cert_cache_dir=getenv('CERT_CACHE_DIR', cls.cert_cache_dir))
@classmethod
def from_env(cls) -> 'GatewayConfig':
return cls(
nats_url=getenv('NATS_URL', cls.nats_url),
nats_write_subject=getenv('NATS_WRITE_SUBJECT', cls.nats_write_subject),
nats_queue=getenv('NATS_QUEUE', cls.nats_queue),
grpc_enabled=_env_bool('GRPC_ENABLED', cls.grpc_enabled),
grpc_port=int(getenv('GRPC_PORT', cls.grpc_port)),
cert_cache_dir=getenv('CERT_CACHE_DIR', cls.cert_cache_dir),
write_timestamp=_env_bool('WRITE_TIMESTAMP', cls.write_timestamp),
write_quality=_env_bool('WRITE_QUALITY', cls.write_quality),
write_min_pause_s=float(getenv('WRITE_MIN_PAUSE_S', cls.write_min_pause_s)),
idle_conn_max_s=float(getenv('IDLE_CONN_MAX_S', cls.idle_conn_max_s)),
)

0
ingestor/connection/__init__.py
View File

92
ingestor/connection/certificates.py
View File

@@ -1,92 +0,0 @@
"""Client certificate of a tag database connector (SIENTIAPDE-1974 D5).
Mirrors the Orchestrator's `tag_database.resolve_certificate`, which puts the
certificate in the read slot: the write lookup (ServerConfigResolver, D1) resolves a
server that is not in the pool with the same rule, so a write uses the certificate the
slot would have. Keep both in sync.
"""
from typing import Any
# security fields of an OPC UA connector: the source of its OPC security (D4)
SECURITY_FIELDS = ('mode', 'policy')
CERTIFICATE_FIELDS = ('certificate_id', 'application_uri')
CERTIFICATE_OBJECTS = ('cert', 'private_key')
STORED_OBJECT_FIELDS = ('bucket', 'object_name')
# opc_certificates lookup of a connectors aggregation: the certificate the connector's
# security.client_certificate_id points to, only when it is active and of kind client.
CERTIFICATE_LOOKUP = {
'$lookup': {
'from': 'opc_certificates',
'let': {'certificate_id': '$security.client_certificate_id'},
'pipeline': [
{
'$match': {
'$expr': {'$eq': ['$certificate_id', '$$certificate_id']},
'status': 'active',
'kind': 'client',
}
},
{
'$project': {
'_id': 0,
'certificate_id': 1,
'application_uri': 1,
'cert.bucket': 1,
'cert.object_name': 1,
'private_key.bucket': 1,
'private_key.object_name': 1,
}
},
],
'as': 'certificates',
}
}
def _is_complete(document: Any, fields: tuple[str, ...]) -> bool:
return isinstance(document, dict) and all(
document.get(field) not in (None, '') for field in fields
)
def _is_usable_certificate(certificate: Any) -> bool:
return _is_complete(certificate, CERTIFICATE_FIELDS) and all(
_is_complete(certificate.get(name), STORED_OBJECT_FIELDS) for name in CERTIFICATE_OBJECTS
)
def client_certificate_id(connector: dict[str, Any]) -> Any:
"""The connector's security.client_certificate_id (None when it has none)."""
security = connector.get('security')
return security.get('client_certificate_id') if isinstance(security, dict) else None
def has_security(connector: dict[str, Any]) -> bool:
"""Whether the connector has security.mode and security.policy (D4)."""
return _is_complete(connector.get('security'), SECURITY_FIELDS)
def resolve_certificate(connector: dict[str, Any]) -> dict[str, Any] | None:
"""
The client certificate of a connector from CERTIFICATE_LOOKUP, or None (D5).
None means no certificate: the connection is opened without security. A connector
whose client_certificate_id is set but has no usable certificate is most likely a
registration error; the caller reports it (see client_certificate_id).
Returns:
dict[str, Any] | None: certificate_id, application_uri, and cert and private_key
as {bucket, object_name}, as in the slot
"""
for certificate in connector.get('certificates') or []:
if _is_usable_certificate(certificate):
return {
**{field: certificate[field] for field in CERTIFICATE_FIELDS},
**{
name: {field: certificate[name][field] for field in STORED_OBJECT_FIELDS}
for name in CERTIFICATE_OBJECTS
},
}
return None

139
ingestor/connection/certs.py
View File

@@ -1,139 +0,0 @@
import asyncio
import os
from os import getenv
from pathlib import Path
from sientia_do.repository.minio_repository import MinioRepository
from ingestor.config import _env_bool
# The two objects of a client certificate, and the local file each one is saved as
CERTIFICATE_OBJECTS = (('cert', 'cert'), ('private_key', 'key'))
class CertificateNotConfiguredError(RuntimeError):
"""The certificate of a server does not name its MinIO objects (opc_certificates)."""
def _objects(certificate: dict) -> tuple[tuple[str, str], ...]:
"""(bucket, object_name) of the cert and the private key, in CERTIFICATE_OBJECTS order."""
try:
return tuple(
(str(certificate[field]['bucket']), str(certificate[field]['object_name']))
for field, _ in CERTIFICATE_OBJECTS
)
except (KeyError, TypeError) as e:
raise CertificateNotConfiguredError(
f'OPC client certificate {certificate!r} does not name the MinIO bucket and '
f'object of its cert and private_key (opc_certificates)'
) from e
def _certificate_id(certificate: dict) -> str:
"""certificate_id as the name of its cache folder: an integer, never a path."""
try:
return str(int(certificate['certificate_id']))
except (KeyError, TypeError, ValueError) as e:
raise CertificateNotConfiguredError(
f'OPC client certificate {certificate!r} has no integer certificate_id'
) from e
class CertProvider:
"""Downloads the client cert + private key of a connection (the MinIO objects its
slot server or write lookup names, SIENTIAPDE-1974) into a writable cache dir — pod
root FS is read-only (§12), so `cache_dir` must be an emptyDir/tmpfs mount.
Each certificate (`certificate_id`) has its own folder (`<cache_dir>/<certificate_id>/`),
cache entry and lock, so two connections never rewrite the same files at the same
time. A cached certificate is reused while it names the same objects; when the slot
names other objects for the same id (the certificate was replaced), it is downloaded
again. `invalidate(certificate_id)` forces a fresh download (the pool calls it when a
secured connection fails, e.g. after the objects were replaced in MinIO)."""
def __init__(self, minio, cache_dir: str, metadata: dict | None = None):
self.minio = minio
self.cache_dir = cache_dir
self.metadata = metadata or {}
# ponytail: one Lock per certificate, never cleaned up — the set is small.
self._locks: dict[str, asyncio.Lock] = {}
self._paths: dict[str, tuple[tuple[tuple[str, str], ...], tuple[str, str]]] = {}
async def fetch(self, certificate: dict) -> tuple[str, str]:
"""Local paths of the cert and private key of `certificate` (slot format)."""
certificate_id = _certificate_id(certificate)
objects = _objects(certificate)
async with self._locks.setdefault(certificate_id, asyncio.Lock()):
cached = self._paths.get(certificate_id)
if cached is None or cached[0] != objects:
self._paths[certificate_id] = (
objects,
await self._download_all(certificate_id, objects),
)
return self._paths[certificate_id][1]
def invalidate(self, certificate_id) -> None:
self._paths.pop(str(certificate_id), None)
async def _download_all(
self, certificate_id: str, objects: tuple[tuple[str, str], ...]
) -> tuple[str, str]:
folder = Path(self.cache_dir) / certificate_id
folder.mkdir(parents=True, exist_ok=True)
(cert_bucket, cert_object), (key_bucket, key_object) = objects
cert_path = await self._download(cert_bucket, cert_object, folder / 'cert')
key_path = await self._download(key_bucket, key_object, folder / 'key')
return cert_path, key_path
async def _download(self, bucket: str, object_name: str, stem: Path) -> str:
data = await self.minio.download_file(object_name, bucket, self.metadata)
# The extension tells asyncua the format (.pem / .der)
local_path = stem.with_suffix(Path(object_name).suffix)
# Atomic replace: a connection loading the previous file never sees it half-written.
tmp_path = local_path.with_name(local_path.name + '.tmp')
tmp_path.write_bytes(data)
os.replace(tmp_path, local_path)
return str(local_path)
class LazyMinio:
"""Builds one MinIO client per bucket on first use. MinioRepository's constructor
hits the network (bucket check), so building it eagerly would let a MinIO outage at
pod start take the whole read path down. Built lazily, only the servers that need a
certificate fail — and they are retried on the next loop. One client per bucket,
because the constructor creates its bucket when missing: it is only ever given a
bucket a certificate names."""
def __init__(self, factory):
self._factory = factory
self._clients: dict[str, MinioRepository] = {}
async def download_file(self, object_name: str, bucket: str, metadata: dict):
if bucket not in self._clients:
self._clients[bucket] = self._factory(bucket)
return await self._clients[bucket].download_file(object_name, bucket, metadata)
def build_cert_provider(
config, logger, notification_handler, metrics_controller, metadata=None
) -> CertProvider:
"""CertProvider shared by the read path (IngestorManager) and the write path
(GatewayRuntime): one certificate cache for every connection."""
def minio_factory(bucket: str) -> MinioRepository:
return MinioRepository(
endpoint=getenv('MINIO_ENDPOINT', 'localhost:9000'),
access_key=getenv('MINIO_ACCESS_KEY', 'minioadmin'),
secret_key=getenv('MINIO_SECRET_KEY', 'minioadmin'),
secure=_env_bool('MINIO_SECURE', True),
bucket=bucket,
logger=logger,
notification_handler=notification_handler,
metrics_controller=metrics_controller,
)
return CertProvider(
minio=LazyMinio(minio_factory),
cache_dir=config.cert_cache_dir,
metadata=metadata,
)

19
ingestor/connection/clock.py
View File

@@ -1,19 +0,0 @@
import time
class MonotonicClock:
"""Single monotonic reference for latency/timeout math (1879 §2).
Wall-clock (`now_wall`) is only for human-facing timestamps; all elapsed/
timeout math must use `elapsed()` so NTP re-sync can't produce negatives.
"""
def __init__(self):
self._start = time.monotonic()
def elapsed(self) -> float:
return time.monotonic() - self._start
@staticmethod
def now_wall() -> float:
return time.time()

206
ingestor/connection/pool.py
View File

@@ -1,206 +0,0 @@
import asyncio
from asyncua import Client
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
from sientia_do.observability.logger import Logger
from sientia_do.observability.metrics_controller import MetricsController
from sientia_do.observability.sientia_monitoring import SientiaMonitoring
import ingestor.metrics as metrics
from ingestor.connection.certs import CertProvider
from ingestor.connection.clock import MonotonicClock
from ingestor.connection.security import apply_security, is_secured
class OpcConnectionPool(SientiaMonitoring):
"""Per-server OPC UA client cache (F0 capstone): reuses live connections,
evicts idle ones, and reconnects after a failed attempt (never caches it)."""
def __init__(
self,
cert_provider: CertProvider,
logger: Logger,
notification_handler: NotificationHandler,
metrics_controller: MetricsController,
):
self.cert_provider = cert_provider
self.clock = MonotonicClock()
self._connections: dict[str, tuple[Client, float]] = {}
# Active subscriptions per server. A subscribed connection only receives
# callbacks and never calls get(), so without a pin release_idle() would
# evict it as idle and silently stop the reads (SIENTIAPDE-1974 G4).
self._pins: dict[str, int] = {}
# Connection settings each pooled client was opened with (see _check_cfg).
self._configs: dict[str, dict] = {}
# ponytail: one Lock per server_id, never cleaned up — server set is
# small and bounded. Add eviction of stale locks only if it grows.
self._locks: dict[str, asyncio.Lock] = {}
SientiaMonitoring.__init__(
self,
logger=logger,
notification_handler=notification_handler,
metrics_controller=metrics_controller,
)
async def get(self, server_id: str, server_cfg: dict) -> Client:
"""Return the cached client for `server_id`, connecting one if needed.
Double-checked under a per-server lock so concurrent callers for the
same new server share one connection (no leaked, untracked sessions).
`server_id` is normalized to str, so 1 and '1' share the same session."""
server_id = str(server_id)
client = self._cached(server_id)
if client is not None:
self._check_cfg(server_id, server_cfg)
return client
async with self._locks.setdefault(server_id, asyncio.Lock()):
client = self._cached(server_id) # re-check: a racer may have connected
if client is not None:
self._check_cfg(server_id, server_cfg)
return client
return await self._open(server_id, server_cfg)
def _check_cfg(self, server_id: str, server_cfg: dict) -> None:
"""Warn when a caller asks for a server with other connection settings than
the pooled client was opened with. The client is reused as is: reconnecting
on every mismatch would let one malformed write command drop the reads.
Config changes from the slot (security, certificate) evict the client first:
the IngestorManager re-initializes the reader (ServerReader.shutdown)."""
opened = self._configs.get(server_id, {})
diff = [k for k in ('url', 'security', 'certificate') if opened.get(k) != server_cfg.get(k)]
if diff:
self.logger.warning(
f'OPC server {server_id}: reusing the pooled connection, but {diff} differ '
f'from the settings it was opened with'
)
def config_for(self, server_id: str) -> dict | None:
"""Connection settings of the pooled client for `server_id`, or None when the
server has no live pooled connection. Lets a writer reuse exactly the settings
the read path opened the connection with."""
server_id = str(server_id)
if server_id not in self._connections:
return None
cfg = self._configs.get(server_id)
return dict(cfg) if cfg is not None else None
def _cached(self, server_id: str) -> Client | None:
entry = self._connections.get(server_id)
if entry is None:
return None
client, _ = entry
self._connections[server_id] = (client, self.clock.elapsed())
return client
async def _open(self, server_id: str, server_cfg: dict) -> Client:
try:
client = await self._connect(server_cfg)
except Exception:
if is_secured(server_cfg):
# The cached certificate may be the cause (replaced in MinIO,
# rejected by the server): download it again on the next attempt.
self.cert_provider.invalidate(server_cfg['certificate']['certificate_id'])
await self.emit_metric(
metric_object=metrics.OPC_CONNECTIONS_FAILED,
tags={'pod_id': self.pod_id, 'server_name': server_id},
)
raise
self._connections[server_id] = (client, self.clock.elapsed())
self._configs[server_id] = dict(server_cfg)
await self._record_connected(server_id, server_cfg['url'])
return client
async def _connect(self, server_cfg: dict) -> Client:
client = Client(server_cfg['url'], timeout=10)
# Identify the pod on the OPC server, as the legacy OpcManager did: asyncua
# sends `name` as ApplicationName and "<description> Session<n>" as the
# session name, so each pod's session is recognizable in the server console.
client.name = self.pod_id
client.description = self.pod_id
pod_uri = self.pod_id.replace('-', ':')
client.product_uri = pod_uri
client.application_uri = pod_uri
# Security only with a client certificate (SIENTIAPDE-1974 D5); apply_security
# sets the certificate's application URI, which must match its SAN URI.
if is_secured(server_cfg):
certificate = server_cfg['certificate']
cert, key = await self.cert_provider.fetch(certificate)
await apply_security(client, server_cfg['security'], certificate, cert, key)
await client.connect()
return client
async def _record_connected(self, server_id: str, url: str):
await self.emit_metric(
metric_object=metrics.OPC_CONNECTIONS_TOTAL,
tags={'pod_id': self.pod_id, 'server_name': server_id},
)
await self.emit_metric(
metric_object=metrics.OPC_CONNECTION_STATUS,
method='set',
value=1,
tags={'pod_id': self.pod_id, 'server_name': server_id, 'server_url': url},
)
def pin(self, server_id: str) -> None:
"""Mark one more active subscription on `server_id` (never evicted as idle)."""
server_id = str(server_id)
self._pins[server_id] = self._pins.get(server_id, 0) + 1
def unpin(self, server_id: str) -> None:
"""Drop one active subscription mark from `server_id`."""
server_id = str(server_id)
count = self._pins.get(server_id, 0) - 1
if count > 0:
self._pins[server_id] = count
else:
self._pins.pop(server_id, None)
async def release_idle(self, max_idle_s: float):
"""Disconnect and drop clients unused for longer than `max_idle_s`,
except the ones pinned by an active subscription."""
now = self.clock.elapsed()
idle_ids = [
sid
for sid, (_, last_used) in self._connections.items()
if now - last_used > max_idle_s and not self._pins.get(sid)
]
for server_id in idle_ids:
await self._evict(server_id)
async def evict(self, server_id: str):
"""Disconnect `server_id` now (lost server or changed config); the next
get() reconnects. No-op when the server has no pooled client."""
server_id = str(server_id)
if server_id in self._connections:
await self._evict(server_id)
async def close_all(self):
"""Disconnect every pooled client (e.g. on shutdown)."""
# list() snapshots the keys: _evict() pops from self._connections,
# so iterating the dict directly would raise "changed size during iteration".
for server_id in list(self._connections): # NOSONAR
await self._evict(server_id)
async def _evict(self, server_id: str):
client, _ = self._connections.pop(server_id)
self._configs.pop(server_id, None)
await self._disconnect_with_retries(client, server_id)
await self.emit_metric(
metric_object=metrics.OPC_DISCONNECTIONS_TOTAL,
tags={'pod_id': self.pod_id, 'server_name': server_id},
)
async def _disconnect_with_retries(self, client: Client, server_id: str) -> None:
"""Tries 5 times to disconnect, with a delay of 100ms x try (mirrors
the legacy OpcManager.disconnection_fallback)."""
for i in range(5):
try:
await client.disconnect()
return
except Exception as e:
self.logger.error(
f'Failed to disconnect from {server_id}, attempt {i + 1} of 5: {e}'
)
await asyncio.sleep(0.1 * i)

65
ingestor/connection/security.py
View File

@@ -1,65 +0,0 @@
from asyncua import ua
from asyncua.crypto.security_policies import (
SecurityPolicyAes128Sha256RsaOaep,
SecurityPolicyAes256Sha256RsaPss,
SecurityPolicyBasic128Rsa15,
SecurityPolicyBasic256,
SecurityPolicyBasic256Sha256,
)
SESSION_TIMEOUT_MS = 600_000 # 10 min; tune per server (Samarco: long sessions degrade)
# connectors.security.policy / .mode (SIENTIAPDE-1974 D4) to the asyncua classes
POLICIES = {
'Basic256Sha256': SecurityPolicyBasic256Sha256,
'Aes128Sha256RsaOaep': SecurityPolicyAes128Sha256RsaOaep,
'Aes256Sha256RsaPss': SecurityPolicyAes256Sha256RsaPss,
'Basic256': SecurityPolicyBasic256,
'Basic128Rsa15': SecurityPolicyBasic128Rsa15,
}
MODES = {
'Sign': ua.MessageSecurityMode.Sign,
'SignAndEncrypt': ua.MessageSecurityMode.SignAndEncrypt,
}
class UnsupportedSecurityError(ValueError):
"""The server's security mode or policy has no asyncua counterpart."""
def is_secured(server_cfg: dict) -> bool:
"""Whether a connection applies security: only with a client certificate (D5), and
not when its mode is 'None'. Without one, it connects without security, and a server
that requires a certificate refuses the connection."""
security = server_cfg.get('security') or {}
return server_cfg.get('certificate') is not None and security.get('mode') != 'None'
async def apply_security(client, security, certificate, cert_path, private_key_path):
"""Apply the server's mode + policy with the connector's client certificate, and set
channel > session lifetime (1879 §3: secure_channel_timeout = session_timeout * 1.25).
The application URI is the certificate's (its SAN URI): the server rejects a
different one (KEPServer: BadCertificateUriInvalid).
Raises:
UnsupportedSecurityError: When the mode or policy is unknown.
"""
policy = POLICIES.get(security.get('policy'))
mode = MODES.get(security.get('mode'))
if policy is None or mode is None:
raise UnsupportedSecurityError(
f'Unsupported OPC security mode={security.get("mode")!r} '
f'policy={security.get("policy")!r} (modes: {", ".join(MODES)}; '
f'policies: {", ".join(POLICIES)})'
)
client.application_uri = certificate['application_uri']
await client.set_security(
policy,
certificate=cert_path,
private_key=private_key_path,
server_certificate=None,
mode=mode,
)
client.session_timeout = SESSION_TIMEOUT_MS
client.secure_channel_timeout = int(SESSION_TIMEOUT_MS * 1.25)

126
ingestor/connection/servers.py
View File

@@ -1,126 +0,0 @@
"""Connection settings of the OPC server a write command targets (SIENTIAPDE-1975 GW1).
The Laborious only knows the `server_id` of a write tag (pipeline `write_tags`); the
connection (url, security, certificate) belongs to the gateway, which owns every OPC
session. A server already in the pool is written through that connection. Otherwise it
is resolved from the tag database (`connectors` → `opc_certificates`) with the rule the
Orchestrator uses to build the read slots (SIENTIAPDE-1974 D1), in the slot format.
"""
from typing import Any
from sientia_do.observability.logger import Logger
from ingestor.connection.certificates import (
CERTIFICATE_LOOKUP,
client_certificate_id,
has_security,
resolve_certificate,
)
from ingestor.connection.clock import MonotonicClock
from ingestor.connection.pool import OpcConnectionPool
CONNECTORS = 'connectors'
def _connector_pipeline(server_id: str) -> list[dict[str, Any]]:
"""The active OPC UA connector `server_id` (connector_id is its slug), with its
certificate. Only the fields used are projected."""
return [
{'$match': {'connector_id': server_id, 'type': 'opc_ua', 'status': 'Active'}},
{'$limit': 1},
CERTIFICATE_LOOKUP,
{
'$project': {
'_id': 0,
'connector_id': 1,
'url': 1,
'subscription_period_ms': 1,
'security.mode': 1,
'security.policy': 1,
'security.client_certificate_id': 1,
'certificates': 1,
}
},
]
class ServerConfigResolver:
"""Resolve and cache the connection settings of an OPC server by `server_id`.
Order: the live pooled connection (what the read path opened, so writes never fight
the reads over settings), then MongoDB. Lookups are cached for `ttl_s`; a server
that is not found, for `missing_ttl_s`, so a pipeline pointing at a removed server
does not query MongoDB on every command.
# ponytail: the MongoDB fallback goes away once write commands are routed to the pod
# that owns the connection (SIENTIAPDE-1974 D1, new card).
"""
def __init__(
self,
pool: OpcConnectionPool,
mongo,
logger: Logger,
ttl_s: float = 60.0,
missing_ttl_s: float = 10.0,
metadata: dict | None = None,
):
self.pool = pool
self.mongo = mongo
self.logger = logger
self.ttl_s = ttl_s
self.missing_ttl_s = missing_ttl_s
self.metadata = metadata or {}
self.clock = MonotonicClock()
self._cache: dict[str, tuple[dict[str, Any] | None, float]] = {}
async def resolve(self, server_id: str) -> dict[str, Any] | None:
"""Connection settings for `server_id`, or None when no source knows it.
Raises:
Exception: When MongoDB cannot be read (the caller reports the write as
failed).
"""
server_id = str(server_id)
pooled = self.pool.config_for(server_id)
if pooled is not None:
return pooled
now = self.clock.elapsed()
cached = self._cache.get(server_id)
if cached is not None:
cfg, cached_at = cached
if now - cached_at < (self.ttl_s if cfg is not None else self.missing_ttl_s):
return dict(cfg) if cfg is not None else None
cfg = await self._lookup(server_id)
self._cache[server_id] = (cfg, now)
return dict(cfg) if cfg is not None else None
async def _lookup(self, server_id: str) -> dict[str, Any] | None:
connectors = await self.mongo.aggregate(
CONNECTORS, _connector_pipeline(server_id), self.metadata
)
connector = connectors[0] if connectors else None
if connector is None or not connector.get('url') or not has_security(connector):
self.logger.warning(
f'OPC server {server_id} not found, not active or incomplete (url, '
f'security.mode, security.policy) in {CONNECTORS}'
)
return None
certificate = resolve_certificate(connector)
if certificate is None and client_certificate_id(connector) is not None:
self.logger.warning(
f'OPC server {server_id}: client_certificate_id '
f'{client_certificate_id(connector)} has no active client certificate in '
f'opc_certificates; connecting without security'
)
return {
'server_id': server_id,
'url': connector['url'],
'subscription_period_ms': connector.get('subscription_period_ms'),
'security': {
'mode': connector['security']['mode'],
'policy': connector['security']['policy'],
},
'certificate': certificate,
}

44
ingestor/converters.py
View File

@@ -1,44 +0,0 @@
"""Safe coercion of JSON-transported values into typed OPC write values.
Fixes 1879 BUG-1 (bool('false') is True in Python) and BUG-2 (uncaught
TypeError/ValueError from int/float on bad input).
"""
_FALSE_STRINGS = {'false', '0', 'no', 'off', ''}
class ConversionError(ValueError):
"""Raised when a raw value cannot be coerced to the requested type."""
def _to_bool(raw) -> bool:
if isinstance(raw, bool):
return raw
if isinstance(raw, (int, float)):
return bool(raw)
return str(raw).strip().lower() not in _FALSE_STRINGS
def to_typed_value(raw, data_type: str):
"""Coerce raw (often a JSON string) to the given data_type.
data_type: 'boolean' | 'int' | 'float' | 'double' | 'string'.
Raises ConversionError on any invalid input (never a bare TypeError).
"""
if not isinstance(data_type, str):
raise ConversionError(f'data_type must be a string, got {type(data_type).__name__}')
dtype = data_type.strip().lower()
try:
if dtype in ('bool', 'boolean'):
return _to_bool(raw)
if dtype in ('int', 'integer'):
if isinstance(raw, str) and ('.' in raw or raw.strip() == ''):
raise ConversionError(f'invalid int: {raw!r}')
return int(raw)
if dtype in ('float', 'double', 'real'):
return float(raw)
if dtype in ('str', 'string'):
return str(raw)
except (TypeError, ValueError) as e:
raise ConversionError(f'cannot convert {raw!r} to {data_type}: {e}') from e
raise ConversionError(f'unknown data_type: {data_type!r}')

0
ingestor/engines/__init__.py
View File

132
ingestor/engines/active_read.py
View File

@@ -1,132 +0,0 @@
"""Active OPC reads (F3): on-demand (`read_now`, used by the gRPC server -
Task 16) and scheduled (`scheduled_tick`, driven by the app main loop each
collect cycle). Emits metrics only -- unlike WriteEngine there is no
notification path here."""
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
from sientia_do.observability.logger import Logger
from sientia_do.observability.metrics_controller import MetricsController
from sientia_do.observability.sientia_monitoring import SientiaMonitoring
from sientia_do.temporal.constants import DATETIME_FORMAT_WITH_TZ, OPC_TIMEZONE
import ingestor.metrics as metrics
from ingestor.connection.clock import MonotonicClock
from ingestor.connection.pool import OpcConnectionPool
from ingestor.io.data_sink import DataSink
class ActiveReadEngine(SientiaMonitoring):
"""On-demand + scheduled OPC reads, sharing the pool/sink with WriteEngine
and SubscriptionEngine."""
def __init__(
self,
pool: OpcConnectionPool,
sink: DataSink,
logger: Logger,
notification_handler: NotificationHandler,
metrics_controller: MetricsController,
):
self.pool = pool
self.sink = sink
self.clock = MonotonicClock()
self._ticks: dict[tuple[str, str], int] = {}
SientiaMonitoring.__init__(
self,
logger=logger,
notification_handler=notification_handler,
metrics_controller=metrics_controller,
)
async def read_now(
self,
server_id: str,
addr: str,
names: list[dict] | None = None,
server_cfg: dict | None = None,
) -> dict:
"""Read `addr` on `server_id` right now; persist via sink only if `names` given."""
tags = {'pod_id': self.pod_id, 'server_name': server_id}
start = self.clock.elapsed()
try:
client = await self.pool.get(server_id, server_cfg or {})
dv = await client.get_node(addr).read_data_value()
except Exception as e:
await self.emit_metric(
metric_object=metrics.OPC_ACTIVE_READ_TOTAL, tags={**tags, 'result': 'error'}
)
self.logger.error(f'Active read failed for {server_id}:{addr}: {e}')
raise
await self.emit_metric(
metric_object=metrics.OPC_ACTIVE_READ_LATENCY,
method='observe',
value=self.clock.elapsed() - start,
tags=tags,
)
await self.emit_metric(
metric_object=metrics.OPC_ACTIVE_READ_TOTAL, tags={**tags, 'result': 'success'}
)
return await self._build_result(dv, addr, names, tags)
async def _build_result(self, dv, addr: str, names: list[dict] | None, tags: dict) -> dict:
value, status_name, source_ts, server_ts = self._extract(dv)
await self._record_read_lag(source_ts, tags)
source_str = self._format_ts(source_ts)
persisted = bool(names)
if names:
await self.sink.publish_reading(
tag=addr, value=value, timestamp=source_str, names=names
)
return {
'value': value,
'opc_status_code': status_name,
'source_timestamp': source_str,
'server_timestamp': self._format_ts(server_ts),
'persisted': persisted,
}
@staticmethod
def _extract(dv):
status_name = dv.StatusCode.name if dv.StatusCode is not None else None
return dv.Value.Value, status_name, dv.SourceTimestamp, dv.ServerTimestamp
@staticmethod
def _format_ts(ts) -> str | None:
if ts is None:
return None
return ts.replace(tzinfo=OPC_TIMEZONE).strftime(DATETIME_FORMAT_WITH_TZ)
async def _record_read_lag(self, source_ts, tags: dict) -> None:
if source_ts is None:
return
lag = MonotonicClock.now_wall() - source_ts.replace(tzinfo=OPC_TIMEZONE).timestamp()
await self.emit_metric(
metric_object=metrics.OPC_READ_LAG, method='set', value=lag, tags=tags
)
async def scheduled_tick(self, slot: dict) -> None:
"""Sweep every server/tag in a parsed slot, reading tags due for an active read."""
for server in slot.values():
server_id = server['server_id']
server_cfg = {k: v for k, v in server.items() if k != 'tags'}
for addr, tag in server['tags'].items():
await self._tick_tag(server_id, addr, tag, server_cfg)
async def _tick_tag(self, server_id: str, addr: str, tag: dict, server_cfg: dict) -> None:
active_read = tag.get('active_read') or {}
if not active_read.get('enabled'):
return
if not self._should_read(server_id, addr, active_read.get('cycle_multiple', 1)):
return
try:
await self.read_now(server_id, addr, names=tag['names'], server_cfg=server_cfg)
except Exception as e:
self.logger.error(f'Scheduled active read failed for {server_id}:{addr}: {e}')
def _should_read(self, server_id: str, addr: str, cycle_multiple: int) -> bool:
key = (server_id, addr)
count = self._ticks.get(key, 0) + 1
self._ticks[key] = count
return count % max(cycle_multiple, 1) == 0

221
ingestor/engines/subscription.py
View File

@@ -1,221 +0,0 @@
import json
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
from sientia_do.notifications.models import NotificationLevel
from sientia_do.observability.logger import Logger
from sientia_do.observability.metrics_controller import MetricsController
from sientia_do.observability.sientia_monitoring import SientiaMonitoring
from sientia_do.temporal.constants import DATETIME_FORMAT_WITH_TZ, OPC_TIMEZONE
import ingestor.metrics as metrics
from ingestor.connection.clock import MonotonicClock
from ingestor.connection.pool import OpcConnectionPool
from ingestor.io.data_sink import DataSink
class SubscriptionEngine(SientiaMonitoring):
"""Adapts OpcManager's subscription/datachange path (F1 capstone) to the
shared OpcConnectionPool and the multi-name slot v2 structure: on data
change it fans one reading out to every configured name via DataSink,
instead of the old flat per-`topics` publish loop."""
def __init__(
self,
pool: OpcConnectionPool,
sink: DataSink,
logger: Logger,
notification_handler: NotificationHandler,
metrics_controller: MetricsController,
metadata: dict | None = None,
):
self.pool = pool
self.sink = sink
self.metadata = metadata or {}
self.nodes: dict = {}
self.subscriptions: dict = {}
self._clients: dict = {}
self._servers: dict = {}
# Addresses each subscription monitors, so unsubscribe can forget them
# (otherwise check_cycles keeps alarming on tags nobody reads anymore).
self._sub_nodes: dict[str, set[str]] = {}
self.non_receive_count = 0
SientiaMonitoring.__init__(
self,
logger=logger,
notification_handler=notification_handler,
metrics_controller=metrics_controller,
)
async def create_subscription(self, server_id: str, server_cfg: dict, name: str):
"""Get a client for `server_id` from the pool and create a subscription
named `name` on it, with the engine itself as the datachange handler."""
client = await self.pool.get(server_id, server_cfg)
try:
self.subscriptions[name] = await client.create_subscription(
server_cfg['subscription_period_ms'], self
)
except Exception as e:
self.logger.error(f'Failed to create subscription {name} on {server_id}: {e}')
raise
self._clients[name] = client
self._servers[name] = server_id
self.pool.pin(server_id)
self.logger.info(f'Subscription {name} created on {server_id}.')
await self.emit_metric(
metric_object=metrics.OPC_SUBSCRIPTIONS_CREATED,
method='inc',
value=1,
tags={'pod_id': self.pod_id, 'server_name': server_id, 'slot_name': name},
)
async def subscribe(self, subscription: str, nodes: dict, collect_period: int):
"""Register `nodes` (slot v2 tags: frequency + names + active_read) under
`subscription` and start monitoring them for data changes."""
if not self.subscriptions.get(subscription):
raise ValueError('Subscription not created. Call create_subscription first.')
client = self._clients[subscription]
server_id = self._servers[subscription]
addr_nodes = [client.get_node(n) for n in nodes]
self.nodes.update(nodes)
self._sub_nodes.setdefault(subscription, set()).update(nodes)
self._init_cycle_rules(nodes, collect_period, server_id)
await self.subscriptions[subscription].subscribe_data_change(addr_nodes)
await self.emit_metric(
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
method='set',
value=len(self.nodes),
tags={'pod_id': self.pod_id, 'server_name': server_id},
)
def _init_cycle_rules(self, nodes: dict, collect_period: int, server_id: str):
for node in nodes:
self.nodes[node]['server_name'] = server_id
self.nodes[node]['cycle_rule'] = {
'cycle_increment': collect_period * 1000 / float(self.nodes[node]['frequency']),
'cycle_count': 0,
}
async def unsubscribe(self, subscription: str):
if not self.subscriptions.get(subscription):
self.logger.warning(f"Subscription '{subscription}' not found. Cannot unsubscribe.")
return
sub = self.subscriptions.pop(subscription)
self._clients.pop(subscription, None)
server_id = self._servers.pop(subscription, None)
if server_id is not None:
self.pool.unpin(server_id)
self._forget_nodes(subscription)
try:
await sub.delete()
except Exception as e:
# The connection may already be gone (lost server); the subscription is
# dropped from the books either way so the slot can be re-subscribed.
self.logger.warning(f'Failed to delete subscription {subscription}: {e}')
self.logger.info(f'Unsubscribed from {subscription}.')
if server_id is not None:
await self.emit_metric(
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
method='set',
value=len(self.nodes),
tags={'pod_id': self.pod_id, 'server_name': server_id},
)
def _forget_nodes(self, subscription: str) -> None:
"""Drop the subscription's addresses, except those another subscription
of this engine still monitors."""
nodes = self._sub_nodes.pop(subscription, set())
still_monitored = set().union(*self._sub_nodes.values())
for node in nodes:
if node not in still_monitored:
self.nodes.pop(node, None)
async def datachange_notification(self, node, _val, data):
"""asyncua subscription callback: fan the new value out to every name
configured for this address, then reset cycle/lag bookkeeping."""
value = data.monitored_item.Value.Value.Value
source_timestamp = data.monitored_item.Value.SourceTimestamp.replace(tzinfo=OPC_TIMEZONE)
tag = str(node)
config = self.nodes[tag]
await self.sink.publish_reading(
tag=tag,
value=value,
timestamp=source_timestamp.strftime(DATETIME_FORMAT_WITH_TZ),
names=config['names'],
)
await self._record_read_lag(config, source_timestamp)
await self._reset_cycle_state(config)
async def _record_read_lag(self, config: dict, source_timestamp):
lag_seconds = MonotonicClock.now_wall() - source_timestamp.timestamp()
await self.emit_metric(
metric_object=metrics.OPC_READ_LAG,
method='set',
value=lag_seconds,
tags={'pod_id': self.pod_id, 'server_name': config.get('server_name', 'unknown')},
)
async def _reset_cycle_state(self, config: dict):
config['cycle_rule']['cycle_count'] = 0
self.non_receive_count = 0
await self.emit_metric(
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
method='set',
value=0,
tags={'pod_id': self.pod_id, 'server_name': config.get('server_name', 'unknown')},
)
async def check_cycles(self):
"""Increment every node's cycle count and warn on nodes stuck >= 5 cycles."""
for node, config in self.nodes.items():
config['cycle_rule']['cycle_count'] += config['cycle_rule']['cycle_increment']
if config['cycle_rule']['cycle_count'] >= 5:
await self._notify_node_stopped(node, config)
async def _notify_node_stopped(self, node: str, config: dict):
cycles = config['cycle_rule']['cycle_count']
names = ','.join(n['tag_name'] for n in config['names'])
await self.send_notification_async(
metadata=self.metadata,
notification_id=f'TAG_{node}:{names}_LISTENNING_STOPPED',
message=f'{cycles} cycles without receive from {node}:{names}',
block='subscription_engine',
level=NotificationLevel.WARNING,
)
async def check_opc_listenning(self, server_name: str) -> bool:
"""Track cycles without any datachange for `server_name`; warn at 5,
error + signal reconnect at 15. `server_name` is explicit (unlike
OpcManager's `self.name`) since one engine can serve several servers."""
self.non_receive_count += 1
await self.emit_metric(
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
method='set',
value=self.non_receive_count,
tags={'pod_id': self.pod_id, 'server_name': server_name},
)
if self.non_receive_count >= 5:
await self.send_notification_async(
metadata=self.metadata,
notification_id=f'OPC_LISTENNING_STOPPED__{server_name}',
message=f'{self.non_receive_count} cycles without receive from OPC '
f'{server_name}. Tags: {json.dumps(self.nodes, default=str)}',
block='subscription_engine',
level=NotificationLevel.ERROR,
)
if self.non_receive_count >= 15:
await self.emit_metric(
metric_object=metrics.OPC_RECONNECTIONS_TOTAL,
tags={'pod_id': self.pod_id, 'server_name': server_name},
)
await self.send_notification_async(
metadata=self.metadata,
notification_id=f'OPC_CONNECTION_RETRY__{server_name}',
message=f'Retrying to connect to server {server_name}',
block='subscription_engine',
level=NotificationLevel.ERROR,
)
return True
return False

228
ingestor/engines/write.py
View File

@@ -1,228 +0,0 @@
"""OPC write execution (F2 capstone) from a NATS C2 command dict.
Fixes 1879 correctness gaps: safe value coercion (never raises on bad
input), StatusCode/SourceTimestamp fidelity on the DataValue actually sent
to the server, a rate-limit floor between writes, and OPC error typing.
"""
import asyncio
from dataclasses import dataclass
from asyncua import ua
from asyncua.ua.uaerrors import UaStatusCodeError
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
from sientia_do.observability.logger import Logger
from sientia_do.observability.metrics_controller import MetricsController
from sientia_do.observability.sientia_monitoring import SientiaMonitoring
import ingestor.metrics as metrics
from ingestor.connection.clock import MonotonicClock
from ingestor.connection.pool import OpcConnectionPool
from ingestor.connection.servers import ServerConfigResolver
from ingestor.converters import ConversionError, to_typed_value
from ingestor.io.data_sink import DataSink
_VARIANT_TYPES = {
'float': ua.VariantType.Float,
'double': ua.VariantType.Double,
'real': ua.VariantType.Double,
'int': ua.VariantType.Int32,
'integer': ua.VariantType.Int32,
'bool': ua.VariantType.Boolean,
'boolean': ua.VariantType.Boolean,
'str': ua.VariantType.String,
'string': ua.VariantType.String,
}
# ponytail: same reconnect-worthy Bad*-status subset as laborious's
# opc_repository.py (different repo, not importable here); keep in sync by
# hand or promote to a shared lib if it drifts.
_SESSION_BAD_NAMES = frozenset(
{
'BadSessionIdInvalid',
'BadSessionClosed',
'BadSessionNotActivated',
'BadSecureChannelIdInvalid',
'BadSecureChannelClosed',
'BadSecureChannelTokenUnknown',
'BadTcpSecureChannelUnknown',
'BadServerNotConnected',
'BadConnectionClosed',
'BadConnectionRejected',
'BadNotConnected',
'BadTimeout',
'BadRequestTimeout',
'BadTooManySessions',
}
)
@dataclass
class WriteResult:
ok: bool
status: str # 'ok' | 'error' | 'session_bad'
opc_status_code: str
message: str
class WriteEngine(SientiaMonitoring):
"""Executes one OPC write from a NATS C2 command."""
def __init__(
self,
pool: OpcConnectionPool,
sink: DataSink,
logger: Logger,
notification_handler: NotificationHandler,
metrics_controller: MetricsController,
write_timestamp: bool = True,
write_quality: bool = True,
min_pause_s: float = 0.0,
servers: ServerConfigResolver | None = None,
):
self.pool = pool
self.sink = sink
# Resolves the server connection by server_id (SIENTIAPDE-1975 GW1). Without it,
# no server is known and every write fails as server_not_found.
self.servers = servers
self.write_timestamp = write_timestamp
self.write_quality = write_quality
self.min_pause_s = min_pause_s
self.clock = MonotonicClock()
self._last_write_at: float | None = None
SientiaMonitoring.__init__(
self,
logger=logger,
notification_handler=notification_handler,
metrics_controller=metrics_controller,
)
async def write(self, cmd: dict) -> WriteResult:
await self._rate_limit()
missing = [k for k in ('value', 'data_type', 'server_id', 'addr') if k not in cmd]
if missing:
return await self._emit_error(cmd, 'bad_command', f'missing fields: {missing}', 'error')
try:
typed = to_typed_value(cmd['value'], cmd['data_type'])
except ConversionError as e:
return await self._emit_error(cmd, 'conversion', str(e), 'error')
try:
dv = self._build_datavalue(cmd, typed)
except (KeyError, TypeError, ValueError) as e:
# write_timestamp on with a missing or non-ISO timestamp.
return await self._emit_error(cmd, 'bad_timestamp', repr(e), 'error')
try:
server_cfg = await self._server_cfg(cmd)
except Exception as e:
return await self._emit_error(cmd, 'server_lookup', str(e), 'error')
if server_cfg is None:
return await self._emit_error(
cmd, 'server_not_found', f'OPC server {cmd["server_id"]} is not configured', 'error'
)
start = self.clock.elapsed()
try:
client = await self.pool.get(cmd['server_id'], server_cfg)
node = client.get_node(cmd['addr'])
await node.write_value(dv)
except Exception as e:
status = 'session_bad' if self._is_session_bad(e) else 'error'
return await self._emit_error(cmd, type(e).__name__, str(e), status)
return await self._emit_success(cmd, start)
async def _rate_limit(self) -> None:
now = self.clock.elapsed()
if self.min_pause_s > 0 and self._last_write_at is not None:
remaining = self.min_pause_s - (now - self._last_write_at)
if remaining > 0:
await asyncio.sleep(remaining)
now = self.clock.elapsed()
self._last_write_at = now
async def _server_cfg(self, cmd: dict) -> dict | None:
"""Connection settings of the target server, resolved by server_id (live pooled
connection, then connectors → opc_certificates). The command carries only the
server_id (C2)."""
if self.servers is None:
return None
return await self.servers.resolve(str(cmd['server_id']))
def _variant_type(self, data_type: str) -> ua.VariantType:
# direct index: unknown dtypes already became ConversionError in to_typed_value
return _VARIANT_TYPES[data_type.strip().lower()]
def _build_datavalue(self, cmd: dict, typed_value) -> ua.DataValue:
variant = ua.Variant(typed_value, self._variant_type(cmd['data_type']))
write_quality = cmd.get('write_quality', self.write_quality)
write_timestamp = cmd.get('write_timestamp', self.write_timestamp)
# NOTE: DataValue.StatusCode default_factory populates a Good status
# even when omitted, so "off" must pass StatusCode=None explicitly
# or the flag silently no-ops.
status_code = ua.StatusCode(ua.UInt32(ua.StatusCodes.Good)) if write_quality else None
source_ts = ua.DateTime.fromisoformat(cmd['timestamp']) if write_timestamp else None
return ua.DataValue(variant, StatusCode=status_code, SourceTimestamp=source_ts)
def _is_session_bad(self, exc: Exception) -> bool:
return isinstance(exc, UaStatusCodeError) and type(exc).__name__ in _SESSION_BAD_NAMES
async def _emit_error(self, cmd: dict, error_type: str, message: str, status: str):
# The reason only lives here: the row gets write_status alone (SIENTIAPDE-1975).
self.logger.error(
f'OPC write failed ({status}, {error_type}): server={cmd.get("server_id")} '
f'addr={cmd.get("addr")} model_id={cmd.get("model_id")} '
f'timestamp={cmd.get("timestamp")}: {message}'
)
tags = {'pod_id': self.pod_id, 'server_name': cmd.get('server_id', 'unknown')}
await self.emit_metric(
metric_object=metrics.OPC_ERROR_BY_TYPE, tags={**tags, 'error_type': error_type}
)
await self.emit_metric(
metric_object=metrics.OPC_WRITE_TOTAL,
tags={**tags, 'model_id': cmd.get('model_id', 'unknown'), 'result': 'error'},
)
# Same series as a success (status='ok'), so it always shows the last outcome.
await self.emit_metric(
metric_object=metrics.OPC_WRITE_STATUS_BY_MODEL,
method='set',
value=0,
tags={
'pod_id': self.pod_id,
'model_id': cmd.get('model_id', 'unknown'),
'status': 'ok',
},
)
result = WriteResult(ok=False, status=status, opc_status_code=error_type, message=message)
await self.sink.update_write_status(cmd, result)
return result
async def _emit_success(self, cmd: dict, start: float):
tags = {'pod_id': self.pod_id, 'server_name': cmd.get('server_id', 'unknown')}
await self.emit_metric(
metric_object=metrics.OPC_WRITE_TOTAL,
tags={**tags, 'model_id': cmd.get('model_id', 'unknown'), 'result': 'success'},
)
await self.emit_metric(
metric_object=metrics.OPC_WRITE_LATENCY,
method='observe',
value=self.clock.elapsed() - start,
tags=tags,
)
await self.emit_metric(
metric_object=metrics.OPC_WRITE_STATUS_BY_MODEL,
method='set',
value=1,
tags={
'pod_id': self.pod_id,
'model_id': cmd.get('model_id', 'unknown'),
'status': 'ok',
},
)
result = WriteResult(ok=True, status='ok', opc_status_code='Good', message='')
await self.sink.update_write_status(cmd, result)
return result

148
ingestor/gateway_runtime.py
View File

@@ -1,148 +0,0 @@
"""GatewayRuntime (Task 18 F4 capstone): builds and owns the write/active-read/gRPC
object graph and wires it into the app's lifecycle.
Reuses the existing ingestor's logger/notification_handler/metrics_controller and
its already-connected DataManager. The OpcConnectionPool (and its CertProvider, which
downloads each server's client certificate from MinIO) belongs to the read path in IngestorManager (SIENTIAPDE-1974):
reads start in prepare_ingestor, before this runtime exists, and must keep running
if anything here fails. The runtime only borrows the pool for writes and gRPC.
# ponytail: scheduled_tick + gRPC slot_lookup wiring land in the next 1974 RC.
"""
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
from sientia_do.notifications.models import NotificationLevel
from sientia_do.observability.logger import Logger
from sientia_do.observability.metrics_controller import MetricsController
from sientia_do.observability.sientia_monitoring import SientiaMonitoring
from sientia_do.temporal.activities.postgres import Postgres
from sientia_do.utils.connectors_config import build_postgres_config
from ingestor.adapters.grpc_server import GrpcServer
from ingestor.adapters.nats_consumer import NatsWriteConsumer
from ingestor.config import GatewayConfig
from ingestor.connection.pool import OpcConnectionPool
from ingestor.connection.servers import ServerConfigResolver
from ingestor.engines.active_read import ActiveReadEngine
from ingestor.engines.write import WriteEngine
from ingestor.io.data_sink import DataSink
from ingestor.managers.data_manager import DataManager
class GatewayRuntime(SientiaMonitoring):
"""Owns the gateway write/active-read/gRPC object graph. Resilient to NATS
being unreachable at startup: the read path (existing subscription loop,
gRPC on-demand reads) must keep working even if the write path can't
connect yet -- so a NATS connect failure is logged/notified, not raised,
and retried from `idle_sweep()` on each main-loop iteration."""
def __init__(
self,
config: GatewayConfig,
data_manager: DataManager,
pool: OpcConnectionPool,
logger: Logger,
notification_handler: NotificationHandler,
metrics_controller: MetricsController,
metadata: dict | None = None,
):
SientiaMonitoring.__init__(self, logger, notification_handler, metrics_controller)
self.config = config
self.metadata = metadata or {}
self._nats_started = False
self._nats_notified = False # notify once per outage, not per 5s retry
self.pool = pool # owned by IngestorManager (read path); closed there
postgres = Postgres(
**build_postgres_config(),
logger=logger,
notification_handler=notification_handler,
metrics_controller=metrics_controller,
)
self.sink = DataSink(data_manager, postgres)
# Write commands carry only the server_id (SIENTIAPDE-1975): the connection is
# resolved here, from the live pooled connection or the connectors →
# opc_certificates documents, with the rule the orchestrator uses for the read
# slots (SIENTIAPDE-1974 D1).
self.servers = ServerConfigResolver(
self.pool, data_manager.mongo_repository, logger, metadata=self.metadata
)
self.write_engine = WriteEngine(
self.pool,
self.sink,
logger,
notification_handler,
metrics_controller,
write_timestamp=config.write_timestamp,
write_quality=config.write_quality,
min_pause_s=config.write_min_pause_s,
servers=self.servers,
)
self.nats_consumer = NatsWriteConsumer(
config.nats_url,
self.write_engine,
logger,
subject=config.nats_write_subject,
queue=config.nats_queue,
)
self.active_read = ActiveReadEngine(
self.pool, self.sink, logger, notification_handler, metrics_controller
)
# slot_lookup=None for now: address -> names resolution needs slot v2
# (1974); until then ReadTag responses just aren't persisted.
self.grpc_server = (
GrpcServer(self.active_read, logger, port=config.grpc_port, slot_lookup=None)
if config.grpc_enabled
else None
)
async def start(self) -> None:
await self._start_nats()
if self.grpc_server is not None:
await self.grpc_server.start()
self.logger.info(
f'GatewayRuntime started: nats_started={self._nats_started} '
f'grpc_enabled={self.grpc_server is not None}'
)
async def stop(self) -> None:
if self.grpc_server is not None:
await self.grpc_server.stop()
if self._nats_started:
await self.nats_consumer.stop()
self._nats_started = False
# The pool is not closed here: IngestorManager.shutdown() owns it and runs
# after this, once its readers have dropped their subscriptions.
async def idle_sweep(self) -> None:
await self.pool.release_idle(self.config.idle_conn_max_s)
if self._nats_started and self.nats_consumer.closed:
# Reconnection is unbounded once connected; a closed connection means it
# was closed for good (e.g. by the server), so subscribe again.
self.logger.warning('GatewayRuntime: NATS connection closed; starting again')
self._nats_started = False
if not self._nats_started:
await self._start_nats()
async def _start_nats(self) -> None:
if self._nats_started:
return
try:
await self.nats_consumer.start()
self._nats_started = True
self._nats_notified = False
except Exception as e:
self.logger.error(f'GatewayRuntime: failed to start NATS write consumer: {e}')
if self._nats_notified:
return
self._nats_notified = True
await self.send_notification_async(
metadata=self.metadata,
notification_id='GATEWAY_NATS_CONSUMER_START_FAILED',
message=f'Failed to start NATS write consumer: {e}',
block='gateway_runtime',
level=NotificationLevel.ERROR,
)

6
ingestor/ingestor.py
View File

@@ -268,14 +268,12 @@ class Ingestor(SientiaMonitoring):
overleases = list(self.ingestor_manager.managed_tags.keys())[1:]
# Unsubscribe before giving the leases back: another pod may take a released
# slot right away, and must not find this pod still reading it
await self.ingestor_manager.drop_slot_leases(overleases)
for lease in overleases:
await self.ingestor_manager.unsubscribe_slot(lease)
self.ingestor_manager.managed_tags.pop(lease)
await self.ingestor_manager.drop_slot_leases(overleases)
await self.emit_metric(
metric_object=metrics.SLOTS_MANAGED,
method='set',

0
ingestor/io/__init__.py
View File

145
ingestor/io/data_sink.py
View File

@@ -1,145 +0,0 @@
import asyncio
import re
from sqlalchemy import Select, Update, case, column, select, table, update
import ingestor.metrics as metrics
# fullmatch (no ^/$ anchors) so a trailing newline can't slip through — `$` would
# match before a trailing '\n', which is not what we want for a SQL identifier.
_IDENTIFIER_RE = re.compile(r'\w+', re.ASCII)
# Worst-wins precedence of predictions.write_status (SIENTIAPDE-1975), from best to worst;
# NULL, 'pending' and unknown values rank 0. The gateway is the only writer that updates the
# column: the Laborious only sets the value of the rows it inserts ('pending', PI outcome).
STATUS_RANK = {'ok': 1, 'error': 2, 'session_bad': 3}
def write_status_statements(
schema: str, table_name: str, status: str, timestamp, model_id
) -> tuple[Update, Select]:
"""The UPDATE that records `status` unless the row has a worse one, and the SELECT that
tells whether the row exists. Built with SQLAlchemy Core: identifiers and values are
never formatted into the SQL text."""
predictions = table(
table_name,
column('write_status'),
column('timestamp'),
column('model_id'),
schema=schema,
)
row = (predictions.c.timestamp == timestamp) & (predictions.c.model_id == model_id)
current_rank = case(STATUS_RANK, value=predictions.c.write_status, else_=0)
return (
update(predictions)
.where(row, current_rank <= STATUS_RANK.get(status, 0))
.values(write_status=status),
select(1).select_from(predictions).where(row),
)
class DataSink:
"""Gateway egress wrapper: fans a reading out to every configured name/collection."""
def __init__(self, data_manager, postgres) -> None:
self.data_manager = data_manager
self.postgres = postgres
async def emit_metric(self, **kwargs) -> None:
# ponytail: delegate to data_manager, it's already a SientiaMonitoring
await self.data_manager.emit_metric(**kwargs)
async def publish_reading(self, tag: str, value, timestamp, names: list[dict]) -> None:
"""Publish one MongoDB document per name entry and record its tag value metric.
Every name of the slot v2 carries collection, tag_name and model_id (parse_slot
checks it)."""
for name in names:
doc = {
'tag': tag,
'name': name['tag_name'],
'model_id': name['model_id'],
'timestamp': timestamp,
'value': value,
}
await self.data_manager.publish(name['collection'], doc)
await self.emit_metric(
metric_object=metrics.OPC_TAG_VALUE,
method='set',
value=value,
tags={
'pod_id': self.data_manager.pod_id,
'tag_name': name['tag_name'],
'model_id': name['model_id'],
},
)
async def update_write_status(self, cmd: dict, result) -> str:
"""Record a write outcome on the prediction row (C3), keeping the worst one.
The row is found by timestamp + model_id. A prediction is written to several
tags, each answered by its own command, so a better outcome never replaces a
worse one: ``ok`` < ``error`` < ``session_bad``, and ``pending``/NULL are
replaced by any answer, so the order of the answers does not matter. Only the
gateway updates the column (SIENTIAPDE-1975): the Laborious publishes the commands
and sets ``pending`` (merged with its PI Web API outcome) on the row it inserts.
Never raises: a failure here must not reach the NATS callback nor hide the OPC
write itself, which already happened. The outcome is logged and counted instead.
Return:
str: ``updated``, ``kept`` (the row already has a worse outcome),
``row_not_found`` (e.g. the Laborious INSERT failed), ``invalid`` (bad or
missing schema/table/timestamp/model_id) or ``error``.
"""
try:
schema = self._validate_identifier(cmd.get('schema'))
table = self._validate_identifier(cmd.get('table_name'))
if cmd.get('timestamp') is None or cmd.get('model_id') is None:
raise ValueError('missing timestamp or model_id')
except ValueError as e:
return await self._status_outcome('invalid', cmd, f'write_status not recorded: {e}')
statements = write_status_statements(
schema, table, result.status, cmd['timestamp'], cmd['model_id']
)
try:
# The engine is synchronous (no async pg driver here): run it off the event
# loop, which also serves the subscriptions of the read path.
outcome = await asyncio.to_thread(self._update_sync, *statements)
except Exception as e:
return await self._status_outcome('error', cmd, f'write_status UPDATE failed: {e}')
detail = {
'updated': f'write_status={result.status} recorded',
'kept': f'write_status={result.status} not recorded: the row has a worse outcome',
'row_not_found': f'write_status={result.status} not recorded: no prediction row',
}[outcome]
return await self._status_outcome(outcome, cmd, detail)
def _update_sync(self, update_stmt: Update, exists_stmt: Select) -> str:
with self.postgres.engine.begin() as connection:
if connection.execute(update_stmt).rowcount:
return 'updated'
row = connection.execute(exists_stmt).first()
return 'kept' if row is not None else 'row_not_found'
async def _status_outcome(self, outcome: str, cmd: dict, detail: str) -> str:
message = (
f'{detail} (server={cmd.get("server_id")} addr={cmd.get("addr")} '
f'model_id={cmd.get("model_id")} timestamp={cmd.get("timestamp")})'
)
if outcome in ('updated', 'kept'):
self.data_manager.logger.debug(message)
else:
self.data_manager.logger.warning(message)
await self.emit_metric(
metric_object=metrics.OPC_WRITE_STATUS_UPDATE_TOTAL,
tags={'pod_id': self.data_manager.pod_id, 'result': outcome},
)
return outcome
@staticmethod
def _validate_identifier(name: object) -> str:
if not isinstance(name, str) or not _IDENTIFIER_RE.fullmatch(name):
raise ValueError(f'invalid SQL identifier: {name!r}')
return name

51
ingestor/io/slots.py
View File

@@ -1,51 +0,0 @@
"""Parse the slot v2 the Orchestrator writes to Redis (contract C1, SIENTIAPDE-1974).
Each server carries its connection (`url`, `subscription_period_ms`), its OPC security
(`security`: mode + policy) and its client certificate (`certificate`: certificate_id,
application_uri and the MinIO objects of the cert and the private key; None = connect
without security, D5). Each tag carries `names`: one entry per collection / tag name /
model the reading is written to.
"""
SERVER_FIELDS = ('server_id', 'url', 'subscription_period_ms', 'security', 'certificate')
# What a connection/subscription needs (the pool keys clients by server_id itself).
CONNECTION_FIELDS = tuple(f for f in SERVER_FIELDS if f != 'server_id')
NAME_FIELDS = ('collection', 'tag_name', 'model_id')
def _parse_names(address: str, tag: dict) -> list[dict]:
names = tag.get('names')
if not names:
raise ValueError(f'tag {address!r} has no names')
for name in names:
missing = [field for field in NAME_FIELDS if field not in name]
if missing:
raise ValueError(f'tag {address!r} has a name without {missing}')
return names
def _parse_tag(address: str, tag: dict) -> dict:
return {
'frequency': tag['frequency'],
'names': _parse_names(address, tag),
'active_read': tag.get('active_read', {'enabled': False}),
}
def _parse_server(name: str, server: dict) -> dict:
if not isinstance(server.get('security'), dict):
raise ValueError(f'server {name!r} has no security')
parsed = {field: server.get(field) for field in SERVER_FIELDS}
parsed['name'] = server.get('name', name)
parsed['tags'] = {addr: _parse_tag(addr, tag) for addr, tag in server['tags'].items()}
return parsed
def parse_slot(raw: dict) -> dict:
"""Normalize a raw slot v2 into `{server_name: {..., tags: {addr: tag}}}`.
Raises:
KeyError | TypeError | ValueError: When the slot is not a valid slot v2 (e.g. a
slot v1 with `topics`).
"""
return {name: _parse_server(name, server) for name, server in raw.items()}

23
ingestor/managers/data_manager.py
View File

@@ -12,22 +12,6 @@ from sientia_do.temporal.constants import now
import ingestor.metrics as metrics
def redact_uri(uri: str) -> str:
"""
Returns ``uri`` without the user and password, safe to log
(``mongodb://user:pass@host:27017/db`` -> ``mongodb://***@host:27017/db``).
"""
prefix, sep, rest = uri.partition('://')
if not sep:
prefix, rest = '', uri
ends = [i for i in (rest.find('/'), rest.find('?')) if i != -1]
end = min(ends) if ends else len(rest)
authority, tail = rest[:end], rest[end:]
if '@' not in authority:
return uri
return f'{prefix}{sep}***@{authority.rpartition("@")[2]}{tail}'
class DataManager(SientiaMonitoring):
"""
Manages data persistence operations for the OPC Ingestor.
@@ -82,8 +66,7 @@ class DataManager(SientiaMonitoring):
)
logger.info(
'Trying to initializing DataManager with MongoDB servers: '
f'{redact_uri(mongo_connection_string)}'
f'Trying to initializing DataManager with MongoDB servers: {mongo_connection_string}'
)
self.connection_string = mongo_connection_string
@@ -99,9 +82,7 @@ class DataManager(SientiaMonitoring):
self.metadata = metadata
logger.info(
f'DataManager initialized with MongoDB servers: {redact_uri(self.connection_string)}'
)
logger.info(f'DataManager initialized with MongoDB servers: {self.connection_string}')
def shutdown(self):
"""

232
ingestor/managers/ingestor_manager.py
View File

@@ -8,14 +8,9 @@ from sientia_do.observability.metrics_controller import MetricsController
from sientia_do.observability.sientia_monitoring import SientiaMonitoring
import ingestor.metrics as metrics
from ingestor.config import GatewayConfig
from ingestor.connection.certs import build_cert_provider
from ingestor.connection.pool import OpcConnectionPool
from ingestor.io.data_sink import DataSink
from ingestor.io.slots import parse_slot
from ingestor.managers.data_manager import DataManager
from ingestor.managers.opc_manager import OpcManager
from ingestor.managers.resource_manager import ResourceManager
from ingestor.managers.server_reader import ServerReader
class IngestorManager(SientiaMonitoring):
@@ -52,10 +47,7 @@ class IngestorManager(SientiaMonitoring):
Attributes:
data_manager (DataManager): Manages data persistence
opc_managers (dict): ServerReader per OPC server, keyed by server name
pool (OpcConnectionPool): OPC connections shared by reads, writes and gRPC;
owned here because reads start in prepare_ingestor, before GatewayRuntime
sink (DataSink): read egress (fan-out to every slot v2 `names` entry)
opc_managers (dict): Dictionary of OPC managers keyed by server name
resource_manager (ResourceManager): Manages Redis-based resource coordination
number_of_slots (int): Total number of slots configured in the system
poll_interval (int): Main loop polling interval
@@ -76,7 +68,6 @@ class IngestorManager(SientiaMonitoring):
logger: Logger,
notification_handler: NotificationHandler,
metrics_controller: MetricsController,
gateway_config: GatewayConfig | None = None,
):
redis_host: str = redis_data['host']
redis_port: int = int(redis_data['port'])
@@ -99,22 +90,6 @@ class IngestorManager(SientiaMonitoring):
metrics_controller=metrics_controller,
)
self.opc_managers: dict = {}
# Read path (SIENTIAPDE-1974): pooled connections with each server's client
# certificate (named in the slot) from MinIO. Nothing here touches the network at
# construction (MinIO is lazy), so a MinIO/Postgres/NATS outage can't stop the
# reads before they start.
config = gateway_config or GatewayConfig.certs_from_env()
self.cert_provider = build_cert_provider(
config=config,
logger=logger,
notification_handler=notification_handler,
metrics_controller=metrics_controller,
metadata=metadata,
)
self.pool = OpcConnectionPool(
self.cert_provider, logger, notification_handler, metrics_controller
)
self.sink = DataSink(self.data_manager, None)
self.resource_manager = ResourceManager(
host=redis_host,
port=redis_port,
@@ -134,39 +109,49 @@ class IngestorManager(SientiaMonitoring):
self.metadata = metadata
def _build_reader(self, server_config: dict) -> ServerReader:
return ServerReader(
name=server_config['name'],
server_config=server_config,
pool=self.pool,
sink=self.sink,
logger=self.logger,
notification_handler=self.notification_handler,
metrics_controller=self.metrics_controller,
metadata=self.metadata,
)
async def initialize_opc_from_config(self, server_config: dict) -> ServerReader | None:
async def initialize_opc_from_config(self, server_config: dict) -> OpcManager | None:
"""
Initializes the reader of one OPC UA server from its slot configuration.
Initializes an OPC Manager instance using the provided server configuration.
Creates a ServerReader and opens its pooled connection. With a `certificate`, the
connection uses the slot's security mode and policy, with the certificate
downloaded from MinIO; without one, it connects without security (D5).
This method creates and configures an OPC Manager for a specific OPC UA server,
establishing the connection and preparing it for tag subscriptions.
Args:
server_config (dict): The server configuration of the slot v2, without the
'tags' key: 'name', 'server_id', 'url', 'subscription_period_ms',
'security' and 'certificate'.
server_config (dict): A dictionary containing the OPC server configuration.
Expected keys include:
- 'name' (str): The name of the OPC server.
- 'url' (str): The URL of the OPC server.
- 'server_uri' (str): The URI of the OPC server.
- 'cert_path' (str, optional): Path to the client certificate file.
- 'private_key_path' (str, optional): Path to the private key file.
- 'server_cert_path' (str, optional): Path to the server certificate file.
Returns:
ServerReader | None: The connected reader, or None if the
connection fails (logged and notified, so the other servers keep working).
OpcManager | None: An initialized OpcManager instance if successful,
otherwise None if an error occurs during initialization.
Raises:
Exception: If OPC manager initialization fails, the error is logged and
a notification is sent, but the method returns None to allow
the system to continue operating with other servers.
"""
try:
self.logger.info(f'Initializing OPC reader at {server_config["url"]}')
manager = self._build_reader(server_config)
self.logger.info(f'Initializing OpcManager at {server_config["url"]}')
manager = OpcManager(
name=server_config['name'],
url=server_config['url'],
subscription_period_ms=server_config['subscription_period_ms'],
data_manager=self.data_manager,
logger=self.logger,
server_uri=server_config['server_uri'],
notification_handler=self.notification_handler,
metadata=self.metadata,
cert_path=server_config.get('cert_path'),
private_key_path=server_config.get('private_key_path'),
server_cert_path=server_config.get('server_cert_path'),
metrics_controller=self.metrics_controller,
)
manager.config = server_config
await manager.connect()
@@ -191,29 +176,23 @@ class IngestorManager(SientiaMonitoring):
Gracefully shuts down the IngestorManager and all its components.
This method ensures proper cleanup of:
- All OPC readers, their subscriptions and the pooled connections
- All OPC manager instances and their connections
- Data manager connections and resources
- Active subscriptions and server connections
The shutdown process is performed asynchronously to allow proper cleanup
of all managed resources before termination.
"""
for _server_name, server in self.opc_managers.items():
await server.shutdown()
await self.pool.close_all()
self.data_manager.shutdown()
async def remove_server(self, server: str, evict: bool = True):
async def remove_server(self, server: str):
"""
Removes an OPC server from the ingestor.
Args:
server (str): The server name.
evict (bool): Also drop the pooled connection (lost server, failed
subscription). False when the server merely left this pod's slots:
the shared connection stays for writes/gRPC until the idle sweep.
"""
if server in self.opc_managers:
await self.opc_managers[server].shutdown(evict=evict)
await self.opc_managers[server].shutdown()
del self.opc_managers[server]
for slot, _config in self.managed_tags.items():
self.managed_tags[slot].pop(server, None)
@@ -279,10 +258,9 @@ class IngestorManager(SientiaMonitoring):
for server in servers:
if server not in registered_servers:
self.logger.warning(
f'Server {server} not found in managed tags. Unsubscribing from server.'
f'Server {server} not found in managed tags. Desconnecting from server.'
)
# Slot moved away: keep the shared connection for writes/gRPC.
await self.remove_server(server, evict=False)
await self.remove_server(server)
await self.emit_metric(
metric_object=metrics.OPC_MANAGERS_ACTIVE,
@@ -441,18 +419,14 @@ class IngestorManager(SientiaMonitoring):
Notes:
- If a slot is leased but its details cannot be retrieved
(i.e., `get_tag_slot` returns None) or are malformed, that slot is skipped.
- Slot details are normalized to the slot v2 structure (parse_slot).
(i.e., `get_tag_slot` returns None), that slot is skipped.
"""
acquired = {}
for i in range(1, self.number_of_slots + 1):
if await self.resource_manager.lease_tag(str(i)):
self.logger.info(f'Leased slot {i}')
raw = await self.resource_manager.get_tag_slot(str(i))
if raw is None:
continue
slots = self.parse_slot_config(str(i), raw)
slots = await self.resource_manager.get_tag_slot(str(i))
if slots is None:
continue
acquired[str(i)] = slots
@@ -493,81 +467,26 @@ class IngestorManager(SientiaMonitoring):
async def unsubscribe_slot(self, slot: str):
"""
Unsubscribes a specific slot from every OPC reader that holds a subscription for it.
Unsubscribes a specific slot from all associated OPC servers.
The readers' own subscriptions are the source of truth, not managed_tags: the
main loop removes a slot from managed_tags before update_ingestor_manager asks
to unsubscribe it, both when manage_leases drops an extra lease (which already
unsubscribed it) and when update_slot_config finds the slot gone from Redis
(which did not). Looking the slot up in managed_tags raised KeyError in both
cases, which stopped the main loop and restarted the container; the second case
also left the subscription open on the OPC server. Unsubscribing a slot that no
reader holds is a no-op.
A reader that fails to unsubscribe is handled by _unsubscribe_reader: the other
readers are still unsubscribed and the main loop goes on.
This method removes all subscriptions for a given slot across all
OPC servers that were managing it. It ensures clean cleanup of
resources when slots are released or reconfigured.
Args:
slot (str): The name of the slot to unsubscribe.
Raises:
KeyError: If the specified slot does not exist in the managed tags.
Side Effects:
- Removes the slot's subscription from every OPC reader that has one
- Removes subscriptions from all OPC servers for the specified slot
- Cleans up subscription resources on the OPC servers
"""
# list() snapshots the readers: the loop awaits, so a reader added or removed
# meanwhile must not break the iteration. SonarQube flags it as an unnecessary
# list() (S7504); the issue is accepted in SonarQube for this reason.
for server, reader in list(self.opc_managers.items()):
if slot in reader.subscriptions:
await self._unsubscribe_reader(server, reader, slot)
async def _unsubscribe_reader(self, server: str, reader: ServerReader, slot: str) -> None:
"""
Unsubscribe one reader from a slot without letting a failure stop the main loop.
A failure (e.g. delete() raising when the server session is gone) increments OPC_SUBSCRIPTION_ERRORS, sends
OPC_UNSUBSCRIBE_ERROR_<slot>:<server> and is logged. The slot is dropped from the
reader's books, as the v2 engine already does, so it can be subscribed again.
Known limitation: if delete() failed while the session is still alive (e.g. a
timeout), the server-side subscription may keep delivering data until that
session ends (reconnection or pod restart).
Args:
server (str): The reader's key in opc_managers
reader (ServerReader): The OPC reader
slot (str): The slot to unsubscribe
"""
try:
await reader.unsubscribe(slot)
except Exception as e:
await self.emit_metric(
metric_object=metrics.OPC_SUBSCRIPTION_ERRORS,
tags={
'pod_id': self.pod_id,
'server': server,
'slot': slot,
},
)
trace = traceback.format_exc()
await self.send_notification_async(
metadata=self.metadata,
notification_id=f'OPC_UNSUBSCRIBE_ERROR_{slot}:{server}',
message=f'Failed to unsubscribe slot {slot} from {server}: {e}',
block='opc_manager',
level=NotificationLevel.ERROR,
attachment_content=trace,
)
self.logger.error(trace)
reader.subscriptions.pop(slot, None)
async def _unsubscribe_removed_slots(self, slots: list[str]) -> None:
"""Unsubscribe and forget slots whose configuration is gone from Redis."""
for slot in slots:
self.logger.warning(f'Slot {slot} is no longer configured; unsubscribing it')
await self.unsubscribe_slot(slot)
self.managed_tags.pop(slot, None)
for server in self.managed_tags[slot].keys():
if server in self.opc_managers:
await self.opc_managers[server].unsubscribe(slot)
async def update_slot_config(self):
"""
@@ -577,7 +496,7 @@ class IngestorManager(SientiaMonitoring):
This method performs the following steps:
1. Renews the lease for each managed slot using the resource manager.
2. Fetches the latest configuration for each slot.
3. Logs, unsubscribes and removes slots whose configurations are no longer available.
3. Logs and removes slots whose configurations are no longer available.
4. Updates the configuration of slots if changes are detected.
5. Unsubscribes and re-subscribes to slots with updated configurations.
6. Removes slots from the managed tags if they are no longer valid.
@@ -599,44 +518,15 @@ class IngestorManager(SientiaMonitoring):
removed_slots: list[str] = []
for slot, _slot_config in self.managed_tags.items():
await self.resource_manager.renew_tag_lease(slot)
raw = await self.resource_manager.get_tag_slot(slot)
if raw is None:
update = await self.resource_manager.get_tag_slot(slot)
if update is None:
removed_slots.append(slot)
continue
update = self.parse_slot_config(slot, raw)
if update is None:
# Malformed config: keep reading with the last good one.
continue
self.managed_tags[slot] = update
# Unsubscribe right away: leaving it to update_ingestor_manager's snapshot diff
# would leak the subscription if anything in between failed
await self._unsubscribe_removed_slots(removed_slots)
def parse_slot_config(self, slot: str, raw: dict) -> dict | None:
"""
Normalizes the raw slot read from Redis to the slot v2 structure (contract C1).
Only the slot v2 is accepted (SIENTIAPDE-1974): the gateway and the Orchestrator
are deployed together, and a slot v1 (`topics`, no `security`) is rejected.
Returns:
dict | None: The parsed slot, or None when the config is malformed
(logged and notified).
"""
try:
return parse_slot(raw)
except (AttributeError, KeyError, TypeError, ValueError) as e:
self.send_notification(
metadata=self.metadata,
notification_id=f'OPC_SLOT_CONFIG_ERROR_{slot}',
message=f'Slot {slot} has an invalid configuration: {e!r}',
block='ingestor_manager',
level=NotificationLevel.ERROR,
)
self.logger.error(f'Slot {slot} has an invalid configuration: {e!r}')
return None
for slot in removed_slots:
self.managed_tags.pop(slot, None)
async def drop_slot_leases(self, ids: list[str]) -> None:
"""
@@ -744,7 +634,7 @@ class IngestorManager(SientiaMonitoring):
self.logger.error(trace)
self.logger.warning(f'Removing subscription from server {server} for slot {slot}')
await self._unsubscribe_reader(server, self.opc_managers[server], slot)
await self.opc_managers[server].unsubscribe(slot)
return 2
return 0

588
ingestor/managers/opc_manager.py Normal file
View File

@@ -0,0 +1,588 @@
import asyncio
import json
import traceback
from pathlib import Path
from asyncua import Client
from asyncua.crypto.security_policies import SecurityPolicyBasic256
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
from sientia_do.notifications.models import NotificationLevel
from sientia_do.observability.logger import Logger
from sientia_do.observability.metrics_controller import MetricsController
from sientia_do.observability.sientia_monitoring import SientiaMonitoring
from sientia_do.temporal.constants import DATETIME_FORMAT_WITH_TZ, OPC_TIMEZONE
import ingestor.metrics as metrics
from ingestor.managers.data_manager import DataManager
class OpcManager(SientiaMonitoring):
"""
Manages OPC UA server connections and tag subscriptions.
The OpcManager is responsible for:
- Establishing and maintaining secure connections to OPC UA servers
- Managing tag subscriptions and data collection
- Handling server reconnection and error recovery
- Processing OPC data and forwarding it to the data manager
- Monitoring connection health and performance metrics
The manager supports both secure and unsecured connections, with optional
certificate-based authentication for enhanced security.
Args:
name (str): Unique identifier for the OPC server
url (str): OPC UA server endpoint URL
data_manager (DataManager): Manager for data persistence and export
logger (Logger): Logger instance for application logging
server_uri (str): OPC UA server application URI
notification_handler (NotificationHandler): Handler for sending notifications
metadata (dict): Application metadata for notifications and tracking
cert_path (str, optional): Path to client certificate file for secure connections
private_key_path (str, optional): Path to client private key file
server_cert_path (str, optional): Path to server certificate file for validation
Attributes:
url (str): OPC UA server endpoint URL
name (str): Unique identifier for the OPC server
server_uri (str): OPC UA server application URI
data_queue (dict): Queue for buffering OPC data before processing
non_receive_count (int): Counter for cycles without data reception
client (Client): OPC UA client instance
cert_path (str): Path to client certificate file
private_key_path (str): Path to client private key file
server_cert_path (str): Path to server certificate file
nodes (dict): Dictionary of OPC node references
subscriptions (dict): Active OPC subscriptions
data_manager (DataManager): Manager for data persistence and export
metadata (dict): Application metadata
"""
def __init__(
self,
name: str,
url: str,
subscription_period_ms: int,
data_manager: DataManager,
logger: Logger,
server_uri: str,
notification_handler: NotificationHandler,
metrics_controller: MetricsController,
metadata: dict,
cert_path: str | None = None,
private_key_path: str | None = None,
server_cert_path: str | None = None,
):
self.url = url
self.name = name
self.server_uri = server_uri
self.data_queue: dict = {}
self.non_receive_count = 0
self.client: Client | None = None
self.subscription_period_ms = subscription_period_ms
self.cert_path = cert_path
self.private_key_path = private_key_path
self.server_cert_path = server_cert_path
self.nodes: dict = {}
self.subscriptions: dict = {}
self.data_manager = data_manager
self.metadata = metadata
SientiaMonitoring.__init__(
self,
logger=logger,
notification_handler=notification_handler,
metrics_controller=metrics_controller,
)
metrics.OPC_CONNECTION_STATUS.labels(
pod_id=self.pod_id, server_name=self.name, server_url=self.url
).set(0)
metrics.OPC_TAGS_SUBSCRIBED.labels(pod_id=self.pod_id, server_name=self.name).set(0)
def __str__(self):
"""
String representation of the OPC Manager.
Returns:
str: Human-readable representation showing server details and current state.
"""
return (
f'OpcManager(name={self.name}, url={self.url}, server_uri={self.server_uri})\n'
f'nodes={self.nodes}, subscriptions={self.subscriptions}'
)
async def shutdown(self):
"""
Comprehensive cleanup method for graceful shutdown.
This method ensures proper cleanup of all OPC UA resources:
- Closes active subscriptions
- Disconnects from the OPC server
- Releases allocated resources
Should be called before the application terminates to prevent resource leaks
and ensure clean disconnection from OPC servers.
"""
try:
await self.disconnect()
except Exception as e:
self.logger.error(f'Error during cleanup: {e}')
async def set_security(self):
"""
Configures the security settings for the OPC UA client.
This method sets up the security policy, certificates, and timeouts
required for establishing a secure connection with the OPC UA server.
It implements Basic256 security policy with certificate-based authentication.
Raises:
ValueError: If either the certificate path or private key path is not provided.
Security Settings:
- Security Policy: Basic256
- Secure Channel Timeout: 10,000,000 ms
- Session Timeout: 10,000,000 ms
The method configures:
- Client application URI
- Certificate-based authentication
- Server certificate validation (if provided)
- Connection timeouts for stability
"""
if not all([self.cert_path, self.private_key_path]):
raise ValueError(
'Certificate and private key paths must be provided for secure connection.'
)
cert = str(Path(self.cert_path)) if self.cert_path else None
private_key = str(Path(self.private_key_path)) if self.private_key_path else None
server_cert = str(Path(self.server_cert_path)) if self.server_cert_path else None
assert cert is not None and private_key is not None
if self.client:
self.client.application_uri = self.server_uri
self.logger.info('Setting security...')
await self.client.set_security(
SecurityPolicyBasic256,
certificate=cert,
private_key=private_key,
server_certificate=server_cert,
)
self.client.secure_channel_timeout = 10000000
self.client.session_timeout = 10000000
async def connect(self):
"""
Establishes a connection to the OPC server.
This method initializes the OPC client using the provided URL and
sets up security if a certificate path is specified. It then
attempts to connect to the server and logs the connection status.
The connection process includes:
1. Client initialization with server URL
2. Security configuration (if certificates are provided)
3. Connection establishment
4. Metrics recording for monitoring
Raises:
Exception: If the connection to the OPC server fails.
Metrics:
- OPC_CONNECTIONS_TOTAL: Incremented on connection attempt
- OPC_CONNECTION_STATUS: Set to 1 on successful connection
"""
await self.emit_metric(
metric_object=metrics.OPC_CONNECTIONS_TOTAL,
tags={
'pod_id': self.pod_id,
'server_name': self.name,
},
)
try:
self.client = Client(self.url, timeout=10, watchdog_intervall=3600000)
assert self.client is not None # Informa ao mypy que client não é None
self.client.name = self.pod_id
self.client.description = self.pod_id
pod_uri = self.pod_id.replace('-', ':')
self.client.application_uri = pod_uri
self.client.product_uri = pod_uri
if self.cert_path:
await self.set_security()
self.logger.info(f'Starting connection to {self.name}...')
await self.client.connect()
await self.emit_metric(
metric_object=metrics.OPC_CONNECTION_STATUS,
method='set',
value=1,
tags={
'pod_id': self.pod_id,
'server_name': self.name,
'server_url': self.url,
},
)
self.logger.info(f'Connection to {self.name} successful.')
except Exception:
await self.disconnect()
raise
async def create_subscription(self, name: str):
"""
Creates a subscription with the specified monitoring period.
This method establishes a subscription to monitor data changes or events
from the OPC UA server. If the client is not connected, an exception is raised.
Args:
name (str): The name identifier for the subscription
period (int, optional): The monitoring period in milliseconds. Defaults to 500 ms.
Raises:
ValueError: If the client is not connected.
Side Effects:
- Sets the `self.period` attribute to the specified or default period.
- Creates a subscription and assigns it to `self.subscriptions[name]`.
- Logs the creation of the subscription.
- Increments subscription creation metrics.
"""
if not self.client:
raise ValueError('Client not connected. Call connect first.')
try:
self.subscriptions[name] = await self.client.create_subscription(
self.subscription_period_ms, self
)
self.logger.info(f'Subscription {name} created on {self.name}.')
await self.emit_metric(
metric_object=metrics.OPC_SUBSCRIPTIONS_CREATED,
method='inc',
value=1,
tags={
'pod_id': self.pod_id,
'server_name': self.name,
'slot_name': name,
},
)
except Exception as e:
self.logger.error(f'Failed to create subscription {name} on {self.name}: {e}')
raise
async def subscribe(self, subscription: str, nodes: dict, collect_period: int):
"""
Subscribes to a set of OPC UA nodes for data change notifications.
This method adds the specified nodes to the subscription and configures
their data collection rules based on the provided collection period and
node-specific frequency.
Args:
subscription (str): The name of the subscription to use
nodes (dict): A dictionary where keys are node identifiers and values are
configurations for each node. Each configuration must include a 'frequency'
key indicating the frequency of data collection in Hz.
collect_period (int): The data collection period in seconds.
Raises:
ValueError: If the subscription has not been created by calling
`create_subscription` prior to this method.
Side Effects:
- Updates internal node tracking and cycle rules
- Establishes data change monitoring for specified nodes
- Updates metrics for subscribed tags count
"""
if not self.subscriptions.get(subscription):
raise ValueError('Subscription not created. Call create_subscription first.')
self.logger.info(f'Subscribing to {subscription} on {self.name}...')
self.logger.info(f'Subscribing to nodes: {nodes}')
assert self.client is not None # Informa ao mypy que client não é None
addr_nodes = [self.client.get_node(n) for n in nodes]
self.logger.debug(f'Addr nodes: {addr_nodes}')
self.nodes.update(nodes)
self.logger.debug(f'Nodes: {self.nodes}')
self.collect_period = collect_period
for node, config in self.nodes.items():
self.nodes[node]['cycle_rule'] = {
'cycle_increment': collect_period * 1000 / float(config['frequency']),
'cycle_count': 0,
}
await self.subscriptions[subscription].subscribe_data_change(addr_nodes)
await self.emit_metric(
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
method='set',
value=len(self.nodes),
tags={
'pod_id': self.pod_id,
'server_name': self.name,
},
)
async def unsubscribe(self, subscription: str):
"""
Unsubscribes from a given subscription.
This method removes the specified subscription and cleans up associated
resources. It handles cases where the subscription doesn't exist gracefully.
Args:
subscription (str): The name of the subscription to unsubscribe from.
Logs:
- A warning if the specified subscription does not exist.
- An info message upon successful unsubscription.
Behavior:
- If the subscription exists, it is deleted and removed from the
subscriptions dictionary.
- If the subscription does not exist, no action is taken.
"""
if not self.subscriptions.get(subscription):
self.logger.warning(f"Subscription '{subscription}' not found. Cannot unsubscribe.")
return
await self.subscriptions[subscription].delete()
del self.subscriptions[subscription]
self.logger.info(f'Unsubscribed from {subscription}.')
async def disconnection_fallback(self) -> list:
"""
Tries 5 times to disconnect from the OPC UA server, with a delay of 100ms x try.
"""
assert self.client is not None
error_stack = []
for i in range(5):
try:
self.logger.info(f'Disconnecting from OPC UA server, attempt {i + 1} of 5')
await self.client.disconnect()
return []
except Exception as e:
self.logger.error(
f'Failed to disconnect from OPC UA serve in attempt {i + 1} of 5: {e}'
)
error_stack.append(
{
'attempt': i + 1,
'error': str(e),
'traceback': traceback.format_exc(),
}
)
await asyncio.sleep(0.1 * i)
return error_stack
async def disconnect(self):
"""
Disconnects from the OPC UA server.
This method handles the disconnection process by deleting all subscriptions
and disconnecting the client from the OPC UA server. It logs the disconnection
process and handles any exceptions that may occur during cleanup.
Raises:
Exception: If an error occurs while deleting the subscription or disconnecting
from the OPC UA server, it logs the error details.
Side Effects:
- Deletes all active subscriptions
- Disconnects the OPC client
- Updates connection status metrics
- Clears internal client reference
"""
self.logger.warning('Disconnecting from OPC server')
if self.client is None:
self.logger.warning('Client already disconnected.')
return
try:
for sub in self.subscriptions:
await self.subscriptions[sub].delete()
self.logger.warning('Deleted all subscriptions.')
except Exception as sub_error:
self.logger.error(f'Failed to clean up subscription: {sub_error}')
errors = await self.disconnection_fallback()
if errors:
await self.send_notification_async(
metadata=self.metadata,
notification_id=f'OPC_DISCONNECTION_ERROR_{self.name}',
message=f'Failed to disconnect from OPC UA server {self.name} after 5 attempts',
block='opc_manager',
level=NotificationLevel.ERROR,
attachment_content=json.dumps(errors, indent=4),
)
else:
self.logger.warning('Disconnected from OPC UA server.')
del self.client
self.client = None
await self.emit_metric(
metric_object=metrics.OPC_CONNECTION_STATUS,
method='set',
value=0,
tags={
'pod_id': self.pod_id,
'server_name': self.name,
'server_url': self.url,
},
)
await self.emit_metric(
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
method='set',
value=0,
tags={
'pod_id': self.pod_id,
'server_name': self.name,
},
)
async def datachange_notification(self, node, _val, data):
"""
Handles data change notifications for monitored OPC UA nodes.
This method is triggered when a monitored node's value changes. It processes
the notification, updates internal state, and publishes the data to the
appropriate topics.
Args:
node (NodeId): The OPC UA node that triggered the data change notification.
_val (Any): The new value of the node (unused in this implementation).
data (DataChangeNotification): The data change notification object containing
details about the change.
Behavior:
- Extracts the value and source timestamp from the monitored item.
- Resets the cycle count for the node's cycle rule.
- Resets the non-receive count.
- Constructs a data dictionary containing the tag, tag name, timestamp, and value.
- Publishes the data to all topics associated with the node.
"""
# get data value
monitored_item = data.monitored_item
value = monitored_item.Value.Value.Value
# source_timestamp
source_timestamp = monitored_item.Value.SourceTimestamp.replace(tzinfo=OPC_TIMEZONE)
tag = str(node)
self.logger.debug(
f'Data change notification received for tag:'
f'{tag} after {self.nodes[tag]["cycle_rule"]["cycle_count"]} cycles'
)
data = {
'tag': tag,
'name': self.nodes[str(node)]['tag_name'],
'timestamp': source_timestamp.strftime(DATETIME_FORMAT_WITH_TZ),
'value': value,
}
for topic in self.nodes[tag]['topics']:
await self.data_manager.publish(topic, data)
self.nodes[tag]['cycle_rule']['cycle_count'] = 0
self.non_receive_count = 0
await self.emit_metric(
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
method='set',
value=0,
tags={
'pod_id': self.pod_id,
'server_name': self.name,
},
)
async def check_cycles(self):
"""
Checks the cycle counts for all monitored nodes and sends
notifications if thresholds are exceeded.
This method iterates through all monitored nodes and updates their cycle counts based on
configured increments. If a node's cycle count exceeds a threshold (5 cycles), it triggers
a warning notification.
Side Effects:
- Updates cycle counts for all monitored nodes
- Sends warning notifications for nodes exceeding cycle thresholds
"""
for node, config in self.nodes.items():
self.nodes[node]['cycle_rule']['cycle_count'] += config['cycle_rule']['cycle_increment']
if self.nodes[node]['cycle_rule']['cycle_count'] >= 5:
name = config['tag_name']
cycles = self.nodes[node]['cycle_rule']['cycle_count']
await self.send_notification_async(
metadata=self.metadata,
notification_id=f'TAG_{node}:{name}_LISTENNING_STOPPED',
message=f'{cycles} cycles without receive from {node}:{name}',
block='opc_manager',
level=NotificationLevel.WARNING,
)
async def check_opc_listenning(self) -> bool:
"""
Checks the OPC connection and triggers notifications if the connection is lost.
This method monitors the data reception health by tracking cycles without
data. It sends notifications at different thresholds and can trigger
reconnection attempts.
Returns:
bool: True if the connection is lost and reconnection should be attempted,
False otherwise.
Side Effects:
- Increments non-receive count
- Updates metrics for cycles without data
- Sends warning notifications at 5 cycles
- Sends error notifications and triggers reconnection at 15 cycles
"""
self.non_receive_count += 1
await self.emit_metric(
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
method='set',
value=self.non_receive_count,
tags={
'pod_id': self.pod_id,
'server_name': self.name,
},
)
if self.non_receive_count >= 5:
await self.send_notification_async(
metadata=self.metadata,
notification_id=f'OPC_LISTENNING_STOPPED__{self.name}',
message=f'{self.non_receive_count} cycles without '
f'receive from OPC {self.name}. Tags: {json.dumps(self.nodes)}',
block='opc_manager',
level=NotificationLevel.ERROR,
)
if self.non_receive_count >= 15:
await self.emit_metric(
metric_object=metrics.OPC_RECONNECTIONS_TOTAL,
tags={
'pod_id': self.pod_id,
'server_name': self.name,
},
)
await self.send_notification_async(
metadata=self.metadata,
notification_id=f'OPC_CONNECTION_RETRY__{self.name}',
message=f'Retrying to connect to server {self.name}',
block='opc_manager',
level=NotificationLevel.ERROR,
)
return True
return False

105
ingestor/managers/server_reader.py
View File

@@ -1,105 +0,0 @@
"""Per-server read path driven by the IngestorManager (SIENTIAPDE-1974).
Exposes the interface the IngestorManager drives (connect / create_subscription /
subscribe / unsubscribe / check_cycles / check_opc_listenning / shutdown):
- the connection comes from the shared OpcConnectionPool, with the slot's security
mode and policy and the client certificate the slot names, which CertProvider
downloads from MinIO (no certificate: no security);
- readings go through SubscriptionEngine, which fans each value out to every
`names` entry of the slot v2 (one document per collection / tag name / model).
"""
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
from sientia_do.observability.logger import Logger
from sientia_do.observability.metrics_controller import MetricsController
import ingestor.metrics as metrics
from ingestor.connection.pool import OpcConnectionPool
from ingestor.engines.subscription import SubscriptionEngine
from ingestor.io.data_sink import DataSink
from ingestor.io.slots import CONNECTION_FIELDS
class ServerReader:
"""Reads one OPC server: one SubscriptionEngine (its `non_receive_count` is
engine-wide, hence one per server) with one subscription per leased slot."""
def __init__(
self,
name: str,
server_config: dict,
pool: OpcConnectionPool,
sink: DataSink,
logger: Logger,
notification_handler: NotificationHandler,
metrics_controller: MetricsController,
metadata: dict | None = None,
):
self.name = name
self.server_id = str(server_config['server_id'])
self.url = server_config['url']
self.server_cfg = {k: server_config[k] for k in CONNECTION_FIELDS if k in server_config}
self.pool = pool
self.logger = logger
self.config: dict | None = None
self.engine = SubscriptionEngine(
pool, sink, logger, notification_handler, metrics_controller, metadata
)
def __str__(self):
return f'ServerReader(name={self.name}, server_id={self.server_id}, url={self.url})'
@property
def subscriptions(self) -> dict:
return self.engine.subscriptions
async def connect(self):
"""Open (or reuse) the pooled connection; raises if the server is unreachable
or the certificate can't be fetched."""
await self.pool.get(self.server_id, self.server_cfg)
async def create_subscription(self, slot: str):
await self.engine.create_subscription(self.server_id, self.server_cfg, slot)
async def subscribe(self, slot: str, nodes: dict, collect_period: int):
await self.engine.subscribe(slot, nodes, collect_period)
async def unsubscribe(self, slot: str):
await self.engine.unsubscribe(slot)
async def check_cycles(self):
await self.engine.check_cycles()
async def check_opc_listenning(self) -> bool:
return await self.engine.check_opc_listenning(self.server_id)
async def shutdown(self, evict: bool = True):
"""Drop every subscription of this server.
With `evict` (lost server or changed server config) the pooled connection is
also dropped, so the next initialization reconnects. Without it (the slot just
moved to another pod) the connection stays for writes/gRPC: the connection is
shared, and the idle sweep closes it if nobody uses it anymore."""
# list() snapshots the keys: unsubscribe() pops from engine.subscriptions, so
# iterating the dict directly would raise "changed size during iteration".
for slot in list(self.engine.subscriptions): # NOSONAR
await self.engine.unsubscribe(slot)
if evict:
await self.pool.evict(self.server_id)
await self.engine.emit_metric(
metric_object=metrics.OPC_CONNECTION_STATUS,
method='set',
value=0,
tags={
'pod_id': self.engine.pod_id,
'server_name': self.server_id,
'server_url': self.url,
},
)
await self.engine.emit_metric(
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
method='set',
value=0,
tags={'pod_id': self.engine.pod_id, 'server_name': self.server_id},
)

57
ingestor/metrics.py
View File

@@ -93,7 +93,7 @@ OPC_MANAGERS_ACTIVE = Gauge(
)
OPC_SUBSCRIPTION_ERRORS = Counter(
'ingestor_opc_subscription_errors_total',
'Errors when trying to subscribe to or unsubscribe from OPC tags',
'Errors when trying to subscribe to OPC tags',
['pod_id', 'server', 'slot'],
)
@@ -139,58 +139,3 @@ NOTIFICATIONS_SENT = Counter(
'Total number of notifications sent',
NOTIFICATION_LABELS,
)
# --- Gateway: Write metrics ---
OPC_WRITE_TOTAL = Counter(
'opc_write_total',
'OPC active write attempts',
['pod_id', 'server_name', 'model_id', 'result'], # result: success|error
)
OPC_WRITE_LATENCY = Histogram(
'opc_write_latency_seconds',
'OPC active write latency',
['pod_id', 'server_name'],
)
OPC_WRITE_STATUS_BY_MODEL = Gauge(
'opc_write_status_by_model',
'Last write status per model (1=ok,0=error)',
['pod_id', 'model_id', 'status'],
)
OPC_WRITE_STATUS_UPDATE_TOTAL = Counter(
'opc_write_status_update_total',
'write_status UPDATEs on the prediction rows',
# result: updated|kept|row_not_found|invalid|error
['pod_id', 'result'],
)
# --- Gateway: Active read metrics ---
OPC_ACTIVE_READ_TOTAL = Counter(
'opc_active_read_total',
'OPC active read attempts',
['pod_id', 'server_name', 'result'],
)
OPC_ACTIVE_READ_LATENCY = Histogram(
'opc_active_read_latency_seconds',
'OPC active read latency',
['pod_id', 'server_name'],
)
OPC_READ_LAG = Gauge(
'opc_read_lag_seconds',
'Lag between OPC source timestamp and machine time',
['pod_id', 'server_name'],
)
OPC_TAG_VALUE = Gauge(
'opc_tag_value',
'Last value read per tag/model',
['pod_id', 'tag_name', 'model_id'],
)
# --- Gateway: connection/error metrics ---
OPC_DISCONNECTIONS_TOTAL = Counter(
'opc_disconnections_total',
'OPC disconnections',
['pod_id', 'server_name'],
)
OPC_ERROR_BY_TYPE = Counter(
'opc_error_by_type_total',
'OPC errors by type',
['pod_id', 'server_name', 'error_type'],
)

0
ingestor/proto/__init__.py
View File

18
ingestor/proto/opc_gateway.proto
View File

@@ -1,18 +0,0 @@
syntax = "proto3";
package opcgateway;
service OpcGateway {
rpc ReadTag(ReadTagRequest) returns (ReadTagResponse);
}
message ReadTagRequest {
string server_id = 1;
string addr = 2;
string data_type = 3;
}
message ReadTagResponse {
string value = 1;
string opc_status_code = 2;
string source_timestamp = 3;
string server_timestamp = 4;
bool persisted = 5;
}

40
ingestor/proto/opc_gateway_pb2.py
View File

@@ -1,40 +0,0 @@
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# NO CHECKED-IN PROTOBUF GENCODE
# source: opc_gateway.proto
# Protobuf Python Version: 7.35.0
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import runtime_version as _runtime_version
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_runtime_version.ValidateProtobufRuntimeVersion(
_runtime_version.Domain.PUBLIC,
7,
35,
0,
'',
'opc_gateway.proto'
)
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x11opc_gateway.proto\x12\nopcgateway\"D\n\x0eReadTagRequest\x12\x11\n\tserver_id\x18\x01 \x01(\t\x12\x0c\n\x04\x61\x64\x64r\x18\x02 \x01(\t\x12\x11\n\tdata_type\x18\x03 \x01(\t\"\x80\x01\n\x0fReadTagResponse\x12\r\n\x05value\x18\x01 \x01(\t\x12\x17\n\x0fopc_status_code\x18\x02 \x01(\t\x12\x18\n\x10source_timestamp\x18\x03 \x01(\t\x12\x18\n\x10server_timestamp\x18\x04 \x01(\t\x12\x11\n\tpersisted\x18\x05 \x01(\x08\x32P\n\nOpcGateway\x12\x42\n\x07ReadTag\x12\x1a.opcgateway.ReadTagRequest\x1a\x1b.opcgateway.ReadTagResponseb\x06proto3')
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'opc_gateway_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
DESCRIPTOR._loaded_options = None
_globals['_READTAGREQUEST']._serialized_start=33
_globals['_READTAGREQUEST']._serialized_end=101
_globals['_READTAGRESPONSE']._serialized_start=104
_globals['_READTAGRESPONSE']._serialized_end=232
_globals['_OPCGATEWAY']._serialized_start=234
_globals['_OPCGATEWAY']._serialized_end=314
# @@protoc_insertion_point(module_scope)

97
ingestor/proto/opc_gateway_pb2_grpc.py
View File

@@ -1,97 +0,0 @@
# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
"""Client and server classes corresponding to protobuf-defined services."""
import grpc
import warnings
from . import opc_gateway_pb2 as opc__gateway__pb2
GRPC_GENERATED_VERSION = '1.82.1'
GRPC_VERSION = grpc.__version__
_version_not_supported = False
try:
from grpc._utilities import first_version_is_lower
_version_not_supported = first_version_is_lower(GRPC_VERSION, GRPC_GENERATED_VERSION)
except ImportError:
_version_not_supported = True
if _version_not_supported:
raise RuntimeError(
f'The grpc package installed is at version {GRPC_VERSION},'
+ ' but the generated code in opc_gateway_pb2_grpc.py depends on'
+ f' grpcio>={GRPC_GENERATED_VERSION}.'
+ f' Please upgrade your grpc module to grpcio>={GRPC_GENERATED_VERSION}'
+ f' or downgrade your generated code using grpcio-tools<={GRPC_VERSION}.'
)
class OpcGatewayStub:
"""Missing associated documentation comment in .proto file."""
def __init__(self, channel):
"""Constructor.
Args:
channel: A grpc.Channel.
"""
self.ReadTag = channel.unary_unary(
'/opcgateway.OpcGateway/ReadTag',
request_serializer=opc__gateway__pb2.ReadTagRequest.SerializeToString,
response_deserializer=opc__gateway__pb2.ReadTagResponse.FromString,
_registered_method=True)
class OpcGatewayServicer:
"""Missing associated documentation comment in .proto file."""
def ReadTag(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def add_OpcGatewayServicer_to_server(servicer, server):
rpc_method_handlers = {
'ReadTag': grpc.unary_unary_rpc_method_handler(
servicer.ReadTag,
request_deserializer=opc__gateway__pb2.ReadTagRequest.FromString,
response_serializer=opc__gateway__pb2.ReadTagResponse.SerializeToString,
),
}
generic_handler = grpc.method_handlers_generic_handler(
'opcgateway.OpcGateway', rpc_method_handlers)
server.add_generic_rpc_handlers((generic_handler,))
server.add_registered_method_handlers('opcgateway.OpcGateway', rpc_method_handlers)
# This class is part of an EXPERIMENTAL API.
class OpcGateway:
"""Missing associated documentation comment in .proto file."""
@staticmethod
def ReadTag(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(
request,
target,
'/opcgateway.OpcGateway/ReadTag',
opc__gateway__pb2.ReadTagRequest.SerializeToString,
opc__gateway__pb2.ReadTagResponse.FromString,
options,
channel_credentials,
insecure,
call_credentials,
compression,
wait_for_ready,
timeout,
metadata,
_registered_method=True)

21
pyproject.toml
View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "ingestor"
version = "2.1.0"
version = "0.0.0"
description = "Sientia DataOps Ingestor - OPC Tag Ingestor"
readme = "README.md"
requires-python = ">=3.11"
@@ -23,7 +23,6 @@ exclude = [
"*.pyc",
".pytest_cache",
"htmlcov",
"ingestor/proto/*_pb2*.py",
]
[tool.ruff.lint]
@@ -59,11 +58,6 @@ ignore = [
"S105", # hardcoded passwords ok in tests
"S106", # hardcoded passwords ok in tests
]
"e2e/**/*.py" = [
"S101", # assert allowed in tests
"S105", # hardcoded passwords ok in tests
"S106", # hardcoded passwords ok in tests
]
[tool.ruff.lint.mccabe]
max-complexity = 15
@@ -86,9 +80,6 @@ warn_unused_ignores = false
warn_no_return = true
strict_equality = true
ignore_missing_imports = true
# Generated protoc/grpcio-tools output - not a checking target, dynamic attrs
# mypy can't see (message classes built at runtime from DESCRIPTOR).
exclude = ["^ingestor/proto/"]
# Ignore missing imports for external packages
[[tool.mypy.overrides]]
@@ -115,14 +106,6 @@ ignore_missing_imports = true
module = "pandas.*"
ignore_missing_imports = true
# Generated protoc/grpcio-tools output - not hand-maintained, dynamic attrs
# mypy can't see (message classes built at runtime from DESCRIPTOR). Skip
# following it entirely so importers see Any instead of false attr-defined errors.
[[tool.mypy.overrides]]
module = "ingestor.proto.*"
ignore_errors = true
follow_imports = "skip"
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
@@ -134,7 +117,7 @@ addopts = [
]
markers = [
"asyncio: marks tests as async",
"integration: marks tests as integration tests (the e2e/ suite: Docker required, run with -o testpaths=e2e -m integration)",
"integration: marks tests as integration tests",
"unit: marks tests as unit tests",
]

10
requirements-dev.txt
View File

@@ -13,15 +13,7 @@ types-requests>=2.31.0 # Type stubs for requests
pytest>=7.4.0 # Testing framework
pytest-cov>=4.1.0 # Coverage plugin for pytest
pytest-asyncio>=0.21.0 # Async test support (already in main requirements)
testcontainers[postgres,minio,mongodb,redis]>=4.15.0 # E2E suite (e2e/): real MongoDB, Redis, PostgreSQL,
# MinIO and NATS containers; `testcontainers.community.*` namespace
# Development Tools
ipython>=8.12.0 # Enhanced Python shell
ipdb>=0.13.13 # IPython debugger
python-dotenv>=1.0 # local dev only: app/seed read .env if present (in-cluster env comes from Helm)
# gRPC code generation (regenerate ingestor/proto/*_pb2*.py) — not needed
# at runtime: the generated stubs are committed. Kept out of requirements.txt
# because workers install only from the cluster package index.
grpcio-tools>=1.60
ipdb>=0.13.13 # IPython debugger

2
requirements-local.txt
View File

@@ -1,4 +1,4 @@
asyncua>=2.0
asyncua==1.1.5
redis
git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.12.2
prometheus_client

15
requirements.txt
View File

@@ -1,14 +1,5 @@
# >=2.0: WriteEngine builds DataValue(StatusCode=...); 1.x names the field StatusCode_,
# so every write would fail with TypeError (reported as bad_timestamp).
asyncua>=2.0
asyncua
redis
# 1.12.5 builds the Postgres engine with an explicit postgresql+psycopg2:// URL: SQLAlchemy 2.1
# defaults postgresql:// to psycopg (v3), which is not installed ("No module named 'psycopg'").
sientia_do==1.12.5
sientia_do==1.12.2
prometheus_client
pymongo
nats-py>=2.6
grpcio>=1.60
# Runtime of the committed gRPC stubs: ingestor/proto/opc_gateway_pb2.py was
# generated with protobuf 7.35.0 and validates the runtime version on import.
protobuf>=7.35
pymongo

95
scripts/generate-and-push-opc-cert.sh Executable file
View File

@@ -0,0 +1,95 @@
#!/usr/bin/env bash
# Gera o certificado X.509 de cliente OPC UA do ingestor e sobe pro Gitea
# in-cluster (repo que o pod clona no boot — nada muda no values/deployment,
# só rollout restart depois do push).
#
# Uso:
# GITEA_PASSWORD='<senha>' ./scripts/generate-and-push-opc-cert.sh
#
# Vars (defaults pro cluster dev/gcp):
# GITEA_URL default https://git.sientia.ai
# GITEA_USER default gitea_admin
# GITEA_PASSWORD obrigatória
# GITEA_REPO default gitea_admin/sientia-dataops-opc-ingestor
# BRANCH default main
# APP_URI default urn:sientia:opc-ingestor (SAN URI do cert — TEM que
# ser igual ao campo `uri` do documento na collection
# OPC_servers: o ingestor usa esse valor como application_uri
# e o KEPServer rejeita com BadCertificateUriInvalid se divergir)
# DAYS default 3650
# FORCE=1 sobrescreve cert já existente no repo
set -euo pipefail
GITEA_URL=${GITEA_URL:-https://git.sientia.ai}
GITEA_USER=${GITEA_USER:-gitea_admin}
GITEA_PASSWORD=${GITEA_PASSWORD:?export GITEA_PASSWORD com a senha do Gitea}
GITEA_REPO=${GITEA_REPO:-gitea_admin/sientia-dataops-opc-ingestor}
BRANCH=${BRANCH:-main}
APP_URI=${APP_URI:-urn:sientia:opc-ingestor}
CN=${CN:-sientia-opc-ingestor}
DAYS=${DAYS:-3650}
CERT_DIR_IN_REPO="certs"
CERT_NAME="opc-ingestor-cert.pem"
KEY_NAME="opc-ingestor-key.pem"
DER_NAME="opc-ingestor-cert.der"
workdir=$(mktemp -d)
trap 'rm -rf "$workdir"' EXIT
echo ">> Gerando certificado (CN=${CN}, URI=${APP_URI}, ${DAYS} dias)..."
# Extensões exigidas por OPC UA (KEPServer valida keyUsage/EKU e o SAN URI).
openssl req -x509 -newkey rsa:2048 -sha256 -nodes -days "$DAYS" \
-keyout "$workdir/$KEY_NAME" -out "$workdir/$CERT_NAME" \
-subj "/CN=${CN}/O=Aignosi/OU=Sientia" \
-addext "subjectAltName=URI:${APP_URI},DNS:${CN}" \
-addext "keyUsage=critical,digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment" \
-addext "extendedKeyUsage=critical,clientAuth,serverAuth" \
-addext "basicConstraints=critical,CA:FALSE"
# Cópia DER — só referência p/ import manual no KEPServer (o trust normal é
# aceitar o cert rejeitado na 1ª conexão).
openssl x509 -in "$workdir/$CERT_NAME" -outform der -out "$workdir/$DER_NAME"
echo ">> Clonando ${GITEA_REPO}@${BRANCH} do Gitea..."
encoded_pass=$(python3 - <<EOF
import urllib.parse; print(urllib.parse.quote('''${GITEA_PASSWORD}''', safe=''))
EOF
)
clone_url="${GITEA_URL%%/}"
clone_url="${clone_url/:\/\//:\/\/${GITEA_USER}:${encoded_pass}@}/${GITEA_REPO}.git"
git clone --depth 1 --branch "$BRANCH" "$clone_url" "$workdir/repo" 2>&1 | sed "s#${encoded_pass}#***#g"
if [[ -f "$workdir/repo/$CERT_DIR_IN_REPO/$CERT_NAME" && "${FORCE:-0}" != "1" ]]; then
echo "ERRO: $CERT_DIR_IN_REPO/$CERT_NAME já existe no repo. Re-rodar com FORCE=1 sobrescreve" >&2
echo " (o cert antigo deixa de valer — retrust no KEPServer necessário)." >&2
exit 1
fi
mkdir -p "$workdir/repo/$CERT_DIR_IN_REPO"
cp "$workdir/$CERT_NAME" "$workdir/$KEY_NAME" "$workdir/$DER_NAME" "$workdir/repo/$CERT_DIR_IN_REPO/"
cd "$workdir/repo"
git add "$CERT_DIR_IN_REPO"
git -c user.name="opc-cert-script" -c user.email="platform@aignosi.com.br" \
commit -m "Add OPC UA client certificate for KEPServer connection (URI ${APP_URI})"
git push origin "$BRANCH" 2>&1 | sed "s#${encoded_pass}#***#g"
fingerprint=$(openssl x509 -in "$CERT_DIR_IN_REPO/$CERT_NAME" -noout -fingerprint -sha1)
cat <<EOF
== PRONTO ==
Cert no Gitea: ${GITEA_REPO}@${BRANCH} -> ${CERT_DIR_IN_REPO}/{${CERT_NAME},${KEY_NAME},${DER_NAME}}
${fingerprint}
Paths dentro do pod (o entrypoint clona o repo em /app):
cert_path: /app/${CERT_DIR_IN_REPO}/${CERT_NAME}
private_key_path: /app/${CERT_DIR_IN_REPO}/${KEY_NAME}
Próximos passos:
1. Documento na collection OPC_servers com "uri": "${APP_URI}" (ver README/PR).
2. kubectl -n sientia rollout restart deployment/opc-ingestor # re-clona o repo
3. Pod tenta conectar -> no KEPServer: OPC UA Configuration Manager ->
Trusted Clients -> cert '${CN}' rejeitado -> Trust -> restart runtime.
EOF

172
scripts/seed_e2e.py
View File

@@ -1,172 +0,0 @@
"""Seed test data for the OPC Gateway e2e validation session (SIENTIAPDE-1972).
Populates:
1. Mongo `connectors` + `connector_tags` (each tag with `metadata: {}` — required
for documentation, per the feature owner).
2. Redis slot v2 (`slot:opc_tags:1`, contract C1) so the subscription path collects them.
3. Postgres test predictions table (with the `write_status` column) + one row the
NATS write test can UPDATE (correlated by timestamp + model_id).
Idempotent: connectors/tags are upserted, the slot is overwritten, the table uses
CREATE IF NOT EXISTS + ON CONFLICT DO NOTHING.
Usage (from the repo root, sientia-dataops conda env):
python scripts/seed_e2e.py --print-only # dry-run: show docs/SQL, touch nothing
python scripts/seed_e2e.py # seed Mongo + Redis + Postgres
Env (same names the gateway/app use):
MONGODB_URL/MONGODB_USERNAME/MONGODB_PASSWORD/MONGODB_DATABASE
REDIS_HOST/REDIS_PORT POSTGRES_HOST/PORT/USER/PASSWORD/DBNAME
OPC_SERVER_URL (default opc.tcp://localhost:49320 — KepServer default port)
# ponytail: plain sync pymongo/redis/sqlalchemy, no sientia_do async plumbing —
# this is a test fixture, not production code.
"""
import argparse
import json
import os
import sys
try: # optional: local dev reads .env; in-cluster env comes from Helm
from dotenv import load_dotenv
load_dotenv()
except ImportError:
pass
OPC_URL = os.getenv('OPC_SERVER_URL', 'opc.tcp://localhost:49320')
# No client certificate: the gateway connects without security (SIENTIAPDE-1974 D5).
# To test security, point client_certificate_id at an active client certificate of
# opc_certificates and set mode/policy (e.g. SignAndEncrypt / Basic256Sha256).
SECURITY = {'mode': 'None', 'policy': 'None', 'client_certificate_id': None}
CONNECTOR = {
'connector_id': 'e2e-kepserver',
'type': 'opc_ua',
'name': 'e2e-kepserver',
'url': OPC_URL,
'subscription_period_ms': 1000,
'security': SECURITY,
'status': 'Active',
'description': 'OPC Gateway e2e validation connector (SIENTIAPDE-1972)',
}
# metadata: {} on every tag — feature owner asked test tags to carry the field.
TAGS = [
{'tag_id': 9101, 'connector_id': 'e2e-kepserver', 'address': 'ns=2;s=Simulation Examples.Functions.Ramp1',
'tag_name': 'e2e_ramp', 'category': 'Read', 'data_type': 'float', 'unit': '-',
'status': 'Active', 'description': 'e2e read tag', 'metadata': {}},
{'tag_id': 9102, 'connector_id': 'e2e-kepserver', 'address': 'ns=2;s=Simulation Examples.Functions.User1',
'tag_name': 'e2e_write_target', 'category': 'Write', 'data_type': 'float', 'unit': '-',
'status': 'Active', 'description': 'e2e write tag', 'metadata': {}},
]
# Slot v2 (contract C1), as the Orchestrator writes it for the connector above.
SLOT = {
'e2e-kepserver': {
'name': 'e2e-kepserver',
'server_id': 'e2e-kepserver',
'url': OPC_URL,
'subscription_period_ms': 1000,
'security': {'mode': SECURITY['mode'], 'policy': SECURITY['policy']},
'certificate': None,
'tags': {
TAGS[0]['address']: {
'frequency': 5000,
'active_read': {'enabled': False},
'names': [
{
'collection': 'raw_e2e-gateway-test',
'tag_name': TAGS[0]['tag_name'],
'model_id': '9001',
'model_name': 'e2e-model',
}
],
}
},
}
}
PREDICTIONS_SQL = """
CREATE SCHEMA IF NOT EXISTS sientia_data;
CREATE TABLE IF NOT EXISTS sientia_data.e2e_predictions (
model_id text NOT NULL,
prediction double precision,
prediction_confidence double precision,
timestamp text NOT NULL,
write_status text,
PRIMARY KEY (model_id, timestamp)
);
INSERT INTO sientia_data.e2e_predictions (model_id, prediction, prediction_confidence, timestamp)
VALUES ('9001', 42.5, 0.97, '2026-07-15T12:00:00Z')
ON CONFLICT DO NOTHING;
"""
# The matching NATS write command for the row above (paste into `nats pub opc.write '...'`).
NATS_CMD = {
'msg_id': 'e2e-0001', 'server_id': 'e2e-kepserver', 'addr': TAGS[1]['address'],
'value': 42.5, 'data_type': 'float', 'tag_type': 'prediction',
'model_id': '9001', 'model_name': 'e2e-model',
'schema': 'sientia_data', 'table_name': 'e2e_predictions',
'timestamp': '2026-07-15T12:00:00Z',
}
def seed_mongo():
from pymongo import MongoClient
url = os.getenv('MONGODB_URL', 'localhost:27017')
user, pwd = os.getenv('MONGODB_USERNAME', 'sientia'), os.getenv('MONGODB_PASSWORD', 'sientia')
db = MongoClient(f'mongodb://{user}:{pwd}@{url}')[os.getenv('MONGODB_DATABASE', 'sientia')]
db['connectors'].replace_one({'connector_id': CONNECTOR['connector_id']}, CONNECTOR, upsert=True)
for tag in TAGS:
db['connector_tags'].replace_one({'tag_id': tag['tag_id']}, tag, upsert=True)
print(f'mongo: upserted 1 connector + {len(TAGS)} tags (metadata: {{}})')
def seed_redis():
import redis
r = redis.Redis(host=os.getenv('REDIS_HOST', 'localhost'), port=int(os.getenv('REDIS_PORT', '6379')))
r.set('slot:opc_tags:1', json.dumps(SLOT))
print('redis: wrote slot:opc_tags:1 (v2)')
def seed_postgres():
from sqlalchemy import create_engine, text
host, port = os.getenv('POSTGRES_HOST', 'localhost'), os.getenv('POSTGRES_PORT', '5432')
user, pwd = os.getenv('POSTGRES_USER', 'sientia'), os.getenv('POSTGRES_PASSWORD', 'sientia')
dbname = os.getenv('POSTGRES_DBNAME', 'sientia')
engine = create_engine(f'postgresql://{user}:{pwd}@{host}:{port}/{dbname}')
with engine.begin() as conn:
for stmt in filter(str.strip, PREDICTIONS_SQL.split(';')):
conn.execute(text(stmt))
print('postgres: sientia_data.e2e_predictions ready (write_status column + 1 row)')
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--print-only', action='store_true', help='show docs/SQL, touch nothing')
args = parser.parse_args()
if args.print_only:
print('connector:', json.dumps(CONNECTOR, indent=2))
print('tags:', json.dumps(TAGS, indent=2))
print('slot v2:', json.dumps(SLOT, indent=2))
print('sql:', PREDICTIONS_SQL)
print('nats write cmd:', json.dumps(NATS_CMD, indent=2))
return 0
seed_mongo()
seed_redis()
seed_postgres()
print('\nNATS write command for the session:')
print(f"nats pub opc.write '{json.dumps(NATS_CMD)}'")
return 0
if __name__ == '__main__':
sys.exit(main())

4
sonar-project.properties
View File

@@ -8,7 +8,3 @@ sonar.python.coverage.reportPaths=coverage.xml
sonar.python.xunit.reportPath=pytest.xml
sonar.python.version=3.11
sonar.coverage.exclusions=ingestor/app.py
# Generated protoc/grpcio-tools output - not hand-maintained; renaming its
# ReadTag/etc. methods to satisfy naming-convention rules would break gRPC
# wire compatibility (grpcio-tools dictates these names, not style choice).
sonar.exclusions=ingestor/proto/**

0
e2e/__init__.py → tests/functional/__init__.py
View File

47
tests/functional/conftest.py Normal file
View File

@@ -0,0 +1,47 @@
# import subprocess
# from time import sleep
# from typing import Generator
# import uuid
# from kafka import KafkaConsumer
# import pytest
# from redis import Redis
# @pytest.fixture(scope="session", autouse=True)
# def docker_compose():
# """Sobe os containers antes dos testes e derruba depois."""
# print("\n🚀 Subindo Docker Compose...")
# subprocess.run(["docker", "compose", "up", "-d"], check=True)
# print("⏳ Aguardando containers ficarem prontos...")
# sleep(15) # ajuste conforme necessário
# yield # os testes rodam aqui
# print("\n🧹 Derrubando Docker Compose...")
# subprocess.run(["docker", "compose", "down"], check=True)
# @pytest.fixture()
# def redis_client():
# redis = Redis(host="localhost", port=6379, decode_responses=True)
# redis.flushdb()
# yield redis
# # Limpa o banco de dados após os testes
# redis.flushdb()
# redis.close()
# def kafka_searcher(topic) -> Generator[KafkaConsumer, None, None]:
# consumer = KafkaConsumer(
# topic,
# bootstrap_servers="localhost:9092",
# group_id=f"test-group-{uuid.uuid4()}",
# auto_offset_reset="earliest", # Começa a consumir apenas mensagens novas
# enable_auto_commit=True,
# )
# yield consumer
# consumer.close()

118
tests/functional/test_single_node.py Normal file
View File

@@ -0,0 +1,118 @@
# import json
# import subprocess
# from time import sleep
# from tests.functional.conftest import kafka_searcher
# new_data = {
# "slot:opc_tags:1": {
# "server1": {
# "name": "server1",
# "url": "opc.tcp://simulator:4840",
# "server_uri": "http://opcua-server.simulator",
# "tags": {
# 'ns=2;i=2': {
# 'tag_name': 'Counter',
# 'frequency': 1000,
# 'topics': [],
# },
# 'ns=2;i=3': {
# 'tag_name': 'Rollout',
# 'frequency': 1000,
# "topics": [],
# },
# 'ns=2;i=4': {
# 'tag_name': 'Square',
# 'frequency': 1000,
# "topics": [],
# },
# }
# }
# },
# "slot:opc_tags:2": {
# "server2": {
# "name": "server2",
# "url": "opc.tcp://simulator:4840",
# "server_uri": "http://opcua-server.simulator",
# "tags": {
# 'ns=2;i=2': {
# 'tag_name': 'Counter',
# 'frequency': 1000,
# 'topics': [],
# },
# 'ns=2;i=3': {
# 'tag_name': 'Rollout',
# 'frequency': 1000,
# "topics": [],
# },
# 'ns=2;i=4': {
# 'tag_name': 'Square',
# 'frequency': 1000,
# "topics": [],
# },
# }
# }
# }
# }
# def test_simple(redis_client):
# new_data['slot:opc_tags:1']['server1']['tags']['ns=2;i=2']['topics'] = [
# 'test_topic_1']
# redis_client.set("slot:opc_tags:1",
# json.dumps(new_data['slot:opc_tags:1']))
# sleep(20) # Espera o Ingestor processar os dados
# # Check if lease is in Redis
# assert redis_client.get("lease:opc_tags:1") == 'ingestor'
# assert redis_client.get("heartbeat:ingestor:ingestor") == '1'
# # Check if data is in Kafka
# kafka = next(kafka_searcher('test_topic_1'))
# sleep(1)
# messages = kafka.poll(timeout_ms=10000)
# assert messages, "Expected messages in Kafka, but got none."
# def test_simple_double_slot(redis_client):
# new_data['slot:opc_tags:1']['server1']['tags']['ns=2;i=2']['topics'] = [
# 'test_topic_double_slot1']
# redis_client.set("slot:opc_tags:1",
# json.dumps(new_data['slot:opc_tags:1']))
# sleep(20) # Espera o Ingestor processar os dados
# assert redis_client.get("lease:opc_tags:1") == 'ingestor'
# assert redis_client.get("heartbeat:ingestor:ingestor") == '1'
# # Check if data is in Kafka
# kafka1 = next(kafka_searcher('test_topic_double_slot1'))
# messages = kafka1.poll(timeout_ms=10000)
# assert messages, "Expected messages in test_topic_double_slot1, but got none."
# new_data['slot:opc_tags:2']['server2']['tags']['ns=2;i=2']['topics'] = [
# 'test_topic_double_slot2']
# redis_client.set("slot:opc_tags:2",
# json.dumps(new_data['slot:opc_tags:2']))
# sleep(20) # Espera o Ingestor processar os dados
# # Check if lease is in Redis
# assert redis_client.get("lease:opc_tags:2") == 'ingestor'
# assert redis_client.get("lease:opc_tags:1") == 'ingestor'
# assert redis_client.get("heartbeat:ingestor:ingestor") == '1'
# # Check if data is in Kafka
# kafka2 = next(kafka_searcher('test_topic_double_slot2'))
# messages = kafka2.poll(timeout_ms=10000)
# assert messages, "Expected messages in test_topic_double_slot2, but got none."
# messages = kafka1.poll(timeout_ms=10000)
# assert messages, "Expected messages in test_topic_double_slot1, but got none."

0
tests/unit/adapters/__init__.py
View File

97
tests/unit/adapters/test_grpc_server.py
View File

@@ -1,97 +0,0 @@
from unittest.mock import AsyncMock, MagicMock
import grpc
import pytest
from ingestor.adapters.grpc_server import GrpcServer, ReadTagServicer
from ingestor.proto.opc_gateway_pb2 import ReadTagRequest
def make_request(server_id='srv1', addr='ns=2;i=5'):
return ReadTagRequest(server_id=server_id, addr=addr, data_type='float')
@pytest.mark.asyncio
async def test_read_tag_maps_full_result_onto_response():
active_read = AsyncMock()
active_read.read_now.return_value = {
'value': 1.5,
'opc_status_code': 'Good',
'source_timestamp': '2026-07-15T10:00:00-03:00',
'server_timestamp': '2026-07-15T10:00:01-03:00',
'persisted': False,
}
servicer = ReadTagServicer(active_read, MagicMock())
response = await servicer.ReadTag(make_request(), AsyncMock())
assert response.value == '1.5'
assert response.opc_status_code == 'Good'
assert response.source_timestamp == '2026-07-15T10:00:00-03:00'
assert response.server_timestamp == '2026-07-15T10:00:01-03:00'
assert response.persisted is False
@pytest.mark.asyncio
async def test_read_tag_none_timestamps_become_empty_string():
active_read = AsyncMock()
active_read.read_now.return_value = {
'value': 1.5,
'opc_status_code': None,
'source_timestamp': None,
'server_timestamp': None,
'persisted': False,
}
servicer = ReadTagServicer(active_read, MagicMock())
response = await servicer.ReadTag(make_request(), AsyncMock())
assert response.opc_status_code == ''
assert response.source_timestamp == ''
assert response.server_timestamp == ''
@pytest.mark.asyncio
async def test_read_tag_aborts_with_internal_on_engine_failure():
active_read = AsyncMock()
active_read.read_now.side_effect = RuntimeError('boom')
logger = MagicMock()
servicer = ReadTagServicer(active_read, logger)
context = AsyncMock()
context.abort.side_effect = grpc.RpcError('aborted')
request = make_request()
with pytest.raises(grpc.RpcError):
await servicer.ReadTag(request, context)
context.abort.assert_awaited_once()
args = context.abort.call_args.args
assert args[0] == grpc.StatusCode.INTERNAL
logger.error.assert_called_once()
@pytest.mark.asyncio
async def test_read_tag_uses_slot_lookup_names_when_provided():
active_read = AsyncMock()
active_read.read_now.return_value = {
'value': 1,
'opc_status_code': 'Good',
'source_timestamp': None,
'server_timestamp': None,
'persisted': True,
}
names = [{'collection': 'c1', 'name': 't1'}]
slot_lookup = MagicMock(return_value=names)
servicer = ReadTagServicer(active_read, MagicMock(), slot_lookup=slot_lookup)
await servicer.ReadTag(make_request(server_id='srv1', addr='ns=2;i=5'), AsyncMock())
slot_lookup.assert_called_once_with('srv1', 'ns=2;i=5')
active_read.read_now.assert_awaited_once_with('srv1', 'ns=2;i=5', names=names)
@pytest.mark.asyncio
async def test_grpc_server_stop_before_start_does_not_raise():
server = GrpcServer(active_read=AsyncMock(), logger=MagicMock())
await server.stop() # no raise

171
tests/unit/adapters/test_nats_consumer.py
View File

@@ -1,171 +0,0 @@
import asyncio
import inspect
import json
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from ingestor.adapters.nats_consumer import NatsWriteConsumer
def make_consumer(**kwargs):
return NatsWriteConsumer(
nats_url='nats://localhost:4222',
write_engine=AsyncMock(),
logger=MagicMock(),
**kwargs,
)
@pytest.mark.asyncio
async def test_handle_valid_json_dispatches_to_write_engine():
consumer = make_consumer()
payload = {'server_id': 'srv1', 'addr': 'ns=2;i=5', 'value': 1.0, 'data_type': 'float'}
msg = MagicMock(data=json.dumps(payload).encode())
await consumer._handle(msg)
consumer.write_engine.write.assert_awaited_once_with(payload)
@pytest.mark.asyncio
async def test_handle_invalid_json_drops_message_and_logs_error():
consumer = make_consumer()
msg = MagicMock(data=b'not-json')
await consumer._handle(msg)
consumer.write_engine.write.assert_not_awaited()
consumer.logger.error.assert_called_once()
CLIENT = 'ingestor.adapters.nats_consumer.Client'
def _fake_nc(connect=None):
nc = MagicMock()
nc.connect = connect or AsyncMock()
nc.subscribe = AsyncMock()
nc.close = AsyncMock()
nc.drain = AsyncMock()
nc.is_closed = False
return nc
async def _hang(*_args, **_kwargs):
await asyncio.sleep(10)
@pytest.mark.asyncio
async def test_start_subscribes_with_subject_and_queue():
consumer = make_consumer(subject='opc.write', queue='gateway-writers')
nc = _fake_nc()
with patch(CLIENT, return_value=nc):
await consumer.start()
kwargs = nc.connect.await_args.kwargs
assert kwargs['servers'] == 'nats://localhost:4222'
# Reconnection is unbounded once connected: an outage never closes it for good.
assert kwargs['max_reconnect_attempts'] == -1
nc.subscribe.assert_awaited_once_with('opc.write', queue='gateway-writers', cb=consumer._handle)
assert consumer.closed is False
@pytest.mark.asyncio
async def test_start_gives_up_after_the_connect_timeout():
"""start() runs on the main loop: an unreachable NATS must not block it."""
consumer = make_consumer(connect_timeout_s=0.1)
nc = _fake_nc(connect=AsyncMock(side_effect=_hang))
# Outer guard: fails the test instead of hanging it if start() ignores its timeout.
start = asyncio.wait_for(consumer.start(), timeout=5)
with patch(CLIENT, return_value=nc):
with pytest.raises(TimeoutError):
await start
nc.close.assert_awaited_once()
assert consumer.closed is True
@pytest.mark.asyncio
async def test_start_closes_the_connection_when_subscribe_fails():
consumer = make_consumer()
nc = _fake_nc()
nc.subscribe.side_effect = RuntimeError('permissions violation')
with patch(CLIENT, return_value=nc), pytest.raises(RuntimeError):
await consumer.start()
nc.close.assert_awaited_once()
assert consumer.closed is True
@pytest.mark.asyncio
async def test_handle_unexpected_engine_error_is_logged_not_raised():
consumer = make_consumer()
consumer.write_engine.write.side_effect = RuntimeError('bug')
msg = MagicMock(data=json.dumps({'server_id': '1'}).encode())
await consumer._handle(msg)
assert 'bug' in consumer.logger.error.call_args.args[0]
@pytest.mark.asyncio
async def test_connection_events_are_logged_in_one_line():
consumer = make_consumer()
nc = _fake_nc()
with patch(CLIENT, return_value=nc):
await consumer.start()
callbacks = nc.connect.await_args.kwargs
# nats-py raises InvalidCallbackTypeError on connect for a non-coroutine callback.
for name in ('error_cb', 'disconnected_cb', 'reconnected_cb'):
assert inspect.iscoroutinefunction(callbacks[name]), name
await callbacks['error_cb'](OSError('connection refused'))
await callbacks['disconnected_cb']()
await callbacks['reconnected_cb']()
warnings = [c.args[0] for c in consumer.logger.warning.call_args_list]
assert warnings == [
'NATS error: OSError connection refused',
'Disconnected from NATS; reconnecting',
]
@pytest.mark.asyncio
async def test_stop_before_start_does_not_raise():
consumer = make_consumer()
await consumer.stop() # no raise
@pytest.mark.asyncio
async def test_stop_after_start_drains_and_unsubscribes():
consumer = make_consumer()
nc = _fake_nc()
sub = AsyncMock()
nc.subscribe = AsyncMock(return_value=sub)
with patch(CLIENT, return_value=nc):
await consumer.start()
await consumer.stop()
sub.unsubscribe.assert_awaited_once()
nc.drain.assert_awaited_once()
@pytest.mark.asyncio
async def test_stop_on_a_connection_closed_for_good_does_not_raise():
consumer = make_consumer()
nc = _fake_nc()
with patch(CLIENT, return_value=nc):
await consumer.start()
nc.is_closed = True
consumer.sub.unsubscribe = AsyncMock(side_effect=RuntimeError('connection closed'))
await consumer.stop() # no raise
nc.drain.assert_not_awaited()
assert consumer.closed is True

20
tests/unit/conftest.py
View File

@@ -1,20 +0,0 @@
from unittest.mock import AsyncMock, MagicMock
import pytest
@pytest.fixture
def reader_double():
"""Factory of OPC reader doubles holding subscriptions for the given slots.
Like the real ServerReader, `subscriptions` is keyed by slot
and `unsubscribe` drops the slot from it.
"""
def make(*slots):
reader = MagicMock()
reader.subscriptions = {slot: MagicMock() for slot in slots}
reader.unsubscribe = AsyncMock(side_effect=lambda slot: reader.subscriptions.pop(slot))
return reader
return make

0
tests/unit/connection/__init__.py
View File

67
tests/unit/connection/test_certificates.py
View File

@@ -1,67 +0,0 @@
import pytest
from ingestor.connection.certificates import (
client_certificate_id,
has_security,
resolve_certificate,
)
LOOKED_UP = {
'certificate_id': 1,
'application_uri': 'urn:sientia:opc-ingestor',
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/p/cert.pem'},
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/p/key.pem'},
}
def _connector(certificates, certificate_id=1):
return {
'connector_id': 'kepserver',
'url': 'opc.tcp://kep:49320',
'security': {
'mode': 'SignAndEncrypt',
'policy': 'Basic256Sha256',
'client_certificate_id': certificate_id,
},
'certificates': certificates,
}
def test_resolve_certificate_keeps_only_the_slot_fields():
looked_up = {
**LOOKED_UP,
'thumbprint_sha1': 'ab',
'cert': {**LOOKED_UP['cert'], 'sha256': None},
}
assert resolve_certificate(_connector([looked_up])) == LOOKED_UP
@pytest.mark.parametrize(
'certificates',
[
pytest.param(None, id='no-lookup'),
pytest.param([], id='not-found'),
pytest.param([{**LOOKED_UP, 'application_uri': ''}], id='no-application-uri'),
pytest.param([{**LOOKED_UP, 'cert': None}], id='no-cert'),
pytest.param([{**LOOKED_UP, 'private_key': {'object_name': 'k'}}], id='key-no-bucket'),
pytest.param(['not a document'], id='not-a-document'),
],
)
def test_resolve_certificate_without_a_usable_certificate(certificates):
assert resolve_certificate(_connector(certificates)) is None
def test_client_certificate_id():
assert client_certificate_id(_connector([])) == 1
assert client_certificate_id(_connector([], certificate_id=None)) is None
assert client_certificate_id({'connector_id': 'x'}) is None
assert client_certificate_id({'security': None}) is None
def test_has_security():
assert has_security(_connector([]))
assert has_security({'security': {'mode': 'None', 'policy': 'None'}})
assert not has_security({'security': {'mode': 'Sign'}})
assert not has_security({'security': None})
assert not has_security({})

161
tests/unit/connection/test_certs.py
View File

@@ -1,161 +0,0 @@
import asyncio
from unittest.mock import AsyncMock, MagicMock
import pytest
from ingestor.connection.certs import CertificateNotConfiguredError, CertProvider, LazyMinio
def certificate(certificate_id=1, name='sientia-platform', ext='pem', bucket='opc-certificates'):
"""A client certificate as the slot v2 carries it."""
return {
'certificate_id': certificate_id,
'application_uri': 'urn:sientia:opc-ingestor',
'cert': {'bucket': bucket, 'object_name': f'client/{name}/cert.{ext}'},
'private_key': {'bucket': bucket, 'object_name': f'client/{name}/key.{ext}'},
}
def _provider(tmp_path, data=b'pem'):
minio = MagicMock()
minio.download_file = AsyncMock(return_value=data)
return CertProvider(minio=minio, cache_dir=str(tmp_path)), minio
@pytest.mark.asyncio
async def test_fetch_downloads_the_objects_the_certificate_names(tmp_path):
minio = AsyncMock()
minio.download_file.side_effect = [b'cert-bytes', b'key-bytes']
provider = CertProvider(minio, cache_dir=str(tmp_path))
cert, key = await provider.fetch(certificate(7))
assert (cert, key) == (str(tmp_path / '7' / 'cert.pem'), str(tmp_path / '7' / 'key.pem'))
assert [c.args for c in minio.download_file.await_args_list] == [
('client/sientia-platform/cert.pem', 'opc-certificates', {}),
('client/sientia-platform/key.pem', 'opc-certificates', {}),
]
assert (tmp_path / '7' / 'cert.pem').read_bytes() == b'cert-bytes'
assert (tmp_path / '7' / 'key.pem').read_bytes() == b'key-bytes'
@pytest.mark.asyncio
async def test_local_files_keep_the_object_extension(tmp_path):
"""asyncua picks the format (.pem / .der) by the extension"""
provider, _minio = _provider(tmp_path)
cert, key = await provider.fetch(certificate(ext='der'))
assert cert.endswith('cert.der')
assert key.endswith('key.der')
@pytest.mark.asyncio
@pytest.mark.parametrize(
'broken',
[
pytest.param({'cert': None}, id='no-cert'),
pytest.param({'private_key': {'bucket': 'b'}}, id='key-without-object'),
pytest.param({'certificate_id': None}, id='no-id'),
pytest.param({'certificate_id': '../etc'}, id='id-not-an-integer'),
],
)
async def test_fetch_of_an_incomplete_certificate_raises_clear_error(tmp_path, broken):
provider, minio = _provider(tmp_path)
incomplete = {**certificate(), **broken}
with pytest.raises(CertificateNotConfiguredError, match='opc_certificates|certificate_id'):
await provider.fetch(incomplete)
minio.download_file.assert_not_awaited()
@pytest.mark.asyncio
async def test_fetch_is_cached_per_certificate_until_invalidated(tmp_path):
provider, minio = _provider(tmp_path)
first = await provider.fetch(certificate(1))
assert await provider.fetch(certificate(1)) == first
assert minio.download_file.await_count == 2 # cert + key, once
other = await provider.fetch(certificate(2, name='other'))
assert other != first
assert minio.download_file.await_count == 4
provider.invalidate(1)
await provider.fetch(certificate(1))
await provider.fetch(certificate(2, name='other')) # still cached
assert minio.download_file.await_count == 6
@pytest.mark.asyncio
async def test_a_replaced_certificate_is_downloaded_again(tmp_path):
"""Same certificate_id, other objects in the slot: the cache does not serve the old files"""
provider, minio = _provider(tmp_path)
await provider.fetch(certificate(1, name='old'))
await provider.fetch(certificate(1, name='new'))
assert minio.download_file.await_count == 4
assert minio.download_file.await_args_list[-1].args[0] == 'client/new/key.pem'
@pytest.mark.asyncio
async def test_concurrent_fetches_download_once(tmp_path):
provider, minio = _provider(tmp_path)
results = await asyncio.gather(*(provider.fetch(certificate()) for _ in range(5)))
assert len(set(results)) == 1
assert minio.download_file.await_count == 2
@pytest.mark.asyncio
async def test_download_replaces_file_atomically(tmp_path):
provider, _minio = _provider(tmp_path, data=b'new')
(tmp_path / '1').mkdir()
(tmp_path / '1' / 'cert.pem').write_bytes(b'old')
cert_path, _ = await provider.fetch(certificate(1))
assert (tmp_path / '1' / 'cert.pem').read_bytes() == b'new'
assert cert_path == str(tmp_path / '1' / 'cert.pem')
assert not list(tmp_path.glob('**/*.tmp'))
@pytest.mark.asyncio
async def test_lazy_minio_builds_one_client_per_bucket_on_first_download():
clients = {'a': AsyncMock(), 'b': AsyncMock()}
for client in clients.values():
client.download_file.return_value = b'bytes'
factory = MagicMock(side_effect=lambda bucket: clients[bucket])
minio = LazyMinio(factory)
factory.assert_not_called() # no network at construction
assert await minio.download_file('x.pem', 'a', {}) == b'bytes'
await minio.download_file('y.pem', 'a', {})
await minio.download_file('z.pem', 'b', {})
assert [c.args for c in factory.call_args_list] == [('a',), ('b',)]
assert clients['a'].download_file.await_count == 2
clients['b'].download_file.assert_awaited_once_with('z.pem', 'b', {})
def test_build_cert_provider_is_lazy_and_uses_config(monkeypatch):
from ingestor.config import GatewayConfig
from ingestor.connection import certs
minio_cls = MagicMock()
monkeypatch.setattr(certs, 'MinioRepository', minio_cls)
provider = certs.build_cert_provider(
config=GatewayConfig(cert_cache_dir='certs-cache'),
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=MagicMock(),
)
assert provider.cache_dir == 'certs-cache'
assert isinstance(provider.minio, certs.LazyMinio)
minio_cls.assert_not_called() # MinIO is only reached when a cert is needed
provider.minio._factory('opc-certificates')
assert minio_cls.call_args.kwargs['bucket'] == 'opc-certificates'

10
tests/unit/connection/test_clock.py
View File

@@ -1,10 +0,0 @@
from ingestor.connection.clock import MonotonicClock
def test_elapsed_is_non_negative_and_monotonic():
clk = MonotonicClock()
a = clk.elapsed()
b = clk.elapsed()
# never negative even if wall clock jumps back
assert a >= 0
assert b >= a

403
tests/unit/connection/test_pool.py
View File

@@ -1,403 +0,0 @@
import asyncio
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
import ingestor.metrics as metrics
from ingestor.connection.pool import OpcConnectionPool
SECURITY = {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}
CERTIFICATE = {
'certificate_id': 7,
'application_uri': 'urn:sientia:opc-ingestor',
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/p/cert.pem'},
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/p/key.pem'},
}
SECURED = {'url': 'opc.tcp://h:4840', 'security': SECURITY, 'certificate': CERTIFICATE}
def make_pool(cert_provider=None):
pool = OpcConnectionPool(
cert_provider=cert_provider or AsyncMock(),
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
)
pool.emit_metric = AsyncMock()
return pool
@pytest.mark.asyncio
async def test_get_reuses_connection_per_server():
pool = make_pool()
fake_client = AsyncMock()
with patch('ingestor.connection.pool.Client', return_value=fake_client):
c1 = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
c2 = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
assert c1 is c2
assert fake_client.connect.await_count == 1
@pytest.mark.asyncio
async def test_get_failure_records_metric_and_raises():
pool = make_pool()
fake_client = AsyncMock()
fake_client.connect.side_effect = RuntimeError('boom')
with patch('ingestor.connection.pool.Client', return_value=fake_client):
with pytest.raises(RuntimeError):
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
pool.emit_metric.assert_any_await(
metric_object=metrics.OPC_CONNECTIONS_FAILED,
tags={'pod_id': pool.pod_id, 'server_name': 'srv'},
)
assert 'srv' not in pool._connections
# a later successful get must reconnect (failed attempt was not cached)
fake_client2 = AsyncMock()
with patch('ingestor.connection.pool.Client', return_value=fake_client2):
client = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
assert client is fake_client2
assert fake_client2.connect.await_count == 1
@pytest.mark.asyncio
async def test_get_applies_the_certificate_and_security_of_the_server():
cert_provider = AsyncMock()
cert_provider.fetch.return_value = ('cert.pem', 'key.pem')
pool = make_pool(cert_provider=cert_provider)
fake_client = AsyncMock()
with (
patch('ingestor.connection.pool.Client', return_value=fake_client),
patch('ingestor.connection.pool.apply_security', new=AsyncMock()) as mock_apply,
):
await pool.get('srv', SECURED)
cert_provider.fetch.assert_awaited_once_with(CERTIFICATE)
mock_apply.assert_awaited_once_with(fake_client, SECURITY, CERTIFICATE, 'cert.pem', 'key.pem')
@pytest.mark.asyncio
@pytest.mark.parametrize(
'cfg',
[
pytest.param({**SECURED, 'certificate': None}, id='no-certificate'),
pytest.param({**SECURED, 'security': {'mode': 'None', 'policy': 'None'}}, id='mode-none'),
],
)
async def test_get_without_certificate_connects_without_security(cfg):
"""No client certificate (or mode None): no security, no MinIO download (D5)"""
cert_provider = AsyncMock()
pool = make_pool(cert_provider=cert_provider)
with (
patch('ingestor.connection.pool.Client', return_value=AsyncMock()),
patch('ingestor.connection.pool.apply_security', new=AsyncMock()) as mock_apply,
):
await pool.get('srv', cfg)
cert_provider.fetch.assert_not_awaited()
mock_apply.assert_not_awaited()
@pytest.mark.asyncio
async def test_release_idle_evicts_and_disconnects():
pool = make_pool()
fake_client = AsyncMock()
with patch('ingestor.connection.pool.Client', return_value=fake_client):
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
client, _ = pool._connections['srv']
pool._connections['srv'] = (client, -1000.0) # force idle
await pool.release_idle(max_idle_s=0)
fake_client.disconnect.assert_awaited_once()
assert 'srv' not in pool._connections
pool.emit_metric.assert_any_await(
metric_object=metrics.OPC_DISCONNECTIONS_TOTAL,
tags={'pod_id': pool.pod_id, 'server_name': 'srv'},
)
@pytest.mark.asyncio
async def test_release_idle_keeps_recently_used():
pool = make_pool()
fake_client = AsyncMock()
with patch('ingestor.connection.pool.Client', return_value=fake_client):
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
await pool.release_idle(max_idle_s=3600)
fake_client.disconnect.assert_not_awaited()
assert 'srv' in pool._connections
@pytest.mark.asyncio
async def test_close_all_disconnects_everything():
pool = make_pool()
fake_a, fake_b = AsyncMock(), AsyncMock()
with patch('ingestor.connection.pool.Client', side_effect=[fake_a, fake_b]):
await pool.get('a', {'url': 'opc.tcp://h:4840', 'certificate': None})
await pool.get('b', {'url': 'opc.tcp://h:4840', 'certificate': None})
await pool.close_all()
fake_a.disconnect.assert_awaited_once()
fake_b.disconnect.assert_awaited_once()
assert pool._connections == {}
@pytest.mark.asyncio
async def test_release_idle_disconnect_retries_then_gives_up():
pool = make_pool()
fake_client = AsyncMock()
fake_client.disconnect.side_effect = RuntimeError('nope')
with patch('ingestor.connection.pool.Client', return_value=fake_client):
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
client, _ = pool._connections['srv']
pool._connections['srv'] = (client, -1000.0)
await pool.release_idle(max_idle_s=0)
assert fake_client.disconnect.await_count == 5
assert 'srv' not in pool._connections
@pytest.mark.asyncio
async def test_success_path_emits_connection_metrics():
pool = make_pool()
fake_client = AsyncMock()
with patch('ingestor.connection.pool.Client', return_value=fake_client):
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
pool.emit_metric.assert_any_await(
metric_object=metrics.OPC_CONNECTIONS_TOTAL,
tags={'pod_id': pool.pod_id, 'server_name': 'srv'},
)
pool.emit_metric.assert_any_await(
metric_object=metrics.OPC_CONNECTION_STATUS,
method='set',
value=1,
tags={'pod_id': pool.pod_id, 'server_name': 'srv', 'server_url': 'opc.tcp://h:4840'},
)
@pytest.mark.asyncio
async def test_concurrent_get_same_server_connects_once():
pool = make_pool()
started = asyncio.Event()
proceed = asyncio.Event()
clients = [AsyncMock(), AsyncMock()]
async def slow_connect():
started.set()
await proceed.wait()
clients[0].connect.side_effect = slow_connect
cfg = {'url': 'opc.tcp://h:4840', 'certificate': None}
with patch('ingestor.connection.pool.Client', side_effect=clients) as ctor:
t1 = asyncio.create_task(pool.get('srv', cfg))
await started.wait() # t1 is mid-connect, holding the per-server lock
t2 = asyncio.create_task(pool.get('srv', cfg))
await asyncio.sleep(0) # let t2 block on the lock (not build a 2nd Client)
proceed.set()
c1, c2 = await asyncio.gather(t1, t2)
assert c1 is c2 is clients[0]
assert ctor.call_count == 1
assert clients[0].connect.await_count == 1
@pytest.mark.asyncio
async def test_release_idle_keeps_pinned_connection():
"""A subscribed connection only gets callbacks (never get()), so it looks idle:
the pin keeps release_idle from dropping it (SIENTIAPDE-1974 G4)."""
pool = make_pool()
fake_client = AsyncMock()
with patch('ingestor.connection.pool.Client', return_value=fake_client):
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
pool.pin('srv')
await pool.release_idle(max_idle_s=0)
assert 'srv' in pool._connections
fake_client.disconnect.assert_not_awaited()
@pytest.mark.asyncio
async def test_release_idle_evicts_after_last_unpin():
pool = make_pool()
fake_client = AsyncMock()
with patch('ingestor.connection.pool.Client', return_value=fake_client):
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
pool.pin('srv')
pool.pin('srv') # two slots subscribed on the same server
pool.unpin('srv')
await pool.release_idle(max_idle_s=0)
assert 'srv' in pool._connections # one subscription still active
pool.unpin('srv')
await pool.release_idle(max_idle_s=0)
assert 'srv' not in pool._connections
fake_client.disconnect.assert_awaited_once()
def test_unpin_never_goes_negative():
pool = make_pool()
pool.unpin('srv')
pool.pin('srv')
assert pool._pins == {'srv': 1}
@pytest.mark.asyncio
async def test_evict_disconnects_now_and_next_get_reconnects():
pool = make_pool()
first, second = AsyncMock(), AsyncMock()
with patch('ingestor.connection.pool.Client', side_effect=[first, second]):
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
await pool.evict('srv')
client = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
first.disconnect.assert_awaited_once()
assert client is second
@pytest.mark.asyncio
async def test_evict_unknown_server_is_noop():
pool = make_pool()
await pool.evict('nope')
pool.emit_metric.assert_not_awaited()
@pytest.mark.asyncio
async def test_get_normalizes_server_id_to_str():
pool = make_pool()
fake_client = AsyncMock()
with patch('ingestor.connection.pool.Client', return_value=fake_client):
c1 = await pool.get(1, {'url': 'opc.tcp://h:4840', 'certificate': None})
c2 = await pool.get('1', {'url': 'opc.tcp://h:4840', 'certificate': None})
assert c1 is c2
assert fake_client.connect.await_count == 1
@pytest.mark.asyncio
async def test_get_with_different_cfg_reuses_client_and_warns():
pool = make_pool()
fake_client = AsyncMock()
with patch('ingestor.connection.pool.Client', return_value=fake_client):
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
client = await pool.get('srv', {'url': 'opc.tcp://10.0.0.5:4840', 'certificate': None})
assert client is fake_client
assert fake_client.connect.await_count == 1
pool.logger.warning.assert_called_once()
assert 'url' in pool.logger.warning.call_args.args[0]
@pytest.mark.asyncio
async def test_get_with_another_certificate_warns():
cert_provider = AsyncMock()
cert_provider.fetch.return_value = ('cert.pem', 'key.pem')
pool = make_pool(cert_provider=cert_provider)
with (
patch('ingestor.connection.pool.Client', return_value=AsyncMock()),
patch('ingestor.connection.pool.apply_security', new=AsyncMock()),
):
await pool.get('srv', SECURED)
await pool.get('srv', {**SECURED, 'certificate': {**CERTIFICATE, 'certificate_id': 8}})
assert 'certificate' in pool.logger.warning.call_args.args[0]
@pytest.mark.asyncio
async def test_secured_connect_failure_invalidates_cert_cache():
from unittest.mock import MagicMock
cert_provider = MagicMock()
cert_provider.fetch = AsyncMock(return_value=('cert.pem', 'key.pem'))
pool = make_pool(cert_provider=cert_provider)
fake_client = AsyncMock()
fake_client.connect.side_effect = RuntimeError('BadCertificateUntrusted')
with (
patch('ingestor.connection.pool.Client', return_value=fake_client),
patch('ingestor.connection.pool.apply_security', AsyncMock()),
):
with pytest.raises(RuntimeError):
await pool.get('srv', SECURED)
cert_provider.invalidate.assert_called_once_with(7)
@pytest.mark.asyncio
async def test_insecure_connect_failure_keeps_cert_cache():
from unittest.mock import MagicMock
cert_provider = MagicMock()
pool = make_pool(cert_provider=cert_provider)
fake_client = AsyncMock()
fake_client.connect.side_effect = RuntimeError('unreachable')
with patch('ingestor.connection.pool.Client', return_value=fake_client):
with pytest.raises(RuntimeError):
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
cert_provider.invalidate.assert_not_called()
@pytest.mark.asyncio
async def test_connect_identifies_the_pod_on_the_server():
"""Session/application name = pod id, as the legacy OpcManager did, so each pod's
session is recognizable in the KEPServer console."""
pool = make_pool()
fake_client = MagicMock(connect=AsyncMock())
with patch('ingestor.connection.pool.Client', return_value=fake_client):
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
assert fake_client.name == pool.pod_id
assert fake_client.description == pool.pod_id
assert fake_client.product_uri == pool.pod_id.replace('-', ':')
assert fake_client.application_uri == pool.pod_id.replace('-', ':')
@pytest.mark.asyncio
async def test_connect_with_certificate_uses_its_application_uri():
"""apply_security sets the certificate's URI: it must match its SAN URI
(BadCertificateUriInvalid otherwise)"""
cert_provider = AsyncMock()
cert_provider.fetch.return_value = ('cert.pem', 'key.pem')
pool = make_pool(cert_provider=cert_provider)
fake_client = MagicMock(connect=AsyncMock(), set_security=AsyncMock())
with patch('ingestor.connection.pool.Client', return_value=fake_client):
await pool.get('srv', SECURED)
assert fake_client.application_uri == 'urn:sientia:opc-ingestor'
assert fake_client.name == pool.pod_id
@pytest.mark.asyncio
async def test_config_for_returns_a_copy_of_the_live_connection_settings():
pool = make_pool()
cfg = {'url': 'opc.tcp://h:4840', 'security': SECURITY, 'certificate': None}
with patch('ingestor.connection.pool.Client', return_value=AsyncMock()):
await pool.get(1, cfg)
found = pool.config_for('1')
assert found == cfg
found['url'] = 'mutated'
assert pool.config_for(1)['url'] == 'opc.tcp://h:4840'
await pool.evict('1')
assert pool.config_for('1') is None
def test_config_for_unknown_server_is_none():
assert make_pool().config_for('missing') is None

92
tests/unit/connection/test_security.py
View File

@@ -1,92 +0,0 @@
from unittest.mock import AsyncMock, MagicMock
import pytest
from asyncua import ua
from asyncua.crypto.security_policies import (
SecurityPolicyAes128Sha256RsaOaep,
SecurityPolicyBasic256Sha256,
)
from ingestor.connection.security import (
SESSION_TIMEOUT_MS,
UnsupportedSecurityError,
apply_security,
is_secured,
)
CERTIFICATE = {'certificate_id': 1, 'application_uri': 'urn:sientia:opc-ingestor'}
def _client():
client = MagicMock()
client.set_security = AsyncMock()
return client
@pytest.mark.asyncio
async def test_apply_security_uses_mode_and_policy_and_channel_lifetime():
client = _client()
await apply_security(
client, {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}, CERTIFICATE, 'c', 'k'
)
assert client.application_uri == 'urn:sientia:opc-ingestor'
assert client.session_timeout == SESSION_TIMEOUT_MS
assert client.secure_channel_timeout == int(SESSION_TIMEOUT_MS * 1.25)
args, kwargs = client.set_security.call_args
assert args[0] is SecurityPolicyBasic256Sha256
assert kwargs == {
'certificate': 'c',
'private_key': 'k',
'server_certificate': None,
'mode': ua.MessageSecurityMode.SignAndEncrypt,
}
@pytest.mark.asyncio
async def test_apply_security_sign_with_another_policy():
client = _client()
await apply_security(
client, {'mode': 'Sign', 'policy': 'Aes128Sha256RsaOaep'}, CERTIFICATE, 'c', 'k'
)
args, kwargs = client.set_security.call_args
assert args[0] is SecurityPolicyAes128Sha256RsaOaep
assert kwargs['mode'] == ua.MessageSecurityMode.Sign
@pytest.mark.asyncio
@pytest.mark.parametrize(
'security',
[
{'mode': 'Encrypt', 'policy': 'Basic256Sha256'},
{'mode': 'Sign', 'policy': 'Basic999'},
{'mode': 'Sign', 'policy': 'None'},
{},
],
)
async def test_apply_security_rejects_unknown_mode_or_policy(security):
client = _client()
with pytest.raises(UnsupportedSecurityError):
await apply_security(client, security, CERTIFICATE, 'c', 'k')
client.set_security.assert_not_awaited()
@pytest.mark.parametrize(
('server_cfg', 'secured'),
[
({'security': {'mode': 'SignAndEncrypt'}, 'certificate': CERTIFICATE}, True),
({'security': {'mode': 'Sign'}, 'certificate': CERTIFICATE}, True),
# No certificate: no security (D5), whatever the mode
({'security': {'mode': 'SignAndEncrypt'}, 'certificate': None}, False),
({'security': {'mode': 'SignAndEncrypt'}}, False),
# mode None: no security even with a certificate
({'security': {'mode': 'None'}, 'certificate': CERTIFICATE}, False),
({}, False),
],
)
def test_is_secured(server_cfg, secured):
assert is_secured(server_cfg) is secured

149
tests/unit/connection/test_servers.py
View File

@@ -1,149 +0,0 @@
from unittest.mock import AsyncMock, MagicMock
import pytest
from ingestor.connection.servers import CONNECTORS, ServerConfigResolver
LOOKED_UP = {
'certificate_id': 1,
'application_uri': 'urn:sientia:opc-ingestor',
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/p/cert.pem'},
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/p/key.pem'},
}
# A connector as the aggregation returns it ($lookup on opc_certificates)
CONNECTOR = {
'connector_id': 'kepserver',
'url': 'opc.tcp://kepserver:49320',
'subscription_period_ms': 500,
'security': {
'mode': 'SignAndEncrypt',
'policy': 'Basic256Sha256',
'client_certificate_id': 1,
},
'certificates': [LOOKED_UP],
}
# The same server in the slot v2 format
CONFIG = {
'server_id': 'kepserver',
'url': 'opc.tcp://kepserver:49320',
'subscription_period_ms': 500,
'security': {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'},
'certificate': LOOKED_UP,
}
def make_resolver(connector=None, pooled=None, ttl_s=60.0):
pool = MagicMock()
pool.config_for = MagicMock(return_value=pooled)
mongo = MagicMock()
mongo.aggregate = AsyncMock(return_value=[connector] if connector is not None else [])
return ServerConfigResolver(pool, mongo, MagicMock(), ttl_s=ttl_s)
@pytest.mark.asyncio
async def test_live_pooled_connection_wins_over_mongo():
"""Writes reuse exactly the settings the read path opened the connection with."""
pooled = {'url': 'opc.tcp://read:1', 'security': CONFIG['security'], 'certificate': None}
resolver = make_resolver(connector=CONNECTOR, pooled=pooled)
assert await resolver.resolve('kepserver') == pooled
resolver.pool.config_for.assert_called_once_with('kepserver')
resolver.mongo.aggregate.assert_not_awaited()
@pytest.mark.asyncio
async def test_connector_with_its_certificate_in_the_slot_format():
resolver = make_resolver(connector=CONNECTOR)
assert await resolver.resolve('kepserver') == CONFIG
collection, pipeline, _ = resolver.mongo.aggregate.await_args.args
assert collection == CONNECTORS
assert pipeline[0] == {
'$match': {'connector_id': 'kepserver', 'type': 'opc_ua', 'status': 'Active'}
}
# The same rule as the orchestrator: only an active client certificate (D5)
[lookup] = [stage['$lookup'] for stage in pipeline if '$lookup' in stage]
assert lookup['from'] == 'opc_certificates'
assert lookup['pipeline'][0]['$match']['status'] == 'active'
assert lookup['pipeline'][0]['$match']['kind'] == 'client'
resolver.logger.warning.assert_not_called()
@pytest.mark.asyncio
async def test_connector_without_certificate_id_connects_without_security():
connector = {
**CONNECTOR,
'security': {'mode': 'None', 'policy': 'None', 'client_certificate_id': None},
'certificates': [],
}
resolver = make_resolver(connector=connector)
cfg = await resolver.resolve('kepserver')
assert cfg['certificate'] is None
assert cfg['security'] == {'mode': 'None', 'policy': 'None'}
resolver.logger.warning.assert_not_called()
@pytest.mark.asyncio
async def test_certificate_id_without_an_active_client_certificate_warns():
resolver = make_resolver(connector={**CONNECTOR, 'certificates': []})
cfg = await resolver.resolve('kepserver')
assert cfg == {**CONFIG, 'certificate': None}
assert 'client_certificate_id 1' in resolver.logger.warning.call_args.args[0]
@pytest.mark.asyncio
@pytest.mark.parametrize(
'connector',
[
pytest.param({**CONNECTOR, 'url': ''}, id='no-url'),
pytest.param({**CONNECTOR, 'security': {'mode': 'Sign'}}, id='no-policy'),
pytest.param({k: v for k, v in CONNECTOR.items() if k != 'security'}, id='no-security'),
],
)
async def test_incomplete_connectors_are_not_found(connector):
resolver = make_resolver(connector=connector)
assert await resolver.resolve('kepserver') is None
resolver.logger.warning.assert_called_once()
@pytest.mark.asyncio
async def test_unknown_server_is_cached_briefly():
resolver = make_resolver(ttl_s=60.0)
resolver.missing_ttl_s = 10.0
resolver.clock = MagicMock()
resolver.clock.elapsed.side_effect = [0.0, 5.0, 11.0]
assert await resolver.resolve('9') is None
assert await resolver.resolve('9') is None # within missing_ttl_s: no lookup
assert await resolver.resolve('9') is None # expired: looked up again
assert resolver.mongo.aggregate.await_count == 2
assert resolver.logger.warning.call_count == 2
@pytest.mark.asyncio
async def test_lookups_are_cached_for_the_ttl():
resolver = make_resolver(connector=CONNECTOR, ttl_s=60.0)
resolver.clock = MagicMock()
resolver.clock.elapsed.side_effect = [0.0, 30.0, 61.0]
first = await resolver.resolve('kepserver')
first['url'] = 'mutated by the caller'
second = await resolver.resolve('kepserver') # within the TTL: cached, and a copy
await resolver.resolve('kepserver') # expired: looked up again
assert second['url'] == 'opc.tcp://kepserver:49320'
assert resolver.mongo.aggregate.await_count == 2
@pytest.mark.asyncio
async def test_mongo_errors_propagate_to_the_caller():
resolver = make_resolver()
resolver.mongo.aggregate = AsyncMock(side_effect=RuntimeError('mongo down'))
with pytest.raises(RuntimeError, match='mongo down'):
await resolver.resolve('kepserver')

0
tests/unit/engines/__init__.py
View File

179
tests/unit/engines/test_active_read.py
View File

@@ -1,179 +0,0 @@
from datetime import datetime
from unittest.mock import AsyncMock, MagicMock
import pytest
from asyncua import ua
import ingestor.metrics as metrics
from ingestor.engines.active_read import ActiveReadEngine
def make_engine():
eng = ActiveReadEngine(
pool=AsyncMock(),
sink=AsyncMock(),
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
)
eng.emit_metric = AsyncMock()
return eng
def make_dv(value=42.0, status='Good', source_ts=None, server_ts=None):
source_ts = source_ts or datetime(2026, 7, 13, 12, 0, 0)
server_ts = server_ts or source_ts
return ua.DataValue(
ua.Variant(value),
StatusCode=ua.StatusCode(getattr(ua.StatusCodes, status)),
SourceTimestamp=source_ts,
ServerTimestamp=server_ts,
)
def make_client(dv=None):
node = MagicMock()
node.read_data_value = AsyncMock(return_value=dv or make_dv())
client = MagicMock()
client.get_node = MagicMock(return_value=node)
return client, node
NAMES = [{'tag_name': 'a', 'model_id': 'm1', 'collection': 'coll'}]
@pytest.mark.asyncio
async def test_read_now_persists_and_returns_fields_when_names_given():
eng = make_engine()
client, node = make_client()
eng.pool.get.return_value = client
result = await eng.read_now('srv1', 'ns=2;i=5', names=NAMES)
client.get_node.assert_called_once_with('ns=2;i=5')
node.read_data_value.assert_awaited_once()
eng.sink.publish_reading.assert_awaited_once()
_, kwargs = eng.sink.publish_reading.call_args
assert kwargs['tag'] == 'ns=2;i=5'
assert kwargs['value'] == 42.0
assert kwargs['names'] == NAMES
assert result['value'] == 42.0
assert result['opc_status_code'] == 'Good'
assert result['persisted'] is True
assert result['source_timestamp'].startswith('2026-07-13 12:00:00')
assert result['server_timestamp'].startswith('2026-07-13 12:00:00')
eng.emit_metric.assert_any_await(
metric_object=metrics.OPC_ACTIVE_READ_TOTAL,
tags={'pod_id': eng.pod_id, 'server_name': 'srv1', 'result': 'success'},
)
latency_call = next(
c
for c in eng.emit_metric.await_args_list
if c.kwargs['metric_object'] is metrics.OPC_ACTIVE_READ_LATENCY
)
assert latency_call.kwargs['method'] == 'observe'
assert latency_call.kwargs['value'] >= 0
lag_call = next(
c
for c in eng.emit_metric.await_args_list
if c.kwargs['metric_object'] is metrics.OPC_READ_LAG
)
assert lag_call.kwargs['method'] == 'set'
@pytest.mark.asyncio
async def test_read_now_without_names_does_not_persist():
eng = make_engine()
client, _ = make_client()
eng.pool.get.return_value = client
result = await eng.read_now('srv1', 'ns=2;i=5')
eng.sink.publish_reading.assert_not_awaited()
assert result['persisted'] is False
@pytest.mark.asyncio
async def test_read_now_error_emits_metric_and_reraises():
eng = make_engine()
node = MagicMock()
node.read_data_value = AsyncMock(side_effect=RuntimeError('boom'))
client = MagicMock()
client.get_node = MagicMock(return_value=node)
eng.pool.get.return_value = client
with pytest.raises(RuntimeError, match='boom'):
await eng.read_now('srv1', 'ns=2;i=5')
eng.emit_metric.assert_any_await(
metric_object=metrics.OPC_ACTIVE_READ_TOTAL,
tags={'pod_id': eng.pod_id, 'server_name': 'srv1', 'result': 'error'},
)
eng.sink.publish_reading.assert_not_awaited()
def make_slot(cycle_multiple=None, enabled=True):
active_read = {'enabled': enabled}
if cycle_multiple is not None:
active_read['cycle_multiple'] = cycle_multiple
return {
'srv_name': {
'server_id': 'srv1',
'url': 'opc.tcp://x',
'tags': {
'ns=2;i=5': {'frequency': 1000, 'names': NAMES, 'active_read': active_read},
},
}
}
@pytest.mark.asyncio
async def test_scheduled_tick_respects_cycle_multiple():
eng = make_engine()
eng.read_now = AsyncMock()
slot = make_slot(cycle_multiple=3)
await eng.scheduled_tick(slot)
await eng.scheduled_tick(slot)
eng.read_now.assert_not_awaited()
await eng.scheduled_tick(slot)
eng.read_now.assert_awaited_once_with(
'srv1', 'ns=2;i=5', names=NAMES, server_cfg={'server_id': 'srv1', 'url': 'opc.tcp://x'}
)
@pytest.mark.asyncio
async def test_scheduled_tick_skips_disabled_tags():
eng = make_engine()
eng.read_now = AsyncMock()
slot = make_slot(enabled=False)
await eng.scheduled_tick(slot)
eng.read_now.assert_not_awaited()
@pytest.mark.asyncio
async def test_scheduled_tick_continues_past_tag_error():
eng = make_engine()
bad_slot = make_slot(cycle_multiple=1)
good_names = [{'tag_name': 'b', 'model_id': 'm2', 'collection': 'coll'}]
bad_slot['srv_name']['tags']['ns=2;i=6'] = {
'frequency': 1000,
'names': good_names,
'active_read': {'enabled': True, 'cycle_multiple': 1},
}
calls = []
async def fake_read_now(server_id, addr, names=None, server_cfg=None):
calls.append(addr)
if addr == 'ns=2;i=5':
raise RuntimeError('boom')
eng.read_now = AsyncMock(side_effect=fake_read_now)
await eng.scheduled_tick(bad_slot)
assert set(calls) == {'ns=2;i=5', 'ns=2;i=6'}
eng.logger.error.assert_called_once()

300
tests/unit/engines/test_subscription.py
View File

@@ -1,300 +0,0 @@
from datetime import datetime
from unittest.mock import AsyncMock, MagicMock
import pytest
from sientia_do.temporal.constants import OPC_TIMEZONE
import ingestor.metrics as metrics
from ingestor.engines.subscription import SubscriptionEngine
def make_engine():
eng = SubscriptionEngine(
pool=AsyncMock(),
sink=AsyncMock(),
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
metadata={},
)
eng.emit_metric = AsyncMock()
eng.send_notification_async = AsyncMock()
return eng
def make_node(name='ns=2;i=2'):
node = MagicMock()
node.__str__ = lambda self: name
return node
def make_data(value=42, timestamp=None):
data = MagicMock()
data.monitored_item.Value.Value.Value = value
if timestamp is not None:
data.monitored_item.Value.SourceTimestamp = timestamp
return data
@pytest.mark.asyncio
async def test_datachange_fans_out():
eng = make_engine()
eng.nodes['ns=2;i=2'] = {
'names': [{'tag_name': 'a'}, {'tag_name': 'b'}],
'cycle_rule': {'cycle_increment': 1, 'cycle_count': 3},
}
node = make_node()
data = make_data(42)
await eng.datachange_notification(node, 42, data)
assert eng.sink.publish_reading.await_count == 1
_, kwargs = eng.sink.publish_reading.call_args
assert len(kwargs['names']) == 2
assert kwargs['tag'] == 'ns=2;i=2'
assert kwargs['value'] == 42
@pytest.mark.asyncio
async def test_datachange_resets_cycle_count_and_non_receive_count():
eng = make_engine()
eng.non_receive_count = 7
eng.nodes['ns=2;i=2'] = {
'names': [{'tag_name': 'a'}],
'cycle_rule': {'cycle_increment': 1, 'cycle_count': 3},
}
await eng.datachange_notification(make_node(), 42, make_data(42))
assert eng.nodes['ns=2;i=2']['cycle_rule']['cycle_count'] == 0
assert eng.non_receive_count == 0
eng.emit_metric.assert_any_await(
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
method='set',
value=0,
tags={'pod_id': eng.pod_id, 'server_name': 'unknown'},
)
@pytest.mark.asyncio
async def test_datachange_emits_read_lag():
eng = make_engine()
eng.nodes['ns=2;i=2'] = {
'names': [{'tag_name': 'a'}],
'cycle_rule': {'cycle_increment': 1, 'cycle_count': 0},
'server_name': 'srv1',
}
source_ts = datetime.now(tz=OPC_TIMEZONE)
await eng.datachange_notification(make_node(), 42, make_data(42, timestamp=source_ts))
lag_call = next(
c
for c in eng.emit_metric.await_args_list
if c.kwargs['metric_object'] is metrics.OPC_READ_LAG
)
assert lag_call.kwargs['tags'] == {'pod_id': eng.pod_id, 'server_name': 'srv1'}
assert abs(lag_call.kwargs['value']) < 5
@pytest.mark.asyncio
async def test_create_subscription_uses_pool_client():
eng = make_engine()
client = AsyncMock()
eng.pool.get.return_value = client
await eng.create_subscription('srv1', {'subscription_period_ms': 500}, 'sub1')
eng.pool.get.assert_awaited_once_with('srv1', {'subscription_period_ms': 500})
client.create_subscription.assert_awaited_once_with(500, eng)
assert eng.subscriptions['sub1'] is not None
eng.emit_metric.assert_any_await(
metric_object=metrics.OPC_SUBSCRIPTIONS_CREATED,
method='inc',
value=1,
tags={'pod_id': eng.pod_id, 'server_name': 'srv1', 'slot_name': 'sub1'},
)
@pytest.mark.asyncio
async def test_subscribe_registers_nodes_with_cycle_rule():
eng = make_engine()
client = AsyncMock()
eng.pool.get.return_value = client
await eng.create_subscription('srv1', {'subscription_period_ms': 500}, 'sub1')
nodes = {
'ns=2;i=2': {
'frequency': 1000,
'names': [{'tag_name': 'a'}, {'tag_name': 'b'}],
'active_read': {'enabled': False},
}
}
await eng.subscribe('sub1', nodes, collect_period=1)
assert eng.nodes['ns=2;i=2']['server_name'] == 'srv1'
assert eng.nodes['ns=2;i=2']['cycle_rule']['cycle_count'] == 0
assert eng.nodes['ns=2;i=2']['cycle_rule']['cycle_increment'] == pytest.approx(1.0)
eng.subscriptions['sub1'].subscribe_data_change.assert_awaited_once()
@pytest.mark.asyncio
async def test_subscribe_without_subscription_raises():
eng = make_engine()
with pytest.raises(ValueError, match='Subscription not created'):
await eng.subscribe('sub1', {}, collect_period=1)
@pytest.mark.asyncio
async def test_unsubscribe_missing_logs_warning():
eng = make_engine()
await eng.unsubscribe('sub1')
eng.logger.warning.assert_called_once_with("Subscription 'sub1' not found. Cannot unsubscribe.")
@pytest.mark.asyncio
async def test_unsubscribe_deletes_subscription():
eng = make_engine()
client = AsyncMock()
eng.pool.get.return_value = client
await eng.create_subscription('srv1', {'subscription_period_ms': 500}, 'sub1')
sub = eng.subscriptions['sub1']
await eng.unsubscribe('sub1')
sub.delete.assert_awaited_once()
assert 'sub1' not in eng.subscriptions
@pytest.mark.asyncio
async def test_check_cycles_triggers_notification_over_threshold():
eng = make_engine()
eng.nodes['ns=2;i=2'] = {
'names': [{'tag_name': 'a'}],
'cycle_rule': {'cycle_increment': 5.5, 'cycle_count': 0},
}
await eng.check_cycles()
eng.send_notification_async.assert_called_once()
@pytest.mark.asyncio
async def test_check_opc_listenning_below_threshold():
eng = make_engine()
eng.non_receive_count = 2
result = await eng.check_opc_listenning('srv1')
assert eng.non_receive_count == 3
assert result is False
eng.send_notification_async.assert_not_called()
@pytest.mark.asyncio
async def test_check_opc_listenning_triggers_reconnect_at_15():
eng = make_engine()
eng.non_receive_count = 14
result = await eng.check_opc_listenning('srv1')
assert result is True
assert eng.send_notification_async.call_count == 2
@pytest.mark.asyncio
async def test_create_subscription_pins_pool_connection():
eng = make_engine()
eng.pool = MagicMock(get=AsyncMock(return_value=AsyncMock()))
await eng.create_subscription('1', {'subscription_period_ms': 500}, 'slot1')
eng.pool.pin.assert_called_once_with('1')
@pytest.mark.asyncio
async def test_create_subscription_failure_does_not_pin():
eng = make_engine()
client = AsyncMock()
client.create_subscription.side_effect = RuntimeError('boom')
eng.pool = MagicMock(get=AsyncMock(return_value=client))
with pytest.raises(RuntimeError):
await eng.create_subscription('1', {'subscription_period_ms': 500}, 'slot1')
eng.pool.pin.assert_not_called()
@pytest.mark.asyncio
async def test_unsubscribe_unpins_pool_connection():
eng = make_engine()
eng.pool = MagicMock(get=AsyncMock(return_value=AsyncMock()))
await eng.create_subscription('1', {'subscription_period_ms': 500}, 'slot1')
await eng.unsubscribe('slot1')
eng.pool.unpin.assert_called_once_with('1')
assert 'slot1' not in eng.subscriptions
@pytest.mark.asyncio
async def test_unsubscribe_tolerates_dead_connection():
"""A lost server's subscription can't be deleted on the server side; it must
still be dropped locally (and unpinned) so the slot can be re-subscribed."""
eng = make_engine()
sub = AsyncMock()
sub.delete.side_effect = ConnectionError('closed')
client = AsyncMock()
client.create_subscription.return_value = sub
eng.pool = MagicMock(get=AsyncMock(return_value=client))
await eng.create_subscription('1', {'subscription_period_ms': 500}, 'slot1')
await eng.unsubscribe('slot1') # must not raise
assert 'slot1' not in eng.subscriptions
eng.pool.unpin.assert_called_once_with('1')
eng.logger.warning.assert_called_once()
async def _subscribed_engine(slots):
"""Engine with one subscription per slot: {slot: [addresses]}."""
eng = make_engine()
eng.pool = MagicMock(
get=AsyncMock(
return_value=MagicMock(create_subscription=AsyncMock(return_value=AsyncMock()))
)
)
for slot, addresses in slots.items():
await eng.create_subscription('1', {'subscription_period_ms': 500}, slot)
nodes = {
a: {'frequency': '1000', 'names': [{'collection': 'c', 'tag_name': a}]}
for a in addresses
}
await eng.subscribe(slot, nodes, 10)
return eng
@pytest.mark.asyncio
async def test_unsubscribe_forgets_the_slot_nodes():
eng = await _subscribed_engine({'s1': ['ns=2;i=1', 'ns=2;i=2'], 's2': ['ns=2;i=3']})
await eng.unsubscribe('s1')
# no more cycle checks / tag counting for addresses nobody reads anymore
assert set(eng.nodes) == {'ns=2;i=3'}
eng.emit_metric.assert_any_await(
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
method='set',
value=1,
tags={'pod_id': eng.pod_id, 'server_name': '1'},
)
@pytest.mark.asyncio
async def test_unsubscribe_keeps_node_shared_with_another_slot():
eng = await _subscribed_engine({'s1': ['ns=2;i=1', 'ns=2;i=2'], 's2': ['ns=2;i=2']})
await eng.unsubscribe('s1')
assert set(eng.nodes) == {'ns=2;i=2'}

406
tests/unit/engines/test_write.py
View File

@@ -1,406 +0,0 @@
from datetime import datetime
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from asyncua import ua
import ingestor.metrics as metrics
from ingestor.engines.write import WriteEngine
SERVER_CFG = {
'server_id': 'srv1',
'url': 'opc.tcp://kepserver:49320',
'security': {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'},
'certificate': None,
}
def make_resolver(cfg=SERVER_CFG):
resolver = MagicMock()
resolver.resolve = AsyncMock(return_value=cfg)
return resolver
def make_engine(**kwargs):
kwargs.setdefault('servers', make_resolver())
eng = WriteEngine(
pool=AsyncMock(),
sink=AsyncMock(),
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
**kwargs,
)
eng.emit_metric = AsyncMock()
return eng
def make_client(node=None):
node = node or make_node()
client = MagicMock()
client.get_node = MagicMock(return_value=node)
return client
def make_node():
node = MagicMock()
node.write_value = AsyncMock()
return node
def base_cmd(**overrides):
cmd = {
'server_id': 'srv1',
'addr': 'ns=2;i=5',
'value': 42.0,
'data_type': 'float',
'tag_type': 'prediction',
'model_id': 'm1',
'model_name': 'model-1',
'timestamp': '2026-07-13T12:00:00Z',
'schema': 'sientia_data',
'table_name': 'predictions_m1',
}
cmd.update(overrides)
return cmd
@pytest.mark.asyncio
async def test_write_success_writes_datavalue_and_returns_ok():
eng = make_engine()
node = make_node()
client = make_client(node)
eng.pool.get.return_value = client
result = await eng.write(base_cmd())
client.get_node.assert_called_once_with('ns=2;i=5')
node.write_value.assert_awaited_once()
dv = node.write_value.await_args.args[0]
assert isinstance(dv, ua.DataValue)
assert dv.Value.Value == 42.0
assert dv.Value.VariantType == ua.VariantType.Float
assert result.ok is True
assert result.status == 'ok'
assert result.opc_status_code == 'Good'
eng.sink.update_write_status.assert_awaited_once_with(base_cmd(), result)
@pytest.mark.asyncio
async def test_write_success_emits_metrics():
eng = make_engine()
eng.pool.get.return_value = make_client()
await eng.write(base_cmd())
eng.emit_metric.assert_any_await(
metric_object=metrics.OPC_WRITE_TOTAL,
tags={'pod_id': eng.pod_id, 'server_name': 'srv1', 'model_id': 'm1', 'result': 'success'},
)
status_call = next(
c
for c in eng.emit_metric.await_args_list
if c.kwargs['metric_object'] is metrics.OPC_WRITE_STATUS_BY_MODEL
)
assert status_call.kwargs['value'] == 1
assert status_call.kwargs['tags'] == {'pod_id': eng.pod_id, 'model_id': 'm1', 'status': 'ok'}
latency_call = next(
c
for c in eng.emit_metric.await_args_list
if c.kwargs['metric_object'] is metrics.OPC_WRITE_LATENCY
)
assert latency_call.kwargs['method'] == 'observe'
assert latency_call.kwargs['value'] >= 0
@pytest.mark.asyncio
async def test_write_defaults_carry_statuscode_and_timestamp():
eng = make_engine(write_timestamp=True, write_quality=True)
node = make_node()
eng.pool.get.return_value = make_client(node)
await eng.write(base_cmd())
dv = node.write_value.await_args.args[0]
assert dv.StatusCode is not None
assert dv.StatusCode.is_good()
assert dv.SourceTimestamp == datetime.fromisoformat('2026-07-13T12:00:00Z')
@pytest.mark.asyncio
async def test_write_engine_flags_off_omit_statuscode_and_timestamp():
eng = make_engine(write_timestamp=False, write_quality=False)
node = make_node()
eng.pool.get.return_value = make_client(node)
await eng.write(base_cmd())
dv = node.write_value.await_args.args[0]
assert dv.StatusCode is None
assert dv.SourceTimestamp is None
@pytest.mark.asyncio
async def test_write_engine_timestamp_off_keeps_statuscode():
eng = make_engine(write_timestamp=False)
node = make_node()
eng.pool.get.return_value = make_client(node)
await eng.write(base_cmd())
dv = node.write_value.await_args.args[0]
assert dv.Value.Value == 42.0
assert dv.SourceTimestamp is None
assert dv.StatusCode is not None
assert dv.StatusCode.is_good()
@pytest.mark.asyncio
async def test_write_engine_quality_off_keeps_timestamp():
# The command has no write_quality, so the engine (env) default applies.
eng = make_engine(write_quality=False)
node = make_node()
eng.pool.get.return_value = make_client(node)
await eng.write(base_cmd())
dv = node.write_value.await_args.args[0]
assert dv.Value.Value == 42.0
assert dv.StatusCode is None
assert dv.SourceTimestamp == datetime.fromisoformat('2026-07-13T12:00:00Z')
@pytest.mark.asyncio
async def test_per_message_flag_override_disables_timestamp():
# Engine default is True, but the command explicitly turns it off.
eng = make_engine(write_timestamp=True, write_quality=True)
node = make_node()
eng.pool.get.return_value = make_client(node)
await eng.write(base_cmd(write_timestamp=False))
dv = node.write_value.await_args.args[0]
assert dv.SourceTimestamp is None
assert dv.StatusCode is not None # write_quality untouched, stays on
@pytest.mark.asyncio
async def test_write_bad_type_returns_error_without_raising():
eng = make_engine()
result = await eng.write(base_cmd(value='abc', data_type='int'))
assert result.ok is False
assert result.status == 'error'
eng.sink.update_write_status.assert_awaited_once()
eng.pool.get.assert_not_awaited()
eng.emit_metric.assert_any_await(
metric_object=metrics.OPC_ERROR_BY_TYPE,
tags={'pod_id': eng.pod_id, 'server_name': 'srv1', 'error_type': 'conversion'},
)
eng.emit_metric.assert_any_await(
metric_object=metrics.OPC_WRITE_TOTAL,
tags={'pod_id': eng.pod_id, 'server_name': 'srv1', 'model_id': 'm1', 'result': 'error'},
)
@pytest.mark.asyncio
async def test_write_none_data_type_returns_error_without_raising():
eng = make_engine()
result = await eng.write(base_cmd(data_type=None))
assert result.ok is False
assert result.status == 'error'
eng.sink.update_write_status.assert_awaited_once()
eng.pool.get.assert_not_awaited()
@pytest.mark.asyncio
async def test_write_missing_value_key_returns_error_without_raising():
eng = make_engine()
cmd = base_cmd()
del cmd['value']
result = await eng.write(cmd)
assert result.ok is False
assert result.status == 'error'
eng.sink.update_write_status.assert_awaited_once()
eng.pool.get.assert_not_awaited()
@pytest.mark.asyncio
async def test_write_missing_data_type_key_returns_error_without_raising():
eng = make_engine()
cmd = base_cmd()
del cmd['data_type']
result = await eng.write(cmd)
assert result.ok is False
assert result.status == 'error'
eng.sink.update_write_status.assert_awaited_once()
eng.pool.get.assert_not_awaited()
@pytest.mark.asyncio
async def test_write_value_raises_maps_to_error_result_not_raised():
eng = make_engine()
node = make_node()
node.write_value.side_effect = RuntimeError('boom')
eng.pool.get.return_value = make_client(node)
result = await eng.write(base_cmd())
assert result.ok is False
assert result.status == 'error'
assert result.opc_status_code == 'RuntimeError'
eng.sink.update_write_status.assert_awaited_once()
eng.emit_metric.assert_any_await(
metric_object=metrics.OPC_ERROR_BY_TYPE,
tags={'pod_id': eng.pod_id, 'server_name': 'srv1', 'error_type': 'RuntimeError'},
)
@pytest.mark.asyncio
async def test_write_session_bad_status_code_error_maps_to_session_bad():
from asyncua.ua.uaerrors import BadSessionClosed
eng = make_engine()
node = make_node()
node.write_value.side_effect = BadSessionClosed('closed')
eng.pool.get.return_value = make_client(node)
result = await eng.write(base_cmd())
assert result.ok is False
assert result.status == 'session_bad'
@pytest.mark.asyncio
async def test_rate_limit_sleeps_between_back_to_back_writes():
eng = make_engine(min_pause_s=0.05)
eng.pool.get.return_value = make_client()
with patch('ingestor.engines.write.asyncio.sleep', new_callable=AsyncMock) as sleep_mock:
await eng.write(base_cmd())
sleep_mock.assert_not_awaited()
await eng.write(base_cmd())
sleep_mock.assert_awaited_once()
remaining = sleep_mock.await_args.args[0]
assert 0 < remaining <= 0.05
@pytest.mark.asyncio
async def test_no_rate_limit_when_min_pause_is_zero():
eng = make_engine(min_pause_s=0.0)
eng.pool.get.return_value = make_client()
with patch('ingestor.engines.write.asyncio.sleep', new_callable=AsyncMock) as sleep_mock:
await eng.write(base_cmd())
await eng.write(base_cmd())
sleep_mock.assert_not_awaited()
# --- SIENTIAPDE-1975: server resolution (GW1) and error reporting (GW4) ---
@pytest.mark.asyncio
async def test_write_uses_the_resolved_server_connection():
"""The C2 command of the Laborious carries only the server_id."""
eng = make_engine()
eng.pool.get.return_value = make_client()
await eng.write(base_cmd())
eng.servers.resolve.assert_awaited_once_with('srv1')
eng.pool.get.assert_awaited_once_with('srv1', SERVER_CFG)
@pytest.mark.asyncio
async def test_write_unknown_server_records_error_without_connecting():
eng = make_engine(servers=make_resolver(cfg=None))
result = await eng.write(base_cmd())
assert result.status == 'error'
assert result.opc_status_code == 'server_not_found'
eng.pool.get.assert_not_awaited()
eng.sink.update_write_status.assert_awaited_once_with(base_cmd(), result)
@pytest.mark.asyncio
async def test_write_server_lookup_failure_records_error():
resolver = make_resolver()
resolver.resolve.side_effect = RuntimeError('mongo down')
eng = make_engine(servers=resolver)
result = await eng.write(base_cmd())
assert result.status == 'error'
assert result.opc_status_code == 'server_lookup'
assert 'mongo down' in result.message
eng.pool.get.assert_not_awaited()
@pytest.mark.asyncio
async def test_write_ignores_a_connection_carried_by_the_command():
"""The server is always resolved by server_id (SIENTIAPDE-1974): connection fields
in the command are not an override"""
eng = make_engine()
eng.pool.get.return_value = make_client()
await eng.write(base_cmd(url='opc.tcp://h:4840'))
eng.pool.get.assert_awaited_once_with('srv1', SERVER_CFG)
@pytest.mark.asyncio
async def test_write_without_resolver_is_server_not_found():
eng = make_engine(servers=None)
result = await eng.write(base_cmd(url='opc.tcp://h:4840'))
assert result.opc_status_code == 'server_not_found'
eng.pool.get.assert_not_awaited()
@pytest.mark.asyncio
async def test_write_error_logs_the_reason_and_sets_the_model_status():
eng = make_engine()
node = make_node()
node.write_value.side_effect = RuntimeError('BadWriteNotSupported')
eng.pool.get.return_value = make_client(node)
await eng.write(base_cmd())
logged = eng.logger.error.call_args.args[0]
assert 'RuntimeError' in logged
assert 'BadWriteNotSupported' in logged
assert 'addr=ns=2;i=5' in logged
assert 'model_id=m1' in logged
status_call = next(
c
for c in eng.emit_metric.await_args_list
if c.kwargs['metric_object'] is metrics.OPC_WRITE_STATUS_BY_MODEL
)
# Same series as a success, so it shows the last outcome.
assert status_call.kwargs['value'] == 0
assert status_call.kwargs['tags'] == {'pod_id': eng.pod_id, 'model_id': 'm1', 'status': 'ok'}
@pytest.mark.asyncio
@pytest.mark.parametrize('timestamp', [None, 'not-a-timestamp'])
async def test_write_bad_timestamp_records_error_without_raising(timestamp):
eng = make_engine(write_timestamp=True)
cmd = base_cmd(timestamp=timestamp)
if timestamp is None:
del cmd['timestamp']
result = await eng.write(cmd)
assert result.status == 'error'
assert result.opc_status_code == 'bad_timestamp'
eng.pool.get.assert_not_awaited()
eng.sink.update_write_status.assert_awaited_once()

0
tests/unit/io/__init__.py
View File

69
tests/unit/io/test_data_sink_publish.py
View File

@@ -1,69 +0,0 @@
from unittest.mock import AsyncMock
import pytest
import ingestor.metrics as metrics
from ingestor.io.data_sink import DataSink
def make_sink(data_manager=None):
dm = data_manager or AsyncMock()
sink = DataSink(data_manager=dm, postgres=AsyncMock())
sink.emit_metric = AsyncMock()
return sink, dm
@pytest.mark.asyncio
async def test_publish_reading_fans_out_per_name():
sink, dm = make_sink()
names = [
{'collection': 'raw_x', 'tag_name': 'Counter', 'model_id': '1', 'model_name': 'm1'},
{'collection': 'raw_y', 'tag_name': 'Contador', 'model_id': '2', 'model_name': 'm2'},
]
await sink.publish_reading(
tag='ns=2;i=2', value=42, timestamp='2026-07-13T00:00:00Z', names=names
)
assert dm.publish.await_count == 2
@pytest.mark.asyncio
async def test_publish_reading_builds_doc_and_routes_collection():
sink, dm = make_sink()
names = [{'collection': 'raw_x', 'tag_name': 'Counter', 'model_id': '1', 'model_name': 'm1'}]
await sink.publish_reading(
tag='ns=2;i=2', value=42, timestamp='2026-07-13T00:00:00Z', names=names
)
collection, doc = dm.publish.await_args_list[0].args
assert collection == 'raw_x'
assert doc == {
'tag': 'ns=2;i=2',
'name': 'Counter',
'model_id': '1',
'timestamp': '2026-07-13T00:00:00Z',
'value': 42,
}
@pytest.mark.asyncio
async def test_publish_reading_emits_tag_value_metric_per_name():
sink, dm = make_sink()
names = [
{'collection': 'raw_x', 'tag_name': 'Counter', 'model_id': '1', 'model_name': 'm1'},
{'collection': 'raw_y', 'tag_name': 'Contador', 'model_id': '2', 'model_name': 'm2'},
]
await sink.publish_reading(
tag='ns=2;i=2', value=42, timestamp='2026-07-13T00:00:00Z', names=names
)
assert sink.emit_metric.await_count == 2
sink.emit_metric.assert_any_await(
metric_object=metrics.OPC_TAG_VALUE,
method='set',
value=42,
tags={'pod_id': dm.pod_id, 'tag_name': 'Counter', 'model_id': '1'},
)

150
tests/unit/io/test_data_sink_status.py
View File

@@ -1,150 +0,0 @@
from unittest.mock import AsyncMock, MagicMock
import pytest
from sqlalchemy import create_engine, text
from sqlalchemy.dialects import postgresql
import ingestor.metrics as metrics
from ingestor.engines.write import WriteResult
from ingestor.io.data_sink import DataSink, write_status_statements
CMD = {
'server_id': '1',
'addr': 'ns=2;s=Pred',
'schema': 'sientia_data',
'table_name': 'pred_m1',
'model_id': '1',
'timestamp': '2026-07-13T00:00:00Z',
}
def _result(status):
return WriteResult(ok=status == 'ok', status=status, opc_status_code='x', message='')
def make_sink(rowcount=1, existing_row=True):
postgres = MagicMock()
connection = MagicMock()
connection.execute.return_value.rowcount = rowcount
connection.execute.return_value.first.return_value = (1,) if existing_row else None
postgres.engine.begin.return_value.__enter__.return_value = connection
data_manager = MagicMock()
data_manager.emit_metric = AsyncMock()
sink = DataSink(data_manager=data_manager, postgres=postgres)
return sink, connection
def _outcome_metric(sink):
calls = [
c
for c in sink.data_manager.emit_metric.await_args_list
if c.kwargs['metric_object'] is metrics.OPC_WRITE_STATUS_UPDATE_TOTAL
]
assert len(calls) == 1
return calls[0].kwargs['tags']['result']
@pytest.mark.asyncio
async def test_update_write_status_targets_model_row_with_precedence():
sink, connection = make_sink()
outcome = await sink.update_write_status(CMD, _result('ok'))
assert outcome == 'updated'
assert connection.execute.call_count == 1
(stmt,) = connection.execute.call_args[0]
compiled = stmt.compile(dialect=postgresql.dialect())
sql = str(compiled)
assert sql.startswith('UPDATE sientia_data.pred_m1 SET write_status=')
assert 'CASE sientia_data.pred_m1.write_status WHEN' in sql
# values must be bound params, never interpolated into the SQL text itself
assert '2026-07-13T00:00:00Z' not in sql
assert "'ok'" not in sql
assert compiled.params['write_status'] == 'ok'
assert '2026-07-13T00:00:00Z' in compiled.params.values()
assert _outcome_metric(sink) == 'updated'
@pytest.mark.asyncio
async def test_update_write_status_kept_when_the_row_has_a_worse_outcome():
sink, _ = make_sink(rowcount=0, existing_row=True)
assert await sink.update_write_status(CMD, _result('ok')) == 'kept'
assert _outcome_metric(sink) == 'kept'
@pytest.mark.asyncio
async def test_update_write_status_row_not_found_is_logged():
"""E.g. the Laborious INSERT failed and it wrote to OPC anyway (SIENTIAPDE-1975 D10)."""
sink, _ = make_sink(rowcount=0, existing_row=False)
assert await sink.update_write_status(CMD, _result('error')) == 'row_not_found'
assert 'no prediction row' in sink.data_manager.logger.warning.call_args.args[0]
assert _outcome_metric(sink) == 'row_not_found'
@pytest.mark.asyncio
async def test_update_write_status_database_error_is_not_raised():
sink, connection = make_sink()
connection.execute.side_effect = RuntimeError('column "write_status" does not exist')
assert await sink.update_write_status(CMD, _result('ok')) == 'error'
assert 'write_status' in sink.data_manager.logger.warning.call_args.args[0]
assert _outcome_metric(sink) == 'error'
@pytest.mark.parametrize(
('field', 'value'),
[
('schema', 'a; DROP TABLE x'),
('schema', 'sientia_data\n'),
('table_name', 'pred_m1; DROP TABLE x'),
('schema', None),
('timestamp', None),
('model_id', None),
],
)
@pytest.mark.asyncio
async def test_update_write_status_rejects_invalid_commands(field, value):
sink, connection = make_sink()
outcome = await sink.update_write_status({**CMD, field: value}, _result('error'))
assert outcome == 'invalid'
connection.execute.assert_not_called()
assert _outcome_metric(sink) == 'invalid'
def _apply(initial, answers):
"""Run the real statements on SQLite for each answer; return the final status."""
engine = create_engine('sqlite://')
with engine.begin() as connection:
connection.execute(
text('CREATE TABLE pred (model_id INTEGER, timestamp TEXT, write_status TEXT)')
)
connection.execute(
text('INSERT INTO pred VALUES (1, :ts, :status)'),
{'ts': '2026-07-13T00:00:00Z', 'status': initial},
)
for answer in answers:
update_stmt, exists_stmt = write_status_statements(
None, 'pred', answer, '2026-07-13T00:00:00Z', 1
)
connection.execute(update_stmt)
assert connection.execute(exists_stmt).first() is not None
return connection.execute(text('SELECT write_status FROM pred')).scalar()
@pytest.mark.parametrize(
('initial', 'answers', 'expected'),
[
('pending', ['ok'], 'ok'),
('pending', ['error', 'ok'], 'error'),
('pending', ['ok', 'error'], 'error'),
('pending', ['session_bad', 'error', 'ok'], 'session_bad'),
('error', ['ok'], 'error'),
(None, ['ok'], 'ok'),
],
)
def test_update_statement_keeps_the_worst_answer(initial, answers, expected):
assert _apply(initial, answers) == expected

131
tests/unit/io/test_slots.py
View File

@@ -1,131 +0,0 @@
import pytest
from ingestor.io.slots import CONNECTION_FIELDS, parse_slot
SECURITY = {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}
CERTIFICATE = {
'certificate_id': 1,
'application_uri': 'urn:sientia:opc-ingestor',
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/p/cert.pem'},
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/p/key.pem'},
}
NAME = {'collection': 'c', 'tag_name': 't', 'model_name': 'm', 'model_id': '1'}
RAW = {
'server1': {
'server_id': '1',
'name': 'server1',
'url': 'opc.tcp://h:4840',
'subscription_period_ms': 1000,
'security': SECURITY,
'certificate': CERTIFICATE,
'tags': {
'ns=2;i=2': {
'frequency': '15000',
'active_read': {'enabled': True, 'cycle_multiple': 3},
'names': [
{
'collection': 'raw_x',
'tag_name': 'Counter',
'model_name': 'm1',
'model_id': '1',
},
{
'collection': 'raw_y',
'tag_name': 'Contador',
'model_name': 'm2',
'model_id': '2',
},
],
}
},
}
}
def _server(**fields):
server = {
'server_id': '1',
'url': 'u',
'security': {'mode': 'None', 'policy': 'None'},
'certificate': None,
'tags': {'ns=2;i=1': {'frequency': '1000', 'names': [NAME]}},
}
return {'s': {**server, **fields}}
def test_parse_slot_multi_name():
slot = parse_slot(RAW)
tag = slot['server1']['tags']['ns=2;i=2']
assert len(tag['names']) == 2
assert tag['active_read']['cycle_multiple'] == 3
def test_parse_slot_keeps_security_and_certificate():
server = parse_slot(RAW)['server1']
assert server['security'] == SECURITY
assert server['certificate'] == CERTIFICATE
assert {field: server[field] for field in CONNECTION_FIELDS} == {
'url': 'opc.tcp://h:4840',
'subscription_period_ms': 1000,
'security': SECURITY,
'certificate': CERTIFICATE,
}
def test_parse_slot_without_certificate_key_means_no_certificate():
raw = _server()
del raw['s']['certificate']
server = parse_slot(raw)['s']
assert server['certificate'] is None
assert server['name'] == 's' # falls back to the slot key
def test_parse_slot_defaults_active_read_disabled():
tag = parse_slot(_server())['s']['tags']['ns=2;i=1']
assert tag['active_read'] == {'enabled': False}
@pytest.mark.parametrize(
'raw',
[
pytest.param(_server(tags={'ns=2;i=1': {'frequency': '1000'}}), id='tag-without-names'),
pytest.param(
_server(tags={'ns=2;i=1': {'frequency': '1', 'names': [{'collection': 'c'}]}}),
id='name-without-tag-name',
),
pytest.param(
_server(
tags={
'ns=2;i=1': {
'frequency': '1',
'names': [{'collection': 'c', 'tag_name': 't'}],
}
}
),
id='name-without-model-id',
),
pytest.param(_server(security=None), id='no-security'),
# Slot v1 (an Orchestrator before SIENTIAPDE-1974): no longer accepted
pytest.param(
{
'kepserver': {
'server_id': '1',
'url': 'opc.tcp://kepserver:49320',
'server_uri': 'urn:sientia:opc-ingestor',
'cert_path': '/app/certs/opc-ingestor-cert.pem',
'tags': {
'ns=2;s=A': {'tag_name': 'A', 'frequency': 30000, 'topics': ['raw_x']}
},
}
},
id='slot-v1',
),
],
)
def test_parse_slot_rejects_invalid_slots(raw):
with pytest.raises((KeyError, ValueError)):
parse_slot(raw)

45
tests/unit/managers/test_data_manager.py
View File

@@ -1,10 +1,9 @@
from unittest.mock import ANY, AsyncMock, MagicMock, patch
import pytest
from pytest import fixture, mark
from sientia_do.notifications.models import NotificationLevel
from ingestor.managers.data_manager import DataManager, redact_uri
from ingestor.managers.data_manager import DataManager
metadata = {
'metadata': {
@@ -55,48 +54,6 @@ def test___init___success(mongodb_repository):
assert data_manager.connection_string == 'mongodb://localhost:27017'
@patch('ingestor.managers.data_manager.MongoDBRepository')
def test___init___does_not_log_the_credentials(mongodb_repository):
logger_mock = MagicMock()
uri = 'mongodb://root:s3cr3t@mongodb.mongodb.svc.cluster.local:27017'
data_manager = DataManager(
metadata=metadata['metadata'],
mongo_connection_string=uri,
mongo_database='sientia',
logger=logger_mock,
notification_handler=MagicMock(),
metrics_controller=MagicMock(),
)
logged = ' '.join(str(c) for c in logger_mock.info.call_args_list)
assert 's3cr3t' not in logged
assert 'root' not in logged
logger_mock.info.assert_any_call(
'DataManager initialized with MongoDB servers: '
'mongodb://***@mongodb.mongodb.svc.cluster.local:27017'
)
assert data_manager.connection_string == uri
@pytest.mark.parametrize(
('uri', 'expected'),
[
('mongodb://localhost:27017', 'mongodb://localhost:27017'),
(
'mongodb://u:p@h1:27017,h2:27017/db?authSource=admin',
'mongodb://***@h1:27017,h2:27017/db?authSource=admin',
),
('mongodb://u:p@h:27017?authSource=admin', 'mongodb://***@h:27017?authSource=admin'),
('mongodb+srv://u@cluster.example.com', 'mongodb+srv://***@cluster.example.com'),
('u:p@h:27017', '***@h:27017'),
('mongodb://h:27017/db?x=a@b', 'mongodb://h:27017/db?x=a@b'),
],
)
def test_redact_uri(uri, expected):
assert redact_uri(uri) == expected
def test_shutdown(data_manager):
close_mock = MagicMock()
data_manager.mongo_repository.close = close_mock

378
tests/unit/managers/test_ingestor_manager.py
View File

@@ -1,15 +1,9 @@
import gc
from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
from unittest.mock import AsyncMock, MagicMock, call, patch
from pytest import fixture, mark
from sientia_do.notifications.models import NotificationLevel
import ingestor.metrics as metrics
from ingestor.config import GatewayConfig
from ingestor.connection.pool import OpcConnectionPool
from ingestor.io.slots import parse_slot
from ingestor.managers.ingestor_manager import IngestorManager
from ingestor.managers.server_reader import ServerReader
metadata = {
'metadata': {
@@ -21,43 +15,6 @@ metadata = {
}
SECURITY = {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}
CERTIFICATE = {
'certificate_id': 1,
'application_uri': 'urn:sientia:opc-ingestor',
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/p/cert.pem'},
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/p/key.pem'},
}
def raw_slot(tag_name='Counter', address='ns=2;i=2'):
"""A slot v2 as the Orchestrator writes it to Redis (contract C1)."""
name = {'collection': 'raw_a', 'tag_name': tag_name, 'model_id': '1', 'model_name': 'm'}
return {
'server1': {
'name': 'server1',
'server_id': '1',
'url': 'opc.tcp://localhost:4840',
'subscription_period_ms': 500.0,
'security': SECURITY,
'certificate': CERTIFICATE,
'tags': {address: {'frequency': '1000', 'names': [name]}},
}
}
@fixture(autouse=True)
def _collect_garbage_after_test():
# ponytail: IngestorManager.__del__ calls asyncio.run(self.shutdown()); if a
# cyclic-referenced instance from a previous test is swept mid-test by the
# generational GC, it can double-call whatever asyncio mock happens to be
# patched at that moment (e.g. test___del__). Force a deterministic sweep
# after each test so finalizers run in their own test's scope. Upgrade path:
# only needed if IngestorManager stops relying on __del__ for cleanup.
yield
gc.collect()
@fixture
@patch('ingestor.managers.ingestor_manager.DataManager')
@patch('ingestor.managers.ingestor_manager.ResourceManager')
@@ -82,12 +39,12 @@ def ingestor_manager(data_manager_mock, resource_manager_mock):
return ingestor
@patch('ingestor.managers.ingestor_manager.ServerReader')
@patch('ingestor.managers.ingestor_manager.OpcManager')
@patch('ingestor.managers.ingestor_manager.DataManager')
@patch('ingestor.managers.ingestor_manager.ResourceManager')
@patch('ingestor.managers.ingestor_manager.NotificationHandler')
def test___init__(
notification_handler_mock, resource_manager_mock, data_manager_mock, server_reader_mock
notification_handler_mock, resource_manager_mock, data_manager_mock, opc_manager_mock
):
ingestor = IngestorManager(
redis_data={'host': 'localhost', 'port': 6379},
@@ -102,7 +59,7 @@ def test___init__(
metrics_controller=MagicMock(),
)
server_reader_mock.assert_not_called()
opc_manager_mock.assert_not_called()
data_manager_mock.assert_called_once_with(
mongo_connection_string='mongodb://localhost:27017',
mongo_database='sientia',
@@ -130,84 +87,59 @@ def test___init__(
assert ingestor.data_manager == data_manager_mock.return_value
assert ingestor.resource_manager == resource_manager_mock.return_value
# Read path (SIENTIAPDE-1974): pool + read sink built here, nothing connected yet.
assert isinstance(ingestor.pool, OpcConnectionPool)
assert ingestor.pool.cert_provider is ingestor.cert_provider
assert ingestor.sink.data_manager is data_manager_mock.return_value
assert ingestor.sink.postgres is None
@patch('ingestor.managers.ingestor_manager.DataManager')
@patch('ingestor.managers.ingestor_manager.ResourceManager')
def test___init___cert_provider_uses_gateway_config(resource_manager_mock, data_manager_mock):
ingestor = IngestorManager(
redis_data={'host': 'localhost', 'port': 6379},
lease_ttl=60,
heartbeat_ttl=60,
poll_interval=5,
mongo_connection_string='mongodb://localhost:27017',
mongo_database='sientia',
logger=MagicMock(),
notification_handler=MagicMock(),
metadata=metadata['metadata'],
metrics_controller=MagicMock(),
gateway_config=GatewayConfig(cert_cache_dir='certs-cache'),
)
assert ingestor.cert_provider.cache_dir == 'certs-cache'
@mark.asyncio
@patch('ingestor.managers.ingestor_manager.ServerReader')
async def test_initialize_opc_from_config(server_reader, ingestor_manager):
@patch('ingestor.managers.ingestor_manager.OpcManager')
async def test_initialize_opc_from_config(opc_manager, ingestor_manager):
server_config = {
'name': 'server1',
'server_id': '1',
'url': 'opc.tcp://localhost:4840',
'subscription_period_ms': 1000,
'security': SECURITY,
'certificate': CERTIFICATE,
'server_uri': 'http://opcua-server.simulator',
'cert_path': '/path/to/cert',
'private_key_path': '/path/to/private_key',
'server_cert_path': '/path/to/server_cert',
'pod_id': 'test_pod',
}
server_reader.return_value = MagicMock(connect=AsyncMock())
opc_manager.return_value = MagicMock(connect=AsyncMock())
result = await ingestor_manager.initialize_opc_from_config(server_config)
server_reader.assert_called_once_with(
opc_manager.assert_called_once_with(
name=server_config['name'],
server_config=server_config,
pool=ingestor_manager.pool,
sink=ingestor_manager.sink,
url=server_config['url'],
data_manager=ingestor_manager.data_manager,
logger=ingestor_manager.logger,
subscription_period_ms=server_config['subscription_period_ms'],
server_uri=server_config['server_uri'],
notification_handler=ingestor_manager.notification_handler,
metrics_controller=ingestor_manager.metrics_controller,
cert_path=server_config['cert_path'],
private_key_path=server_config['private_key_path'],
server_cert_path=server_config['server_cert_path'],
metadata=metadata['metadata'],
metrics_controller=ingestor_manager.metrics_controller,
)
assert result == server_reader.return_value
assert result.config == server_config
result.connect.assert_awaited_once()
assert result == opc_manager.return_value
result.connect.assert_called_once()
@mark.asyncio
@patch('ingestor.managers.ingestor_manager.ServerReader')
@patch('ingestor.managers.ingestor_manager.OpcManager')
@patch('ingestor.managers.ingestor_manager.traceback')
async def test_initialize_opc_from_config_exception(
traceback_mock, server_reader, ingestor_manager
):
async def test_initialize_opc_from_config_exception(traceback_mock, opc_manager, ingestor_manager):
server_config = {
'name': 'server1',
'server_id': '1',
'url': 'opc.tcp://localhost:4840',
'subscription_period_ms': 1000,
'security': SECURITY,
'certificate': CERTIFICATE,
'server_uri': 'http://opcua-server.simulator',
'cert_path': '/path/to/cert',
'private_key_path': '/path/to/private_key',
'server_cert_path': '/path/to/server_cert',
}
ingestor_manager.logger.error = MagicMock()
# e.g. the server is unreachable or the certificate can't be fetched from MinIO
server_reader.return_value = MagicMock(
connect=AsyncMock(side_effect=Exception('Initialization error'))
)
opc_manager.side_effect = Exception('Initialization error')
result = await ingestor_manager.initialize_opc_from_config(server_config)
@@ -229,14 +161,11 @@ async def test_shutdown(ingestor_manager):
ingestor_manager.opc_managers = {'server1': AsyncMock(), 'server2': AsyncMock()}
ingestor_manager.data_manager.shutdown = MagicMock()
ingestor_manager.pool = MagicMock(close_all=AsyncMock())
await ingestor_manager.shutdown()
ingestor_manager.opc_managers['server1'].shutdown.assert_called_once()
ingestor_manager.opc_managers['server2'].shutdown.assert_called_once()
# The pool is owned by the read path: closed after the readers unsubscribe.
ingestor_manager.pool.close_all.assert_awaited_once()
ingestor_manager.data_manager.shutdown.assert_called_once()
@@ -276,8 +205,9 @@ async def test_remove_server_not_found(ingestor_manager):
@mark.asyncio
@patch('ingestor.managers.ingestor_manager.OpcManager')
@patch('ingestor.managers.ingestor_manager.metrics')
async def test_update_opc_servers(metrics, ingestor_manager):
async def test_update_opc_servers(metrics, opc_manager, ingestor_manager):
manager1 = MagicMock(config={'config': 'config1'})
manager2 = MagicMock(config={'config': 'config2'})
manager3 = MagicMock(config={'config': 'config3'})
@@ -324,9 +254,7 @@ async def test_update_opc_servers(metrics, ingestor_manager):
assert ingestor_manager.opc_managers['server2'].config == {'config': 'config2'}
assert ingestor_manager.opc_managers['server3'].config == {'config': 'config3'}
# server4 left this pod's slots: unsubscribe only, the connection stays for writes
ingestor_manager.remove_server.assert_any_call('server4', evict=False)
# server5 failed to initialize: removed with the default (evict)
ingestor_manager.remove_server.assert_any_call('server4')
ingestor_manager.remove_server.assert_any_call('server5')
assert ingestor_manager.remove_server.call_count == 2
@@ -490,47 +418,25 @@ async def test_get_number_of_slots_empty(metrics, ingestor_manager):
@mark.asyncio
async def test_get_slot_leases_1_success(ingestor_manager):
ingestor_manager.resource_manager.lease_tag = AsyncMock(return_value=True)
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(return_value=raw_slot())
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(return_value={'tags': ['tag1']})
ingestor_manager.number_of_slots = 1
result = await ingestor_manager.get_slot_leases()
# Stored already normalized to slot v2 (v1 converted by parse_slot).
assert result == {'1': parse_slot(raw_slot())}
assert ingestor_manager.managed_tags == result
assert result == {'1': {'tags': ['tag1']}}
@mark.asyncio
async def test_get_slot_leases_2_success(ingestor_manager):
ingestor_manager.resource_manager.lease_tag = AsyncMock(side_effect=[True, True])
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
side_effect=[raw_slot('Counter'), raw_slot('Ramp', 'ns=2;i=3')]
side_effect=[{'tags': ['tag1']}, {'tags': ['tag2']}]
)
ingestor_manager.number_of_slots = 2
result = await ingestor_manager.get_slot_leases(max_slots=2)
assert result == {
'1': parse_slot(raw_slot('Counter')),
'2': parse_slot(raw_slot('Ramp', 'ns=2;i=3')),
}
@mark.asyncio
async def test_get_slot_leases_skips_malformed_slot(ingestor_manager):
ingestor_manager.resource_manager.lease_tag = AsyncMock(return_value=True)
# tag without names/frequency
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
return_value={'server1': {'server_id': '1', 'url': 'u', 'tags': {'ns=2;i=2': {}}}}
)
ingestor_manager.number_of_slots = 1
result = await ingestor_manager.get_slot_leases()
assert result == {}
_, kwargs = ingestor_manager.send_notification.call_args
assert kwargs['notification_id'] == 'OPC_SLOT_CONFIG_ERROR_1'
assert kwargs['level'] == NotificationLevel.ERROR
assert result == {'1': {'tags': ['tag1']}, '2': {'tags': ['tag2']}}
@mark.asyncio
@@ -558,150 +464,40 @@ async def test_get_slot_leases_1_failure(ingestor_manager):
@mark.asyncio
async def test_unsubscribe_slot(ingestor_manager, reader_double):
async def test_unsubscribe_slot(ingestor_manager):
ingestor_manager.managed_tags = {
'slot1': {'server1': {'tags': 'config1'}, 'server2': {'tags': 'config2'}},
'slot2': {'server3': {'tags': 'config3'}, 'server1': {'tags': 'config1'}},
}
ingestor_manager.opc_managers = {
'server1': reader_double('slot1', 'slot2'),
'server2': reader_double('slot1'),
'server3': reader_double('slot2'),
'server1': AsyncMock(),
'server2': AsyncMock(),
'server3': AsyncMock(),
}
await ingestor_manager.unsubscribe_slot('slot1')
ingestor_manager.opc_managers['server1'].unsubscribe.assert_called_once_with('slot1')
ingestor_manager.opc_managers['server2'].unsubscribe.assert_called_once_with('slot1')
ingestor_manager.opc_managers['server3'].unsubscribe.assert_not_called()
assert list(ingestor_manager.opc_managers['server1'].subscriptions) == ['slot2']
@mark.asyncio
async def test_unsubscribe_slot_not_in_managed_tags(ingestor_manager, reader_double):
"""The slot is already gone from managed_tags: the readers still holding it are unsubscribed"""
ingestor_manager.managed_tags = {'slot1': {'server1': {'tags': 'config1'}}}
ingestor_manager.opc_managers = {
'server1': reader_double('slot1', 'slot3'),
'server2': reader_double('slot3'),
}
await ingestor_manager.unsubscribe_slot('slot3')
ingestor_manager.opc_managers['server1'].unsubscribe.assert_called_once_with('slot3')
ingestor_manager.opc_managers['server2'].unsubscribe.assert_called_once_with('slot3')
@mark.asyncio
async def test_unsubscribe_slot_twice_is_a_no_op(ingestor_manager, reader_double):
"""A second unsubscribe of the same slot (lease already dropped) does nothing and never raises"""
ingestor_manager.managed_tags = {}
ingestor_manager.opc_managers = {'server1': reader_double('slot2')}
await ingestor_manager.unsubscribe_slot('slot2')
await ingestor_manager.unsubscribe_slot('slot2')
ingestor_manager.opc_managers['server1'].unsubscribe.assert_called_once_with('slot2')
@mark.asyncio
async def test_unsubscribe_slot_reader_failure_does_not_raise(ingestor_manager, reader_double):
"""A reader that fails to unsubscribe is notified and forgotten; the others still go"""
broken = reader_double('slot1')
broken.unsubscribe = AsyncMock(side_effect=ConnectionError('session closed'))
healthy = reader_double('slot1')
ingestor_manager.opc_managers = {'server1': broken, 'server2': healthy}
await ingestor_manager.unsubscribe_slot('slot1')
healthy.unsubscribe.assert_called_once_with('slot1')
assert 'slot1' not in broken.subscriptions
ingestor_manager.emit_metric.assert_called_once_with(
metric_object=metrics.OPC_SUBSCRIPTION_ERRORS,
tags={'pod_id': ingestor_manager.pod_id, 'server': 'server1', 'slot': 'slot1'},
)
ingestor_manager.send_notification_async.assert_called_once_with(
metadata=ingestor_manager.metadata,
notification_id='OPC_UNSUBSCRIBE_ERROR_slot1:server1',
message='Failed to unsubscribe slot slot1 from server1: session closed',
block='opc_manager',
level=NotificationLevel.ERROR,
attachment_content=ANY,
)
@mark.asyncio
async def test_unsubscribe_slot_with_real_readers(ingestor_manager):
"""Contract: the real reader keeps `subscriptions` keyed by slot, and
unsubscribe_slot closes the slot's subscription"""
v2_subscription = AsyncMock()
client = MagicMock(create_subscription=AsyncMock(return_value=v2_subscription))
pool = MagicMock(get=AsyncMock(return_value=client), evict=AsyncMock())
server_reader = ServerReader(
name='kepserver',
server_config={
'server_id': '1',
'url': 'opc.tcp://kepserver:49320',
'subscription_period_ms': 1000.0,
},
pool=pool,
sink=AsyncMock(),
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
metadata={},
)
server_reader.engine.emit_metric = AsyncMock()
await server_reader.create_subscription('slot1')
assert 'slot1' in server_reader.subscriptions
ingestor_manager.opc_managers = {'kepserver': server_reader}
await ingestor_manager.unsubscribe_slot('slot1')
v2_subscription.delete.assert_awaited_once()
assert 'slot1' not in server_reader.subscriptions
pool.pin.assert_called_once_with('1')
pool.unpin.assert_called_once_with('1')
@mark.asyncio
async def test_update_slot_config_unsubscribes_removed_slot(ingestor_manager, reader_double):
"""A slot deleted from Redis is unsubscribed right away, not left to the main loop"""
ingestor_manager.managed_tags = {
'slot1': parse_slot(raw_slot('Same')),
'slot2': parse_slot(raw_slot('Same')),
}
reader = reader_double('slot1', 'slot2')
ingestor_manager.opc_managers = {'test_server': reader}
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
side_effect=lambda slot: raw_slot('Same') if slot == 'slot1' else None
)
ingestor_manager.resource_manager.renew_tag_lease = AsyncMock()
await ingestor_manager.update_slot_config()
reader.unsubscribe.assert_called_once_with('slot2')
assert list(ingestor_manager.managed_tags) == ['slot1']
assert list(reader.subscriptions) == ['slot1']
@mark.asyncio
async def test_update_slot_config(ingestor_manager):
ingestor_manager.managed_tags = {
'slot1': parse_slot(raw_slot('Old')),
'slot2': parse_slot(raw_slot('Same')),
'slot3': parse_slot(raw_slot('Old')),
'slot1': {'config': 'old_config'},
'slot2': {'config': 'new_config'},
'slot3': {'config': 'old_config'},
}
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
side_effect=[raw_slot('Updated'), raw_slot('Same'), None]
side_effect=[{'config': 'updated_config'}, {'config': 'new_config'}, None]
)
ingestor_manager.resource_manager.renew_tag_lease = AsyncMock()
await ingestor_manager.update_slot_config()
assert ingestor_manager.managed_tags['slot1'] == parse_slot(raw_slot('Updated'))
assert ingestor_manager.managed_tags['slot2'] == parse_slot(raw_slot('Same'))
assert ingestor_manager.managed_tags['slot1'] == {'config': 'updated_config'}
assert ingestor_manager.managed_tags['slot2'] == {'config': 'new_config'}
assert 'slot3' not in ingestor_manager.managed_tags
ingestor_manager.resource_manager.renew_tag_lease.assert_any_call('slot1')
@@ -766,30 +562,6 @@ async def test_manage_server_create_subscription_failure(ingestor_manager):
ingestor_manager.opc_managers['server1'].subscribe.assert_not_called()
@mark.asyncio
async def test_manage_server_subscribe_and_unsubscribe_failure(ingestor_manager):
"""subscribe() fails and so does the cleanup unsubscribe (session gone):
manage_server reports the failure and nothing escapes to the main loop"""
reader = AsyncMock()
reader.subscriptions = {}
reader.subscribe.side_effect = Exception('Subscription error')
reader.unsubscribe.side_effect = ConnectionError('session closed')
ingestor_manager.opc_managers = {'server1': reader}
server_config = {'tags': {'tag1': 'config1'}}
result = await ingestor_manager.manage_server('slot1', 'server1', server_config, server_config)
assert result == 2
reader.unsubscribe.assert_called_once_with('slot1')
notified = [
c.kwargs['notification_id'] for c in ingestor_manager.send_notification_async.call_args_list
]
assert notified == [
'OPC_SUBSCRIPTION_ERROR_slot1:server1',
'OPC_UNSUBSCRIBE_ERROR_slot1:server1',
]
@mark.asyncio
async def test_manage_server(ingestor_manager):
ingestor_manager.opc_managers = {'server1': AsyncMock(), 'server2': AsyncMock()}
@@ -867,59 +639,3 @@ async def test_subscribe_to_tags(ingestor_manager):
assert ingestor_manager.manage_server.call_count == 3
ingestor_manager.managed_tags['slot1'].pop.assert_called_once_with('server3', None)
@mark.asyncio
async def test_update_slot_config_keeps_last_good_config_when_malformed(ingestor_manager):
good = parse_slot(raw_slot('Counter'))
ingestor_manager.managed_tags = {'slot1': good}
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
return_value={'server1': {'server_id': '1', 'url': 'u', 'tags': {'ns=2;i=2': {}}}}
)
ingestor_manager.resource_manager.renew_tag_lease = AsyncMock()
await ingestor_manager.update_slot_config()
# Still reading with the last good config; the bad one is notified, not applied.
assert ingestor_manager.managed_tags == {'slot1': good}
_, kwargs = ingestor_manager.send_notification.call_args
assert kwargs['notification_id'] == 'OPC_SLOT_CONFIG_ERROR_slot1'
@mark.asyncio
async def test_remove_server_forwards_evict(ingestor_manager):
reader = AsyncMock()
ingestor_manager.opc_managers = {'server1': reader}
ingestor_manager.managed_tags = {'slot1': {'server1': {}}}
await ingestor_manager.remove_server('server1', evict=False)
reader.shutdown.assert_awaited_once_with(evict=False)
assert 'server1' not in ingestor_manager.opc_managers
assert ingestor_manager.managed_tags == {'slot1': {}}
@patch('ingestor.managers.ingestor_manager.DataManager')
@patch('ingestor.managers.ingestor_manager.ResourceManager')
def test___init___ignores_malformed_write_env(
resource_manager_mock, data_manager_mock, monkeypatch
):
"""A malformed write/gRPC-only env var must not stop the read path (review #1)."""
monkeypatch.setenv('GRPC_PORT', '')
monkeypatch.setenv('WRITE_MIN_PAUSE_S', '0,5')
monkeypatch.setenv('CERT_CACHE_DIR', 'certs-cache')
ingestor = IngestorManager(
redis_data={'host': 'localhost', 'port': 6379},
lease_ttl=60,
heartbeat_ttl=60,
poll_interval=5,
mongo_connection_string='mongodb://localhost:27017',
mongo_database='sientia',
logger=MagicMock(),
notification_handler=MagicMock(),
metadata=metadata['metadata'],
metrics_controller=MagicMock(),
)
assert ingestor.cert_provider.cache_dir == 'certs-cache'

627
tests/unit/managers/test_opc_manager.py Normal file
View File

@@ -0,0 +1,627 @@
import json
from datetime import datetime
from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
import pytest
from asyncua.crypto.security_policies import SecurityPolicyBasic256
from pytest import fixture, mark
from sientia_do.notifications.models import NotificationLevel
from ingestor.managers.opc_manager import OpcManager
tags = {
'ns=3;i=1001': {
'aggregation_function': 'LTS',
'frequency': 1000,
'max_value': 100,
'min_value': 0,
'tag_name': 'Counter',
},
'ns=3;i=1003': {
'aggregation_function': 'AVG',
'frequency': 1000,
'max_value': 100,
'min_value': 0,
'tag_name': 'Random',
},
'ns=3;i=1004': {
'aggregation_function': 'MDN',
'frequency': 1000,
'max_value': 100,
'min_value': 0,
'tag_name': 'Sawtooth',
},
}
metadata = {
'metadata': {
'model_id': 'test_model',
'model_name': 'test_model',
'workflow_name': 'test_workflow',
'schema_name': 'test_schedule',
},
}
@fixture
@patch('ingestor.managers.opc_manager.metrics')
def raw_opc_manager(mock_metrics):
opc_manager = OpcManager(
name='TestConnector',
url='opc.tcp://localhost:4840',
data_manager=AsyncMock(),
subscription_period_ms=1000,
logger=MagicMock(),
server_uri='opc.tcp://localhost:4840',
notification_handler=MagicMock(),
metadata=metadata['metadata'],
metrics_controller=AsyncMock(),
)
opc_manager.emit_metric = AsyncMock()
opc_manager.send_notification_async = AsyncMock()
opc_manager.send_notification = MagicMock()
return opc_manager
@fixture
def opc_manager(raw_opc_manager):
raw_opc_manager.client = AsyncMock()
raw_opc_manager.cert_path = 'cert.pem'
raw_opc_manager.private_key_path = 'private_key.pem'
raw_opc_manager.server_cert_path = 'server_cert.pem'
return raw_opc_manager
@fixture
def opc_manager_subscribed(opc_manager):
opc_manager.subscriptions['sub1'] = AsyncMock()
return opc_manager
def test___str__(opc_manager):
assert (
str(opc_manager)
== 'OpcManager(name=TestConnector, url=opc.tcp://localhost:4840, server_uri=opc.tcp://localhost:4840)\nnodes={}, subscriptions={}'
)
@mark.asyncio
async def test_shutdown_success(opc_manager):
opc_manager.disconnect = AsyncMock()
await opc_manager.shutdown()
opc_manager.disconnect.assert_called_once()
@mark.asyncio
async def test_shutdown_error(opc_manager):
opc_manager.disconnect = AsyncMock(side_effect=Exception('Test error'))
await opc_manager.shutdown()
opc_manager.logger.error.assert_called_once_with('Error during cleanup: Test error')
@mark.asyncio
async def test_set_security_success(opc_manager):
await opc_manager.set_security()
assert opc_manager.client.application_uri == opc_manager.server_uri
opc_manager.client.set_security.assert_called_once_with(
SecurityPolicyBasic256,
certificate=opc_manager.cert_path,
private_key=opc_manager.private_key_path,
server_certificate=opc_manager.server_cert_path,
)
assert opc_manager.client.secure_channel_timeout == 10000000
assert opc_manager.client.session_timeout == 10000000
@mark.asyncio
async def test_set_security_no_cert(opc_manager):
opc_manager.cert_path = None
opc_manager.private_key_path = None
try:
await opc_manager.set_security()
except ValueError as e:
assert str(e) == 'Certificate and private key paths must be provided for secure connection.'
else:
raise AssertionError('ValueError not raised')
assert opc_manager.client.set_security.call_count == 0
@mark.asyncio
@patch('ingestor.managers.opc_manager.metrics')
@patch('ingestor.managers.opc_manager.Client')
async def test_connect_no_security(client, mock_metrics, raw_opc_manager):
raw_opc_manager.set_security = AsyncMock()
client.return_value = AsyncMock()
await raw_opc_manager.connect()
client.assert_called_once_with(raw_opc_manager.url, timeout=10, watchdog_intervall=3600000)
raw_opc_manager.client.connect.assert_called_once()
raw_opc_manager.set_security.assert_not_called()
raw_opc_manager.emit_metric.assert_has_calls(
[
call(
metric_object=mock_metrics.OPC_CONNECTIONS_TOTAL,
tags={
'pod_id': raw_opc_manager.pod_id,
'server_name': raw_opc_manager.name,
},
),
call(
metric_object=mock_metrics.OPC_CONNECTION_STATUS,
method='set',
value=1,
tags={
'pod_id': raw_opc_manager.pod_id,
'server_name': raw_opc_manager.name,
'server_url': raw_opc_manager.url,
},
),
],
any_order=True,
)
@mark.asyncio
@patch('ingestor.managers.opc_manager.Client')
async def test_connect_with_security(client, raw_opc_manager):
raw_opc_manager.cert_path = 'cert.pem'
raw_opc_manager.private_key_path = 'private_key.pem'
raw_opc_manager.server_cert_path = 'server_cert.pem'
raw_opc_manager.set_security = AsyncMock()
client.return_value = AsyncMock()
await raw_opc_manager.connect()
client.assert_called_once_with(raw_opc_manager.url, timeout=10, watchdog_intervall=3600000)
raw_opc_manager.client.connect.assert_called_once()
raw_opc_manager.set_security.assert_called_once()
@mark.asyncio
@patch('ingestor.managers.opc_manager.Client')
@patch('ingestor.managers.opc_manager.metrics')
async def test_connect_exception_handling_and_metrics(
mock_metrics_module, mock_opc_client_class, raw_opc_manager
):
mock_client_instance = mock_opc_client_class.return_value
simulated_error_message = 'Erro de conexão simulado'
mock_client_instance.connect.side_effect = Exception(simulated_error_message)
raw_opc_manager.disconnect = AsyncMock()
opc_manager_instance = raw_opc_manager
opc_manager_instance.cert_path = None
with pytest.raises(Exception, match=simulated_error_message):
await opc_manager_instance.connect()
opc_manager_instance.disconnect.assert_called_once()
@mark.asyncio
async def test_create_subscription_no_client(raw_opc_manager):
try:
await raw_opc_manager.create_subscription('sub1')
except ValueError as e:
assert str(e) == 'Client not connected. Call connect first.'
else:
raise AssertionError('ValueError not raised')
@mark.asyncio
async def test_create_subscription_success_has_period(opc_manager):
await opc_manager.create_subscription('sub1')
opc_manager.client.create_subscription.assert_called_once_with(
opc_manager.subscription_period_ms, opc_manager
)
assert opc_manager.subscriptions['sub1'] is not None
@mark.asyncio
async def test_create_subscription_success_no_period(opc_manager):
await opc_manager.create_subscription('sub1')
opc_manager.client.create_subscription.assert_called_once_with(
opc_manager.subscription_period_ms, opc_manager
)
assert opc_manager.subscriptions['sub1'] is not None
@patch('ingestor.managers.opc_manager.metrics')
@mark.asyncio
async def test_create_subscription_with_metrics(metrics, opc_manager):
await opc_manager.create_subscription('sub1')
opc_manager.emit_metric.assert_called_once_with(
metric_object=metrics.OPC_SUBSCRIPTIONS_CREATED,
method='inc',
value=1,
tags={
'pod_id': opc_manager.pod_id,
'server_name': opc_manager.name,
'slot_name': 'sub1',
},
)
@patch('ingestor.managers.opc_manager.metrics')
@mark.asyncio
async def test_create_subscription_exception_during_client_call(
mock_metrics_module, raw_opc_manager
):
opc_manager_instance = raw_opc_manager
opc_manager_instance.client = AsyncMock()
subscription_name = 'test_sub_client_error'
simulated_error_message = 'Falha ao criar subscrição no cliente OPC'
opc_manager_instance.client.create_subscription.side_effect = Exception(simulated_error_message)
with pytest.raises(Exception, match=simulated_error_message):
await opc_manager_instance.create_subscription(subscription_name)
opc_manager_instance.client.create_subscription.assert_called_once_with(
opc_manager_instance.subscription_period_ms, opc_manager_instance
)
opc_manager_instance.logger.error.assert_called_once_with(
f'Failed to create subscription {subscription_name} on {opc_manager_instance.name}: {simulated_error_message}'
)
mock_metrics_module.OPC_SUBSCRIPTIONS_CREATED.labels.assert_not_called()
@patch('ingestor.managers.opc_manager.metrics')
@mark.asyncio
async def test_subscribe_no_subscription(metrics, opc_manager):
try:
await opc_manager.subscribe('sub1', tags, 1000)
except ValueError as e:
assert str(e) == 'Subscription not created. Call create_subscription first.'
else:
raise AssertionError('ValueError not raised')
metrics.OPC_TAGS_SUBSCRIBED.labels.assert_not_called()
@mark.asyncio
async def test_subscribe_success(opc_manager_subscribed):
opc_manager_subscribed.client.get_node = MagicMock()
opc_manager_subscribed.nodes = {'ns=3;i=1001': 'data'}
await opc_manager_subscribed.subscribe('sub1', tags, 1000)
assert opc_manager_subscribed.nodes == tags
opc_manager_subscribed.subscriptions['sub1'].subscribe_data_change.assert_called_once_with(
[opc_manager_subscribed.client.get_node(n) for n in tags]
)
@mark.asyncio
async def test_unsubscribe_no_subscription(opc_manager):
await opc_manager.unsubscribe('sub1')
opc_manager.logger.warning.assert_called_once_with(
"Subscription 'sub1' not found. Cannot unsubscribe."
)
assert opc_manager.subscriptions.get('sub1') is None
@mark.asyncio
async def test_unsubscribe_success(opc_manager_subscribed):
await opc_manager_subscribed.unsubscribe('sub1')
assert opc_manager_subscribed.subscriptions.get('sub1') is None
@mark.asyncio
async def test_disconnection_fallback_success(opc_manager):
opc_manager.client = AsyncMock()
opc_manager.client.disconnect.return_value = True
result = await opc_manager.disconnection_fallback()
assert result == []
@mark.asyncio
async def test_disconnection_fallback_fail(opc_manager):
opc_manager.client = AsyncMock()
opc_manager.client.disconnect.side_effect = Exception('Test error')
result = await opc_manager.disconnection_fallback()
assert result == [
{'attempt': 1, 'error': 'Test error', 'traceback': ANY},
{'attempt': 2, 'error': 'Test error', 'traceback': ANY},
{'attempt': 3, 'error': 'Test error', 'traceback': ANY},
{'attempt': 4, 'error': 'Test error', 'traceback': ANY},
{'attempt': 5, 'error': 'Test error', 'traceback': ANY},
]
assert opc_manager.client.disconnect.call_count == 5
@mark.asyncio
async def test_disconnect_success(opc_manager_subscribed):
opc_manager_subscribed.client = MagicMock()
await opc_manager_subscribed.disconnect()
opc_manager_subscribed.subscriptions['sub1'].delete.assert_called_once()
assert opc_manager_subscribed.client is None
@mark.asyncio
async def test_disconnect_no_client(opc_manager_subscribed):
opc_manager_subscribed.client = None
assert await opc_manager_subscribed.disconnect() is None
opc_manager_subscribed.logger.warning.assert_has_calls(
[
call('Client already disconnected.'),
]
)
@mark.asyncio
async def test_disconnect_error_unsubscribe(opc_manager_subscribed):
opc_manager_subscribed.client = MagicMock(disconnect=AsyncMock())
opc_manager_subscribed.subscriptions['sub1'] = MagicMock(
delete=AsyncMock(side_effect=Exception('Test error'))
)
await opc_manager_subscribed.disconnect()
opc_manager_subscribed.subscriptions['sub1'].delete.assert_called_once()
opc_manager_subscribed.client = None
opc_manager_subscribed.logger.error.assert_called_once_with(
'Failed to clean up subscription: Test error'
)
@mark.asyncio
async def test_disconnect_error(opc_manager_subscribed):
opc_manager_subscribed.client = MagicMock()
opc_manager_subscribed.client.disconnect = MagicMock(side_effect=Exception('Test error'))
await opc_manager_subscribed.disconnect()
opc_manager_subscribed.subscriptions['sub1'].delete.assert_called_once()
opc_manager_subscribed.client = None
@patch('ingestor.managers.opc_manager.metrics')
@mark.asyncio
async def test_disconnect_metrics_on_successful_path(mock_metrics_module, raw_opc_manager):
mock_metrics_module.OPC_CONNECTION_STATUS.reset_mock()
mock_metrics_module.OPC_TAGS_SUBSCRIBED.reset_mock()
raw_opc_manager.client = MagicMock()
mock_sub1 = MagicMock()
mock_sub2 = MagicMock()
raw_opc_manager.subscriptions = {'sub1': mock_sub1, 'sub2': mock_sub2}
await raw_opc_manager.disconnect()
raw_opc_manager.emit_metric.assert_has_calls(
[
call(
metric_object=mock_metrics_module.OPC_CONNECTION_STATUS,
method='set',
value=0,
tags={
'pod_id': raw_opc_manager.pod_id,
'server_name': raw_opc_manager.name,
'server_url': raw_opc_manager.url,
},
),
call(
metric_object=mock_metrics_module.OPC_TAGS_SUBSCRIBED,
method='set',
value=0,
tags={
'pod_id': raw_opc_manager.pod_id,
'server_name': raw_opc_manager.name,
},
),
],
any_order=True,
)
@patch('ingestor.managers.opc_manager.metrics')
@mark.asyncio
async def test_datachange_notification(metrics, opc_manager_subscribed):
data = MagicMock(
monitored_item=MagicMock(
Value=MagicMock(
Value=MagicMock(Value=42),
SourceTimestamp=datetime.strptime('2021-01-01T00:00:00', '%Y-%m-%dT%H:%M:%S'),
)
)
)
opc_manager_subscribed.nodes = {
'ns=3;i=1001': {
'tag_name': 'Counter',
'cycle_rule': {'cycle_increment': 1.0, 'cycle_count': 2},
'topics': ['topic1', 'topic2'],
}
}
await opc_manager_subscribed.datachange_notification('ns=3;i=1001', None, data)
opc_manager_subscribed.data_manager.publish.assert_any_call(
'topic1',
{
'tag': 'ns=3;i=1001',
'name': 'Counter',
'timestamp': '2021-01-01 00:00:00-0300',
'value': 42,
},
)
opc_manager_subscribed.data_manager.publish.assert_any_call(
'topic2',
{
'tag': 'ns=3;i=1001',
'name': 'Counter',
'timestamp': '2021-01-01 00:00:00-0300',
'value': 42,
},
)
assert opc_manager_subscribed.nodes['ns=3;i=1001']['cycle_rule']['cycle_count'] == 0
opc_manager_subscribed.emit_metric.assert_called_once_with(
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
method='set',
value=0,
tags={
'pod_id': opc_manager_subscribed.pod_id,
'server_name': opc_manager_subscribed.name,
},
)
@mark.asyncio
async def test_check_cycles_no_notification(opc_manager):
# Setup: node with cycle_count just below threshold
opc_manager.nodes = {
'ns=3;i=1001': {
'tag_name': 'Counter',
'cycle_rule': {'cycle_increment': 1.0, 'cycle_count': 3.0},
}
}
await opc_manager.check_cycles()
# After one increment, cycle_count = 4.0, still below threshold
assert opc_manager.nodes['ns=3;i=1001']['cycle_rule']['cycle_count'] == pytest.approx(4.0)
opc_manager.send_notification_async.assert_not_called()
@mark.asyncio
async def test_check_cycles_triggers_notification(opc_manager):
# Setup: node with cycle_count just below threshold, increment will cross threshold
opc_manager.nodes = {
'ns=3;i=1001': {
'tag_name': 'Counter',
'cycle_rule': {'cycle_increment': 2.5, 'cycle_count': 3.0},
}
}
opc_manager.notification_handler.build_and_send_notification = MagicMock()
await opc_manager.check_cycles()
# After increment, cycle_count = 5.5, should trigger notification
assert opc_manager.nodes['ns=3;i=1001']['cycle_rule']['cycle_count'] == pytest.approx(5.5)
opc_manager.send_notification_async.assert_called_once_with(
notification_id='TAG_ns=3;i=1001:Counter_LISTENNING_STOPPED',
message='5.5 cycles without receive from ns=3;i=1001:Counter',
block='opc_manager',
level=NotificationLevel.WARNING,
metadata=metadata['metadata'],
)
@patch('ingestor.managers.opc_manager.metrics')
@mark.asyncio
async def test_check_opc_listenning_no_notification(metrics, opc_manager):
opc_manager.non_receive_count = 3
result = await opc_manager.check_opc_listenning()
assert opc_manager.non_receive_count == 4
opc_manager.send_notification_async.assert_not_called()
assert result is False
opc_manager.emit_metric.assert_called_once_with(
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
method='set',
value=opc_manager.non_receive_count,
tags={
'pod_id': opc_manager.pod_id,
'server_name': opc_manager.name,
},
)
@mark.asyncio
async def test_check_opc_listenning_warning_notification(opc_manager):
opc_manager.non_receive_count = 4
result = await opc_manager.check_opc_listenning()
assert opc_manager.non_receive_count == 5
opc_manager.send_notification_async.assert_called_once_with(
notification_id=f'OPC_LISTENNING_STOPPED__{opc_manager.name}',
message=f'5 cycles without receive from OPC {opc_manager.name}. Tags: {json.dumps(opc_manager.nodes)}',
block='opc_manager',
level=NotificationLevel.ERROR,
metadata=metadata['metadata'],
)
assert result is False
@patch('ingestor.managers.opc_manager.metrics')
@mark.asyncio
async def test_check_opc_listenning_error_notification_and_retry(metrics, opc_manager):
opc_manager.non_receive_count = 14
result = await opc_manager.check_opc_listenning()
assert opc_manager.non_receive_count == 15
# Should be called twice: once for 5, once for 15
assert opc_manager.send_notification_async.call_count == 2
calls = opc_manager.send_notification_async.call_args_list
# First call: 5 cycles warning
assert calls[0].kwargs == {
'notification_id': f'OPC_LISTENNING_STOPPED__{opc_manager.name}',
'message': f'15 cycles without receive from OPC {opc_manager.name}. Tags: {json.dumps(opc_manager.nodes)}',
'block': 'opc_manager',
'level': NotificationLevel.ERROR,
'metadata': metadata['metadata'],
}
# Second call: 15 cycles retry
assert calls[1].kwargs == {
'notification_id': f'OPC_CONNECTION_RETRY__{opc_manager.name}',
'message': f'Retrying to connect to server {opc_manager.name}',
'block': 'opc_manager',
'level': NotificationLevel.ERROR,
'metadata': metadata['metadata'],
}
assert result is True
opc_manager.emit_metric.assert_has_calls(
[
call(
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
method='set',
value=opc_manager.non_receive_count,
tags={
'pod_id': opc_manager.pod_id,
'server_name': opc_manager.name,
},
),
]
)
opc_manager.emit_metric.assert_has_calls(
[
call(
metric_object=metrics.OPC_RECONNECTIONS_TOTAL,
tags={
'pod_id': opc_manager.pod_id,
'server_name': opc_manager.name,
},
),
]
)

167
tests/unit/managers/test_server_reader.py
View File

@@ -1,167 +0,0 @@
from unittest.mock import AsyncMock, MagicMock, call
import pytest
import ingestor.metrics as metrics
from ingestor.managers.server_reader import ServerReader
SERVER_CONFIG = {
'name': 'kepserver',
'server_id': 1, # the slot may carry it as int; the pool keys by str
'url': 'opc.tcp://kepserver:49320',
'subscription_period_ms': 15000.0,
'security': {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'},
'certificate': {
'certificate_id': 1,
'application_uri': 'urn:sientia:opc-ingestor',
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/p/cert.pem'},
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/p/key.pem'},
},
}
def make_reader():
pool = MagicMock(get=AsyncMock(), evict=AsyncMock())
reader = ServerReader(
name='kepserver',
server_config=SERVER_CONFIG,
pool=pool,
sink=AsyncMock(),
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
metadata={},
)
reader.engine = MagicMock(
subscriptions={},
pod_id='pod',
create_subscription=AsyncMock(),
subscribe=AsyncMock(),
unsubscribe=AsyncMock(),
check_cycles=AsyncMock(),
check_opc_listenning=AsyncMock(return_value=False),
emit_metric=AsyncMock(),
)
return reader, pool
def test_server_cfg_keeps_only_connection_fields():
reader, _ = make_reader()
assert reader.server_id == '1'
assert reader.server_cfg == {
key: SERVER_CONFIG[key]
for key in ('url', 'subscription_period_ms', 'security', 'certificate')
}
@pytest.mark.asyncio
async def test_connect_opens_pooled_connection():
reader, pool = make_reader()
await reader.connect()
# With a certificate, the pool fetches it from MinIO (CertProvider)
pool.get.assert_awaited_once_with('1', reader.server_cfg)
@pytest.mark.asyncio
async def test_connect_propagates_failure():
reader, pool = make_reader()
pool.get.side_effect = ConnectionError('unreachable')
with pytest.raises(ConnectionError):
await reader.connect()
@pytest.mark.asyncio
async def test_subscription_calls_delegate_to_engine_per_slot():
reader, _ = make_reader()
nodes = {'ns=2;i=2': {'frequency': '1000', 'names': [{'collection': 'raw_a', 'tag_name': 'a'}]}}
await reader.create_subscription('1')
await reader.subscribe('1', nodes, 10)
await reader.unsubscribe('1')
reader.engine.create_subscription.assert_awaited_once_with('1', reader.server_cfg, '1')
reader.engine.subscribe.assert_awaited_once_with('1', nodes, 10)
reader.engine.unsubscribe.assert_awaited_once_with('1')
@pytest.mark.asyncio
async def test_supervision_delegates_to_engine():
reader, _ = make_reader()
reader.engine.check_opc_listenning.return_value = True
await reader.check_cycles()
lost = await reader.check_opc_listenning()
reader.engine.check_cycles.assert_awaited_once()
reader.engine.check_opc_listenning.assert_awaited_once_with('1')
assert lost is True
def test_subscriptions_are_the_engine_ones():
reader, _ = make_reader()
reader.engine.subscriptions['1'] = MagicMock()
assert reader.subscriptions is reader.engine.subscriptions
@pytest.mark.asyncio
async def test_shutdown_unsubscribes_all_slots_evicts_and_zeroes_metrics():
reader, pool = make_reader()
reader.engine.subscriptions.update({'1': MagicMock(), '2': MagicMock()})
await reader.shutdown()
reader.engine.unsubscribe.assert_has_awaits([call('1'), call('2')])
# Dropping the pooled connection makes the next initialization reconnect.
pool.evict.assert_awaited_once_with('1')
reader.engine.emit_metric.assert_any_await(
metric_object=metrics.OPC_CONNECTION_STATUS,
method='set',
value=0,
tags={'pod_id': 'pod', 'server_name': '1', 'server_url': 'opc.tcp://kepserver:49320'},
)
reader.engine.emit_metric.assert_any_await(
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
method='set',
value=0,
tags={'pod_id': 'pod', 'server_name': '1'},
)
@pytest.mark.asyncio
async def test_shutdown_without_evict_keeps_the_shared_connection():
"""Slot moved to another pod: drop the subscriptions, keep the pooled
connection for NATS writes/gRPC (review #7)."""
reader, pool = make_reader()
reader.engine.subscriptions.update({'1': MagicMock()})
await reader.shutdown(evict=False)
reader.engine.unsubscribe.assert_awaited_once_with('1')
pool.evict.assert_not_awaited()
for c in reader.engine.emit_metric.await_args_list:
assert c.kwargs['metric_object'] is not metrics.OPC_CONNECTION_STATUS
@pytest.mark.asyncio
async def test_shutdown_unsubscribes_every_slot_while_engine_pops_them():
"""The real unsubscribe() pops from engine.subscriptions: shutdown must iterate a
snapshot (list()), or the loop breaks with "dictionary changed size during
iteration" and the remaining slots stay subscribed (SonarQube S7504 does not apply)."""
reader, pool = make_reader()
subs = {'1': MagicMock(), '2': MagicMock(), '3': MagicMock()}
reader.engine.subscriptions = subs
async def pop(slot):
subs.pop(slot)
reader.engine.unsubscribe = AsyncMock(side_effect=pop)
await reader.shutdown()
assert subs == {}
assert reader.engine.unsubscribe.await_count == 3

0
tests/unit/proto/__init__.py
View File

36
tests/unit/proto/test_proto.py
View File

@@ -1,36 +0,0 @@
from ingestor.proto import opc_gateway_pb2, opc_gateway_pb2_grpc
def test_read_tag_request_roundtrip():
req = opc_gateway_pb2.ReadTagRequest(server_id='s1', addr='ns=2;s=Tag1', data_type='Float')
data = req.SerializeToString()
parsed = opc_gateway_pb2.ReadTagRequest()
parsed.ParseFromString(data)
assert parsed.server_id == 's1'
assert parsed.addr == 'ns=2;s=Tag1'
assert parsed.data_type == 'Float'
def test_read_tag_response_roundtrip():
resp = opc_gateway_pb2.ReadTagResponse(
value='42.0',
opc_status_code='Good',
source_timestamp='2026-07-14T00:00:00Z',
server_timestamp='2026-07-14T00:00:01Z',
persisted=True,
)
data = resp.SerializeToString()
parsed = opc_gateway_pb2.ReadTagResponse()
parsed.ParseFromString(data)
assert parsed.value == '42.0'
assert parsed.opc_status_code == 'Good'
assert parsed.persisted is True
def test_grpc_stub_module_loads():
assert hasattr(opc_gateway_pb2_grpc, 'OpcGatewayStub')
assert hasattr(opc_gateway_pb2_grpc, 'OpcGatewayServicer')

62
tests/unit/test_app.py
View File

@@ -43,15 +43,6 @@ def mock_app_env(monkeypatch):
monkeypatch.setattr(app, 'Ingestor', mocks['Ingestor'])
monkeypatch.setattr(asyncio, 'sleep', mocks['asyncio_sleep'])
mocks['GatewayConfig'] = MagicMock()
mocks['GatewayRuntime'] = MagicMock()
monkeypatch.setattr(app, 'GatewayConfig', mocks['GatewayConfig'])
monkeypatch.setattr(app, 'GatewayRuntime', mocks['GatewayRuntime'])
mock_runtime_instance = mocks['GatewayRuntime'].return_value
mock_runtime_instance.start = AsyncMock()
mock_runtime_instance.idle_sweep = AsyncMock()
mock_runtime_instance.stop = AsyncMock()
monkeypatch.setattr(
app.metrics.APP_UP,
'labels',
@@ -86,7 +77,6 @@ def mock_app_env(monkeypatch):
mock_ingestor_instance = mocks['Ingestor'].return_value
mock_ingestor_instance.poll_interval = 0.01
mock_ingestor_instance.logger = MagicMock()
mock_ingestor_instance.ingestor_manager = MagicMock()
# Make async methods async mocks
mock_ingestor_instance.prepare_ingestor = AsyncMock()
@@ -134,22 +124,6 @@ async def test_main_successful_run_one_loop(mock_app_env, capsys):
mock_ingestor_instance.shutdown.assert_called_once()
mock_ingestor_instance.logger.info.assert_any_call('Main loop exit_signaled.')
# GatewayRuntime (Task 18): built once prepare_ingestor succeeds, started before
# the loop, idle-swept each iteration, stopped on shutdown.
mock_app_env['GatewayRuntime'].assert_called_once_with(
config=mock_app_env['GatewayConfig'].from_env.return_value,
data_manager=mock_ingestor_instance.ingestor_manager.data_manager,
pool=mock_ingestor_instance.ingestor_manager.pool,
logger=mock_ingestor_instance.logger,
notification_handler=mock_ingestor_instance.notification_handler,
metrics_controller=mock_ingestor_instance.metrics_controller,
metadata=mock_ingestor_instance.metadata,
)
mock_runtime_instance = mock_app_env['GatewayRuntime'].return_value
mock_runtime_instance.start.assert_awaited_once()
mock_runtime_instance.idle_sweep.assert_awaited_once()
mock_runtime_instance.stop.assert_awaited_once()
captured = capsys.readouterr()
assert 'Prometheus server started on port 9090.' in captured.out
@@ -323,16 +297,12 @@ async def test_main_prepare_ingestor_failure(mock_app_env, capsys):
mock_exit_signal = mock_app_env['mock_exit_signal']
mock_ingestor_instance.prepare_ingestor.side_effect = Exception('Preparation failed')
mock_ingestor_instance.ingestor_manager = None # prepare failed before assignment
mock_exit_signal.is_set.side_effect = [False, True]
with pytest.raises(OsExitCalledError) as excinfo:
await app.main()
assert excinfo.value.code == 0
# GatewayRuntime must not be built when prepare_ingestor never got a data_manager
mock_app_env['GatewayRuntime'].assert_not_called()
# Verify error metrics were incremented
app.metrics.APP_ERRORS_TOTAL.labels.assert_called_with(pod_id='test_pod')
mock_app_env['metrics_APP_ERRORS_TOTAL_labels_inc'].assert_called_once()
@@ -347,35 +317,3 @@ async def test_main_prepare_ingestor_failure(mock_app_env, capsys):
mock_ingestor_instance.logger.error.assert_called_once_with(
'Failed to prepare ingestor: Preparation failed'
)
@pytest.mark.asyncio
async def test_main_gateway_runtime_init_failure_keeps_read_path_alive(mock_app_env):
"""GatewayRuntime construction failing (e.g. Postgres config) must not kill the
app: the read path lives in the ingestor_manager, so the main loop still runs and
shuts down cleanly."""
mock_ingestor_instance = mock_app_env['Ingestor'].return_value
mock_exit_signal = mock_app_env['mock_exit_signal']
mock_app_env['GatewayRuntime'].side_effect = Exception('MinIO unreachable')
mock_exit_signal.is_set.side_effect = [False, True]
mock_app_env['time_time'].side_effect = [10.0, 11.0]
with pytest.raises(OsExitCalledError) as excinfo:
await app.main()
assert excinfo.value.code == 0
# Loop still ran and shutdown was clean, without any runtime calls
mock_ingestor_instance.loop.assert_called_once()
mock_ingestor_instance.shutdown.assert_called_once()
mock_runtime_instance = mock_app_env['GatewayRuntime'].return_value
mock_runtime_instance.idle_sweep.assert_not_called()
mock_runtime_instance.stop.assert_not_called()
# Failure was logged and counted, but did not set exit_signal
assert any(
'GatewayRuntime failed to initialize' in str(c)
for c in mock_ingestor_instance.logger.error.call_args_list
)
mock_app_env['metrics_APP_ERRORS_TOTAL_labels_inc'].assert_called_once()
mock_exit_signal.set.assert_not_called()

Some files were not shown because too many files have changed in this diff Show More