Compare commits
1 Commits
test-SIENT
...
2.0.0-rc3
| Author | SHA1 | Date | |
|---|---|---|---|
|
52774520e7 |
@@ -40,9 +40,7 @@ app.py (asyncio loop, signals, Prometheus server)
|
||||
-> Ingestor (ingestor.py) — lifecycle: prepare_ingestor() -> loop() -> shutdown()
|
||||
-> IngestorManager (managers/ingestor_manager.py) — central coordinator
|
||||
-> ResourceManager (managers/resource_manager.py) — Redis slot leases, heartbeats, active-ingestor registry
|
||||
-> ServerReader (managers/server_reader.py), one per OPC server — SubscriptionEngine over the shared OpcConnectionPool
|
||||
(READ_ENGINE=legacy: OpcManager (managers/opc_manager.py), the rc3 reader kept as plan B until the end of SIENTIAPDE-1974)
|
||||
-> OpcConnectionPool (connection/pool.py) + CertProvider (connection/certs.py) — pooled connections, cert from MinIO
|
||||
-> OpcManager (managers/opc_manager.py), one instance per OPC server — connection, subscriptions, datachange callbacks
|
||||
-> DataManager (managers/data_manager.py) — MongoDB persistence + optional Kafka publish
|
||||
```
|
||||
|
||||
@@ -58,9 +56,7 @@ app.py (asyncio loop, signals, Prometheus server)
|
||||
|
||||
Any change to slot/lease counts should be traced through `manage_leases`/`manage_no_slots` — that's where the rebalancing math lives, not in `ResourceManager` itself (which just does Redis primitives).
|
||||
|
||||
**ServerReader** (SIENTIAPDE-1974, default `READ_ENGINE=v2`; the legacy `OpcManager` stays as plan B with `READ_ENGINE=legacy` — slot v1 only, certificate files from the Gitea clone — until it is unused and the tests pass the same without it) reads one OPC server through a `SubscriptionEngine`: the connection comes from the shared `OpcConnectionPool` (also used by NATS writes and gRPC), which fetches the platform certificate from MinIO via `CertProvider` and uses `SecurityPolicyBasic256Sha256` — no certificate files in the repo/image. Slots are parsed by `io/slots.py` (`parse_slot`), which accepts slot v2 (`names[]`) and converts the legacy v1 (`topics`). `datachange_notification` fans each value out to every `names` entry via `DataSink`. Subscribed connections are pinned in the pool so the idle sweep never drops them. Tracks `non_receive_count` to detect silently-dead subscriptions (`check_cycles` / `check_opc_listenning`); a lost server is shut down (pool connection evicted) and re-initialized on the next loop.
|
||||
|
||||
**Writes** (SIENTIAPDE-1975): `NatsWriteConsumer` → `WriteEngine` → pool, then `DataSink.update_write_status`. Commands (C2) carry only `server_id`; `ServerConfigResolver` (`connection/servers.py`) resolves the connection (command override → live pooled connection → `connectors` → `opc-servers`). `write_status` keeps the worst outcome (`ok` < `error` < `session_bad`); the gateway is the only one that updates it (the Laborious inserts the row as `pending` and never records OPC outcomes). The NATS first connect is bounded (it runs on the main loop); reconnection after it is unbounded.
|
||||
**OpcManager** owns one OPC UA `Client` connection (asyncua), handles security policy setup (cert/key-based, `SecurityPolicyBasic256`), subscriptions per tag group, and `datachange_notification` callbacks that push into `data_queue` for `DataManager` to consume. Tracks `non_receive_count` to detect silently-dead subscriptions (`check_cycles`).
|
||||
|
||||
**Metrics** (`ingestor/metrics.py`): all Prometheus metrics defined here, always labeled by `pod_id`. Emitted via `SientiaMonitoring.emit_metric(...)` (from `sientia_do`), not the prometheus client directly — new metrics should follow that pattern for consistency with logging/notifications.
|
||||
|
||||
|
||||
150
README.md
150
README.md
@@ -56,8 +56,7 @@ Gateway for on-demand and command-driven interactions with OPC UA servers:
|
||||
|
||||
- **NATS write consumer**: consumes write commands from `opc.write` (see
|
||||
`NATS_WRITE_SUBJECT`) and executes them against the target OPC server
|
||||
(`ingestor/adapters/nats_consumer.py`, `ingestor/engines/write.py`). See
|
||||
[Write commands](#write-commands-c2-and-write_status-c3) below.
|
||||
(`ingestor/adapters/nats_consumer.py`, `ingestor/engines/write.py`).
|
||||
- **On-demand gRPC `ReadTag`**: a gRPC server (`ingestor/adapters/grpc_server.py`,
|
||||
`ingestor/proto/opc_gateway.proto`) for synchronous, single-tag reads.
|
||||
- **Scheduled active reads**: a polling read engine (`ingestor/engines/active_read.py`)
|
||||
@@ -71,37 +70,6 @@ See `docs/deploy-notes.md` for the deployment-facing details (env vars,
|
||||
health probe, NetworkPolicy, cert cache) and the `# OPC Gateway` block in
|
||||
`.env.example` for the full list of new environment variables.
|
||||
|
||||
### Write commands (C2) and `write_status` (C3)
|
||||
|
||||
The Laborious publishes one command per tag (SIENTIAPDE-1975):
|
||||
|
||||
```json
|
||||
{"server_id": "1", "addr": "ns=2;s=Pred", "value": 1.25, "data_type": "float",
|
||||
"model_id": "2", "model_name": "sin-approx", "schema": "sientia_data",
|
||||
"table_name": "predictions", "timestamp": "2026-10-13T12:00:00-03:00",
|
||||
"write_timestamp": false, "write_quality": false}
|
||||
```
|
||||
|
||||
- **Server connection**: the command carries only `server_id`. The connection is
|
||||
resolved by `ServerConfigResolver` (`ingestor/connection/servers.py`): `url`/
|
||||
`server_uri`/`use_security` in the command when present (override), else the
|
||||
live pooled connection opened by the reads, else the `connectors` document
|
||||
(`type: opc_ua`, `status: Active`), else the `opc-servers` document
|
||||
(`use_security = bool(cert_path)`) — the orchestrator's rule for the read slots.
|
||||
Lookups are cached for 60 s. An unknown server is a failed write (`server_not_found`).
|
||||
- **Outcome**: `UPDATE ... SET write_status` on the prediction row (by `timestamp` +
|
||||
`model_id`), keeping the worst outcome: `ok` < `error` < `session_bad`, and
|
||||
`pending`/NULL are replaced by any answer, so the order of the answers does not
|
||||
matter. The gateway is the only one that updates the column: the Laborious inserts the
|
||||
row as `pending` (merged with its PI Web API outcome) and never records OPC outcomes.
|
||||
The UPDATE never raises; its outcome (`updated`, `kept`, `row_not_found`,
|
||||
`invalid`, `error`) is logged and counted in `opc_write_status_update_total`.
|
||||
The reason of a failed write is logged by `WriteEngine`.
|
||||
- **NATS connection**: the first connection gives up after 5 s (it runs on the main
|
||||
loop: nats-py's defaults would block it for minutes while NATS is down); once
|
||||
connected, it reconnects forever. A connection closed for good is started again
|
||||
by `GatewayRuntime.idle_sweep`.
|
||||
|
||||
## 📋 Prerequisites
|
||||
|
||||
- Python 3.11+
|
||||
@@ -155,9 +123,9 @@ The Laborious publishes one command per tag (SIENTIAPDE-1975):
|
||||
# Ensure services are accessible on localhost with appropriate ports
|
||||
```
|
||||
|
||||
6. **Register the OPC UA client certificate** (only if connecting to a secured OPC UA server)
|
||||
6. **Generate an OPC UA client certificate** (only if connecting to a secured OPC UA server)
|
||||
|
||||
See [OPC Client Certificate](#-opc-client-certificate) below.
|
||||
See [Certificate Generation](#-certificate-generation) below.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -225,50 +193,37 @@ pytest --cov=ingestor --cov-report=html
|
||||
|
||||
The feeder creates sample OPC tag configurations in Redis that the ingestor can discover and manage.
|
||||
|
||||
## 🔐 OPC Client Certificate
|
||||
## 🔐 Certificate Generation
|
||||
|
||||
OPC UA servers that require secure connections validate the gateway with a single,
|
||||
shared platform X.509 client certificate. Every OPC connection — subscription reads,
|
||||
NATS writes and gRPC reads — uses it with `Basic256Sha256` (SIENTIAPDE-1974).
|
||||
OPC UA servers that require secure connections (e.g. KEPServer with `Basic256` policy) validate the ingestor via an X.509 client certificate. Use `scripts/generate-and-push-opc-cert.sh` to generate that certificate and push it to the Gitea repo the pod clones on boot.
|
||||
|
||||
The certificate is **never** read from the repository or the image. On connect, the
|
||||
`CertProvider` (`ingestor/connection/certs.py`):
|
||||
1. reads the single document of the Mongo collection `platform_certificates`:
|
||||
`{"client_cert": {"minio_object_name": "..."}, "private_key": {"minio_object_name": "..."}}`;
|
||||
2. downloads both objects from the MinIO bucket `CERT_BUCKET` (default
|
||||
`streamlit-connectors`) into `CERT_CACHE_DIR` (default `/tmp/opc-certs`).
|
||||
```bash
|
||||
GITEA_PASSWORD='<senha-do-gitea>' ./scripts/generate-and-push-opc-cert.sh
|
||||
```
|
||||
|
||||
Requirements for a server to accept the connection:
|
||||
- the server's slot config has `use_security: true` (a slot v1 server with a
|
||||
`cert_path` is treated the same way; the path itself is ignored);
|
||||
- the certificate's SAN URI equals the server's `server_uri` in the slot — it is sent
|
||||
as the client `application_uri` (KEPServer rejects a mismatch with
|
||||
`BadCertificateUriInvalid`);
|
||||
- the server exposes a `Basic256Sha256` endpoint and trusts the certificate (e.g.
|
||||
KEPServer: *OPC UA Configuration Manager → Trusted Clients → Trust*, then restart
|
||||
its runtime).
|
||||
Environment variables (all optional except `GITEA_PASSWORD`):
|
||||
|
||||
MinIO is only contacted when a certificate is first needed, so a MinIO outage only
|
||||
affects secured servers, which are retried on the next main-loop iteration.
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `GITEA_URL` | Gitea in-cluster URL | `https://git.sientia.ai` |
|
||||
| `GITEA_USER` | Gitea user | `gitea_admin` |
|
||||
| `GITEA_PASSWORD` | Gitea password (**required**) | — |
|
||||
| `GITEA_REPO` | Repo the ingestor pod clones | `gitea_admin/sientia-dataops-opc-ingestor` |
|
||||
| `BRANCH` | Branch to push the cert to | `main` |
|
||||
| `APP_URI` | SAN URI of the cert — **must match** the `uri` field of the corresponding document in the `OPC_servers` Mongo collection, since the ingestor uses it as `application_uri` (KEPServer rejects with `BadCertificateUriInvalid` on mismatch) | `urn:sientia:opc-ingestor` |
|
||||
| `CN` | Certificate common name | `sientia-opc-ingestor` |
|
||||
| `DAYS` | Certificate validity (days) | `3650` |
|
||||
| `FORCE` | Set to `1` to overwrite an existing cert already pushed to the repo | unset |
|
||||
|
||||
### Plan B: legacy read engine (`READ_ENGINE=legacy`)
|
||||
What the script does:
|
||||
1. Generates a self-signed RSA-2048 X.509 cert + key with `openssl`, with the SAN URI, `keyUsage`, and `extendedKeyUsage` extensions OPC UA / KEPServer require.
|
||||
2. Clones the ingestor's Gitea repo, copies the cert/key (`.pem`) and a `.der` copy (for manual import) into `certs/`, commits, and pushes.
|
||||
3. Prints the paths the pod will see after cloning (`/app/certs/opc-ingestor-cert.pem`, `/app/certs/opc-ingestor-key.pem`) and the next manual steps.
|
||||
|
||||
Until the end of SIENTIAPDE-1974 the legacy reader is kept for validation and as a
|
||||
fallback. `READ_ENGINE` selects the read path (default `v2`; an invalid value falls
|
||||
back to `v2` with a warning):
|
||||
|
||||
| `READ_ENGINE` | Reader | Slot | Certificate |
|
||||
|---|---|---|---|
|
||||
| `v2` (default) | `ServerReader` + `SubscriptionEngine` over the shared pool | v1 or v2 | MinIO + Mongo (`CertProvider`), `Basic256Sha256` |
|
||||
| `legacy` | the rc3 `OpcManager`, unchanged, with its own connection | **v1 only** (a v2 slot is rejected and notified) | files from the repository the pod clones (`cert_path` / `private_key_path` in the slot), `Basic256` |
|
||||
|
||||
Writes and gRPC always use the pool (MinIO certificate), whatever the read engine.
|
||||
The legacy engine needs the certificate files in the Gitea branch the pod clones;
|
||||
`scripts/generate-and-push-opc-cert.sh` generates them and pushes them to
|
||||
`certs/` (`GITEA_PASSWORD='<senha>' BRANCH=<branch> ./scripts/generate-and-push-opc-cert.sh`).
|
||||
Each version deploy creates a new Gitea branch without `certs/`, so they have to be
|
||||
copied again for the legacy engine. Remove the legacy engine (OpcManager, its tests
|
||||
and the script) once it is unused and the test suite passes the same without it.
|
||||
After pushing, you still need to:
|
||||
- Set `cert_path` / `private_key_path` (and `uri`) on the server's document in the `OPC_servers` collection so `OpcManager` picks them up (see [OPC Server Configuration](#opc-server-configuration)).
|
||||
- Restart the ingestor deployment so it re-clones the repo: `kubectl -n sientia rollout restart deployment/opc-ingestor`.
|
||||
- On first connection, the server rejects the new cert — trust it in the OPC UA server's console (e.g. KEPServer: *OPC UA Configuration Manager → Trusted Clients → Trust*) and restart its runtime.
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
@@ -322,9 +277,6 @@ The OPC Ingestor exposes comprehensive Prometheus metrics for monitoring:
|
||||
### OPC Gateway Metrics
|
||||
- `opc_write_total`: Active write attempts (NATS-driven), by result
|
||||
- `opc_write_latency_seconds`: Active write latency
|
||||
- `opc_write_status_by_model`: Last write outcome per model (1 = ok, 0 = failed; label `status`)
|
||||
- `opc_write_status_update_total`: `write_status` UPDATEs, by result
|
||||
(`updated`, `kept`, `row_not_found`, `invalid`, `error`)
|
||||
- `opc_active_read_total`: Scheduled/on-demand read attempts, by result
|
||||
- `opc_error_by_type_total`: OPC errors by type (connection, conversion, session, ...)
|
||||
|
||||
@@ -354,12 +306,7 @@ gateway-specific environment variables.
|
||||
|
||||
### OPC Server Configuration
|
||||
|
||||
OPC servers are configured by the Orchestrator through Redis, one key per slot
|
||||
(`slot:opc_tags:<n>`). The gateway reads the **slot v2** structure (contract C1):
|
||||
one entry per address, with one `names` item per collection / tag name / model, so
|
||||
the same address can feed several models under different names. For secured
|
||||
connections set `use_security: true` (the certificate comes from MinIO, see
|
||||
[OPC Client Certificate](#-opc-client-certificate)):
|
||||
OPC servers are configured through Redis with the following structure. `cert_path` / `private_key_path` are only required for secured connections (see [Certificate Generation](#-certificate-generation)):
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -368,30 +315,30 @@ connections set `use_security: true` (the certificate comes from MinIO, see
|
||||
"name": "server1",
|
||||
"server_id": "1",
|
||||
"url": "opc.tcp://localhost:4841",
|
||||
"server_uri": "urn:sientia:opc-ingestor",
|
||||
"use_security": true,
|
||||
"subscription_period_ms": 7500,
|
||||
"server_uri": "http://opcua-server.simulator",
|
||||
"cert_path": "/app/certs/opc-ingestor-cert.pem",
|
||||
"private_key_path": "/app/certs/opc-ingestor-key.pem",
|
||||
"tags": {
|
||||
"ns=2;i=2": {
|
||||
"frequency": "15000",
|
||||
"active_read": {"enabled": false},
|
||||
"names": [
|
||||
{"collection": "raw_scouter-opcua-orchestrated-pipeline", "tag_name": "Counter", "model_name": "m1", "model_id": "1"},
|
||||
{"collection": "raw_scouter-basic-sum-model", "tag_name": "Contador", "model_name": "m2", "model_id": "2"}
|
||||
]
|
||||
}
|
||||
"aggr_func": "avg",
|
||||
"data_range": [
|
||||
-100,
|
||||
100
|
||||
],
|
||||
"frequency": "15000",
|
||||
"server_id": "1",
|
||||
"tag_address": "ns=2;i=2",
|
||||
"tag_name": "Counter",
|
||||
"topics": [
|
||||
"raw_scouter-opcua-orchestrated-pipeline",
|
||||
"raw_scouter-basic-sum-model"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The legacy **slot v1** (each tag with `tag_name` + `topics`, the server with
|
||||
`cert_path`) is still accepted and converted on read (`ingestor/io/slots.py`): one
|
||||
`names` item per topic, and `use_security` is true when `cert_path` is set (the path
|
||||
itself is ignored). This lets the gateway and the Orchestrator be deployed in any
|
||||
order. Documents written from a v1 slot have no `model_id`, exactly as before.
|
||||
|
||||
## 🔧 Development
|
||||
|
||||
### Project Structure
|
||||
@@ -401,14 +348,13 @@ sientia-dataops-opc-ingestor/
|
||||
│ ├── managers/ # Component managers
|
||||
│ │ ├── data_manager.py # Data persistence and export
|
||||
│ │ ├── ingestor_manager.py # Main coordination
|
||||
│ │ ├── opc_manager.py # Legacy reader, plan B (READ_ENGINE=legacy)
|
||||
│ │ ├── resource_manager.py # Resource coordination
|
||||
│ │ └── server_reader.py # OPC UA server reads (SubscriptionEngine + pool)
|
||||
│ │ ├── opc_manager.py # OPC UA server management
|
||||
│ │ └── resource_manager.py # Resource coordination
|
||||
│ ├── app.py # Main application entry point
|
||||
│ ├── ingestor.py # Core ingestor logic
|
||||
│ └── metrics.py # Prometheus metrics definitions
|
||||
├── simulator/ # OPC simulation and testing tools
|
||||
├── scripts/ # Ops scripts (e2e seed; legacy OPC cert generation)
|
||||
├── scripts/ # Ops scripts (e.g. OPC UA cert generation)
|
||||
├── tests/ # Test suite
|
||||
└── requirements.txt # Python dependencies
|
||||
```
|
||||
|
||||
BIN
certs/opc-ingestor-cert.der
Normal file
BIN
certs/opc-ingestor-cert.der
Normal file
Binary file not shown.
23
certs/opc-ingestor-cert.pem
Normal file
23
certs/opc-ingestor-cert.pem
Normal file
@@ -0,0 +1,23 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIID0zCCArugAwIBAgIUW3zWE3LXrXXSwmWnqDZWlx1GIrowDQYJKoZIhvcNAQEL
|
||||
BQAwQzEdMBsGA1UEAwwUc2llbnRpYS1vcGMtaW5nZXN0b3IxEDAOBgNVBAoMB0Fp
|
||||
Z25vc2kxEDAOBgNVBAsMB1NpZW50aWEwHhcNMjYwNzE1MjAwMDI3WhcNMzYwNzEy
|
||||
MjAwMDI3WjBDMR0wGwYDVQQDDBRzaWVudGlhLW9wYy1pbmdlc3RvcjEQMA4GA1UE
|
||||
CgwHQWlnbm9zaTEQMA4GA1UECwwHU2llbnRpYTCCASIwDQYJKoZIhvcNAQEBBQAD
|
||||
ggEPADCCAQoCggEBALUoKgKJTlfpWqraCoi4f0GgAtmgQcIu+VD+7Y89PSNKx+bk
|
||||
HCoe67nEHnFHXEhtKVxprQ1/001bQjyrKjZOIjd6G2WvYfYP0FROa67P+3zzQ1sY
|
||||
/mkQBvV6LdZOy6tk9Amk8QYVbVOXmacg8+0elAxeM2swl+02W9yuTcOpTGNLA7bW
|
||||
vTZrFL9DqPcghdeA6ZlD9AuUGaYvj7C/ShR+ELTHblUQO+f4xxpEXDv6w/8UaUaW
|
||||
xZMJtWbBO4fX0DIvMtCBi28oxGD2obbTVXHbgKDzuwqPh5qP4IPJJQEg4M8z6ttf
|
||||
2/OfFOc1O8XcFRK9vji4TBJFbs5wTDE6cPogFJcCAwEAAaOBvjCBuzAdBgNVHQ4E
|
||||
FgQU64naOPKzlGirtomkUtclQOIjtMAwHwYDVR0jBBgwFoAU64naOPKzlGirtomk
|
||||
UtclQOIjtMAwOQYDVR0RBDIwMIYYdXJuOnNpZW50aWE6b3BjLWluZ2VzdG9yghRz
|
||||
aWVudGlhLW9wYy1pbmdlc3RvcjAOBgNVHQ8BAf8EBAMCBPAwIAYDVR0lAQH/BBYw
|
||||
FAYIKwYBBQUHAwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEL
|
||||
BQADggEBAGM4GXMc0EzsF2D02HOV0ZW1Igr5ya+561K0Cpg1SmMyQMFdCSk3s53I
|
||||
OrB/hcUNQ/xtg3MeVNQSGeif/dijNWCVKN8o7dGvZU0gjm7j/ue2gIj53TjONw8K
|
||||
QdF8vxvSDWgRHYFtPB3DlmWrHZaIeYD7Q60tnG1aaaM4pyRF0vH90PJL+XA9Tz4f
|
||||
9Q5+TzyqMPs87+fPpWtI5YwAhpLYfRU/nbpiTEqYUn8L97oFCyn4fjOa7cN5Wil3
|
||||
kiJgexHb0GimUiiKJeKmXDGgwZdw/xol//oB2DjhYPNcaPXTI4cqVtnTjuPwSkSS
|
||||
ZnxDPAwl0ssqzil895bYcq3AHG/jrro=
|
||||
-----END CERTIFICATE-----
|
||||
28
certs/opc-ingestor-key.pem
Normal file
28
certs/opc-ingestor-key.pem
Normal file
@@ -0,0 +1,28 @@
|
||||
-----BEGIN PRIVATE KEY-----
|
||||
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC1KCoCiU5X6Vqq
|
||||
2gqIuH9BoALZoEHCLvlQ/u2PPT0jSsfm5BwqHuu5xB5xR1xIbSlcaa0Nf9NNW0I8
|
||||
qyo2TiI3ehtlr2H2D9BUTmuuz/t880NbGP5pEAb1ei3WTsurZPQJpPEGFW1Tl5mn
|
||||
IPPtHpQMXjNrMJftNlvcrk3DqUxjSwO21r02axS/Q6j3IIXXgOmZQ/QLlBmmL4+w
|
||||
v0oUfhC0x25VEDvn+McaRFw7+sP/FGlGlsWTCbVmwTuH19AyLzLQgYtvKMRg9qG2
|
||||
01Vx24Cg87sKj4eaj+CDySUBIODPM+rbX9vznxTnNTvF3BUSvb44uEwSRW7OcEwx
|
||||
OnD6IBSXAgMBAAECggEAJUF7UkY2rHG7ScPJwtBeq3pi26rPm0XZ1ER5SqoHdsic
|
||||
T+m9uRPe2rdwo09PtT1t8ilxg4M0S4ZnGt8AsWZkNxCPrdvhw58A8zWlg7AWQl4t
|
||||
DgEX4TeAlt9nqdlOnZMHdSxMvTmO4FAfM3XddHiDbf+5h2PY+tnhDoNyTrj0kr6z
|
||||
8JCD35FuOdU1zOBmum7T4eJDK1ol7XZ+UdrPraeM/jqDKaAnutUkRaDCpxUdaR7m
|
||||
RKG9XoqC0mw6t/esPbSbA4gUDpojgle69QXx98M/EhTzE3Q4bSGsjT9G5q3/6rHB
|
||||
RWgC2dWsTJOOI2y8Eym2MFLtf0MksZ4cgBtI0zW9AQKBgQDjjnVHSXC4LZ/Lbzdd
|
||||
bAE0lzTjpoNMKLyL+spWzbnY6kYhq79XC2lb8C1aor9P9Gkq+uUSqyt0yZ0nEjB4
|
||||
baCiKHEwe9qmBqyLEc84d6rt0aYzOxzzG/UGzMyvl2lhpJmofAxjgw4vaG2IVMog
|
||||
qaQymfIPNCXIo0UdUNzDWnaTQQKBgQDLzPks/MtMcCVCReJT3LYGD4D4NDAxv0/q
|
||||
+V0fLngKP3DmeLRbvqWyTQ6bCM6SH6HfOJYLLyK39DBnf0uPbolPH/iBuRqkl7Ez
|
||||
YxcM4EJMR3U+aQH8hH24RJupr16OT3T+bZXK1pdjzM9SqwXiVI1rq/kRokfXT3hC
|
||||
GBkhZrUp1wKBgQCJv5rO5G4Pj87DXzesncy9AStLA2lHmQFJ2y7RDQgLfwWlLEpP
|
||||
buEAfot16j7VL6baFbbFbkiIEePH2wLsY7BxePnL162sAy4/oTBHhmk6PUbWjFoh
|
||||
8DKLQz5ErOtl6DX7/9ugAZ18pUJo3KreytJSr04QVgjpQ6x66pA40XImAQKBgHLF
|
||||
4J5BT4ygq5sDGE76QVhwasVdPrcFctB11se0t82tpLtmxsvUNiSP2OXes9b/EX5Q
|
||||
s89goEDQuGHRApCoQ5PYAW+s+yAufZDfASAzO6MRqymWoZPAWbzohnGtXabvgIe4
|
||||
118AMwn1UdzQi+PGkk5M71vK9IKBMsymHaD94pDJAoGBANAbRSmWgJfZNsYiVV7u
|
||||
kQ6L5Y5aN2/LfHlqNbWFYot00olsUWPz2WUEvKXI2e3L6lmkGJFNREOlAsOlFwSH
|
||||
ZBOSX+9Fq6vg8CUHi8obDgWAnIelTCCzHWC3Ep/ATLZQct82wgYZxIYTt4yCtMQo
|
||||
R5c2DqKOYl8ZAjr3Zah5uwFI
|
||||
-----END PRIVATE KEY-----
|
||||
@@ -3,14 +3,12 @@
|
||||
Thin adapter: decode the message JSON and hand the dict to WriteEngine,
|
||||
which is already raise-safe against malformed commands. No metrics of its
|
||||
own — the engine records write metrics; a malformed-JSON drop is logged
|
||||
here. Once connected, nats-py reconnects by itself, forever.
|
||||
here. No reconnect logic — nats-py handles reconnection internally.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import math
|
||||
import traceback
|
||||
|
||||
import nats
|
||||
from nats.aio.client import Client
|
||||
from nats.aio.subscription import Subscription
|
||||
from sientia_do.observability.logger import Logger
|
||||
@@ -26,55 +24,18 @@ class NatsWriteConsumer:
|
||||
logger: Logger,
|
||||
subject: str = 'opc.write',
|
||||
queue: str = 'gateway-writers',
|
||||
connect_timeout_s: float = 5.0,
|
||||
):
|
||||
self.nats_url = nats_url
|
||||
self.write_engine = write_engine
|
||||
self.logger = logger
|
||||
self.subject = subject
|
||||
self.queue = queue
|
||||
self.connect_timeout_s = connect_timeout_s
|
||||
self.nc: Client | None = None
|
||||
self.sub: Subscription | None = None
|
||||
|
||||
@property
|
||||
def closed(self) -> bool:
|
||||
"""True when there is no connection, or it was closed for good."""
|
||||
return self.nc is None or self.nc.is_closed
|
||||
|
||||
async def start(self) -> None:
|
||||
"""Connect and subscribe, giving up after ``connect_timeout_s``.
|
||||
|
||||
It runs from the main loop (startup and ``GatewayRuntime.idle_sweep``), so it
|
||||
must never block it: nats-py's defaults retry the first connection 60 times,
|
||||
about 2 minutes, longer than the heartbeat TTL, and with
|
||||
``max_reconnect_attempts=-1`` they would retry it forever. Once connected,
|
||||
reconnection is unbounded, so an outage never closes the connection for good.
|
||||
|
||||
Raises:
|
||||
Exception: When the connection or the subscription failed; nothing is left
|
||||
open in that case.
|
||||
"""
|
||||
nc = Client()
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
nc.connect(
|
||||
servers=self.nats_url,
|
||||
connect_timeout=math.ceil(self.connect_timeout_s),
|
||||
max_reconnect_attempts=-1,
|
||||
name='opc-gateway',
|
||||
# One log line per event instead of nats-py's traceback per attempt.
|
||||
error_cb=self._on_error,
|
||||
disconnected_cb=self._on_disconnected,
|
||||
reconnected_cb=self._on_reconnected,
|
||||
),
|
||||
timeout=self.connect_timeout_s,
|
||||
)
|
||||
self.sub = await nc.subscribe(self.subject, queue=self.queue, cb=self._handle)
|
||||
except (Exception, asyncio.CancelledError):
|
||||
await self._discard(nc)
|
||||
raise
|
||||
self.nc = nc
|
||||
self.nc = await nats.connect(servers=self.nats_url)
|
||||
self.sub = await self.nc.subscribe(self.subject, queue=self.queue, cb=self._handle)
|
||||
self.logger.info(f'NatsWriteConsumer started: subject={self.subject} queue={self.queue}')
|
||||
|
||||
async def _handle(self, msg) -> None:
|
||||
@@ -84,41 +45,11 @@ class NatsWriteConsumer:
|
||||
preview = msg.data[:200]
|
||||
self.logger.error(f'NatsWriteConsumer dropped malformed JSON ({e}): {preview!r}')
|
||||
return
|
||||
try:
|
||||
await self.write_engine.write(payload)
|
||||
except Exception as e:
|
||||
# The engine is raise-safe; this only keeps an unexpected bug from going to
|
||||
# nats-py's error callback without context.
|
||||
self.logger.error(
|
||||
f'NatsWriteConsumer: unexpected error handling a write command ({e}):\n'
|
||||
f'{traceback.format_exc()}'
|
||||
)
|
||||
|
||||
# These stay async with nothing to await: nats-py rejects callbacks that are not
|
||||
# coroutine functions (InvalidCallbackTypeError on connect).
|
||||
async def _on_error(self, e: Exception) -> None:
|
||||
self.logger.warning(f'NATS error: {type(e).__name__} {e}')
|
||||
|
||||
async def _on_disconnected(self) -> None:
|
||||
self.logger.warning('Disconnected from NATS; reconnecting')
|
||||
|
||||
async def _on_reconnected(self) -> None:
|
||||
self.logger.info('Reconnected to NATS')
|
||||
|
||||
async def _discard(self, nc: Client) -> None:
|
||||
try:
|
||||
await nc.close()
|
||||
except Exception as e:
|
||||
self.logger.warning(f'Error closing a failed NATS connection: {e}')
|
||||
await self.write_engine.write(payload)
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self.nc is None:
|
||||
return
|
||||
if self.nc.is_closed:
|
||||
# Closed for good already: unsubscribe/drain would raise ConnectionClosedError
|
||||
# and abort the shutdown sequence.
|
||||
self.nc = None
|
||||
return
|
||||
if self.sub is not None:
|
||||
await self.sub.unsubscribe()
|
||||
await self.nc.drain()
|
||||
|
||||
@@ -58,17 +58,16 @@ async def main(): # NOSONAR
|
||||
ingestor.logger.info('Ingestor prepared. Starting main loop.')
|
||||
|
||||
# Gateway write/active-read/gRPC wiring (Task 18). Only built if prepare_ingestor
|
||||
# got far enough to have an ingestor_manager (its data_manager and connection
|
||||
# pool are shared) -- skipped entirely if prepare failed before that point. The
|
||||
# read path (subscriptions) lives in the ingestor_manager and is already running:
|
||||
# a failure here only disables writes/gRPC, the app continues without the runtime.
|
||||
# got far enough to have a data_manager (needed by CertProvider/DataSink) --
|
||||
# skipped entirely if prepare failed before that point. Construction can hit
|
||||
# the network (MinIO bucket check, Postgres engine): a failure there must not
|
||||
# kill the read path, so the app continues without the gateway runtime.
|
||||
runtime: GatewayRuntime | None = None
|
||||
if ingestor.ingestor_manager is not None:
|
||||
try:
|
||||
runtime = GatewayRuntime(
|
||||
config=GatewayConfig.from_env(),
|
||||
data_manager=ingestor.ingestor_manager.data_manager,
|
||||
pool=ingestor.ingestor_manager.pool,
|
||||
logger=ingestor.logger,
|
||||
notification_handler=ingestor.notification_handler,
|
||||
metrics_controller=ingestor.metrics_controller,
|
||||
|
||||
@@ -22,25 +22,13 @@ class GatewayConfig:
|
||||
nats_queue: str = 'gateway-writers'
|
||||
grpc_enabled: bool = False
|
||||
grpc_port: int = 50051
|
||||
# Legacy frontend bucket the platform already provisions (SIENTIAPDE-1974 D7);
|
||||
# the design's `sientia-dataops` bucket does not exist in the platform MinIO.
|
||||
cert_bucket: str = 'streamlit-connectors'
|
||||
cert_bucket: str = 'sientia-dataops'
|
||||
cert_cache_dir: str = '/tmp/opc-certs' # noqa: S108 # nosec B108 - default is spec'd, overridable via env
|
||||
write_timestamp: bool = True
|
||||
write_quality: bool = True
|
||||
write_min_pause_s: float = 0.0
|
||||
idle_conn_max_s: float = 300.0
|
||||
|
||||
@classmethod
|
||||
def certs_from_env(cls) -> 'GatewayConfig':
|
||||
"""Only the certificate settings — what the read path needs. Plain strings,
|
||||
no number parsing, so a malformed write/gRPC env var (GRPC_PORT,
|
||||
WRITE_MIN_PAUSE_S, ...) can only disable the runtime, never the reads."""
|
||||
return cls(
|
||||
cert_bucket=getenv('CERT_BUCKET', cls.cert_bucket),
|
||||
cert_cache_dir=getenv('CERT_CACHE_DIR', cls.cert_cache_dir),
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> 'GatewayConfig':
|
||||
return cls(
|
||||
|
||||
@@ -1,29 +1,12 @@
|
||||
import asyncio
|
||||
import os
|
||||
from os import getenv
|
||||
from pathlib import Path
|
||||
|
||||
from sientia_do.repository.minio_repository import MinioRepository
|
||||
|
||||
from ingestor.config import _env_bool
|
||||
|
||||
COLLECTION_NAME = 'platform_certificates'
|
||||
|
||||
|
||||
class CertificateNotConfiguredError(RuntimeError):
|
||||
"""No usable platform OPC client certificate is registered (Mongo + MinIO)."""
|
||||
|
||||
|
||||
class CertProvider:
|
||||
"""Downloads the platform's client cert + private key (Mongo metadata,
|
||||
MinIO bytes) into a writable cache dir — pod root FS is read-only (§12),
|
||||
so `cache_dir` must be an emptyDir/tmpfs mount.
|
||||
|
||||
The certificate is single and shared by every connection (reads, writes,
|
||||
gRPC), so it is downloaded once and cached; `invalidate()` forces a fresh
|
||||
download (the pool calls it when a secured connection fails, e.g. after the
|
||||
certificate was replaced in MinIO). The lock serializes downloads so two
|
||||
connections never rewrite the cache files at the same time."""
|
||||
so `cache_dir` must be an emptyDir/tmpfs mount."""
|
||||
|
||||
def __init__(self, mongo, minio, cache_dir: str, bucket: str, metadata: dict | None = None):
|
||||
self.mongo = mongo
|
||||
@@ -31,84 +14,16 @@ class CertProvider:
|
||||
self.cache_dir = cache_dir
|
||||
self.bucket = bucket
|
||||
self.metadata = metadata or {}
|
||||
self._lock = asyncio.Lock()
|
||||
self._paths: tuple[str, str] | None = None
|
||||
|
||||
async def fetch(self) -> tuple[str, str]:
|
||||
async with self._lock:
|
||||
if self._paths is None:
|
||||
self._paths = await self._download_all()
|
||||
return self._paths
|
||||
|
||||
def invalidate(self) -> None:
|
||||
self._paths = None
|
||||
|
||||
async def _download_all(self) -> tuple[str, str]:
|
||||
doc = await self.mongo.find_one(COLLECTION_NAME, {}, self.metadata)
|
||||
try:
|
||||
cert_object = doc['client_cert']['minio_object_name']
|
||||
key_object = doc['private_key']['minio_object_name']
|
||||
except (KeyError, TypeError) as e:
|
||||
raise CertificateNotConfiguredError(
|
||||
f'No OPC client certificate configured: the Mongo collection '
|
||||
f"'{COLLECTION_NAME}' needs a document with client_cert.minio_object_name "
|
||||
f'and private_key.minio_object_name (objects in the MinIO bucket '
|
||||
f"'{self.bucket}'). Register the platform certificate before using "
|
||||
f'secured OPC servers.'
|
||||
) from e
|
||||
Path(self.cache_dir).mkdir(parents=True, exist_ok=True)
|
||||
cert_path = await self._download(cert_object)
|
||||
key_path = await self._download(key_object)
|
||||
cert_path = await self._download(doc['client_cert']['minio_object_name'])
|
||||
key_path = await self._download(doc['private_key']['minio_object_name'])
|
||||
return cert_path, key_path
|
||||
|
||||
async def _download(self, object_name: str) -> str:
|
||||
data = await self.minio.download_file(object_name, self.bucket, self.metadata)
|
||||
local_path = Path(self.cache_dir) / Path(object_name).name
|
||||
# Atomic replace: a connection loading the previous file never sees it half-written.
|
||||
tmp_path = local_path.with_name(local_path.name + '.tmp')
|
||||
tmp_path.write_bytes(data)
|
||||
os.replace(tmp_path, local_path)
|
||||
return str(local_path)
|
||||
|
||||
|
||||
class LazyMinio:
|
||||
"""Builds the MinIO client on first use. MinioRepository's constructor hits the
|
||||
network (bucket check), so building it eagerly would let a MinIO outage at pod
|
||||
start take the whole read path down. Built lazily, only the servers that need a
|
||||
certificate fail — and they are retried on the next loop."""
|
||||
|
||||
def __init__(self, factory):
|
||||
self._factory = factory
|
||||
self._client = None
|
||||
|
||||
async def download_file(self, object_name: str, bucket: str, metadata: dict):
|
||||
if self._client is None:
|
||||
self._client = self._factory()
|
||||
return await self._client.download_file(object_name, bucket, metadata)
|
||||
|
||||
|
||||
def build_cert_provider(
|
||||
mongo, config, logger, notification_handler, metrics_controller, metadata=None
|
||||
) -> CertProvider:
|
||||
"""CertProvider shared by the read path (IngestorManager) and the write path
|
||||
(GatewayRuntime): one certificate source — MinIO + Mongo — for every connection."""
|
||||
|
||||
def minio_factory() -> MinioRepository:
|
||||
return MinioRepository(
|
||||
endpoint=getenv('MINIO_ENDPOINT', 'localhost:9000'),
|
||||
access_key=getenv('MINIO_ACCESS_KEY', 'minioadmin'),
|
||||
secret_key=getenv('MINIO_SECRET_KEY', 'minioadmin'),
|
||||
secure=_env_bool('MINIO_SECURE', True),
|
||||
bucket=config.cert_bucket,
|
||||
logger=logger,
|
||||
notification_handler=notification_handler,
|
||||
metrics_controller=metrics_controller,
|
||||
)
|
||||
|
||||
return CertProvider(
|
||||
mongo=mongo,
|
||||
minio=LazyMinio(minio_factory),
|
||||
cache_dir=config.cert_cache_dir,
|
||||
bucket=config.cert_bucket,
|
||||
metadata=metadata,
|
||||
)
|
||||
local_path = str(Path(self.cache_dir) / Path(object_name).name)
|
||||
Path(local_path).write_bytes(data)
|
||||
return local_path
|
||||
|
||||
@@ -26,12 +26,6 @@ class OpcConnectionPool(SientiaMonitoring):
|
||||
self.cert_provider = cert_provider
|
||||
self.clock = MonotonicClock()
|
||||
self._connections: dict[str, tuple[Client, float]] = {}
|
||||
# Active subscriptions per server. A subscribed connection only receives
|
||||
# callbacks and never calls get(), so without a pin release_idle() would
|
||||
# evict it as idle and silently stop the reads (SIENTIAPDE-1974 G4).
|
||||
self._pins: dict[str, int] = {}
|
||||
# Connection settings each pooled client was opened with (see _check_cfg).
|
||||
self._configs: dict[str, dict] = {}
|
||||
# ponytail: one Lock per server_id, never cleaned up — server set is
|
||||
# small and bounded. Add eviction of stale locks only if it grows.
|
||||
self._locks: dict[str, asyncio.Lock] = {}
|
||||
@@ -47,45 +41,16 @@ class OpcConnectionPool(SientiaMonitoring):
|
||||
"""Return the cached client for `server_id`, connecting one if needed.
|
||||
|
||||
Double-checked under a per-server lock so concurrent callers for the
|
||||
same new server share one connection (no leaked, untracked sessions).
|
||||
`server_id` is normalized to str, so 1 and '1' share the same session."""
|
||||
server_id = str(server_id)
|
||||
same new server share one connection (no leaked, untracked sessions)."""
|
||||
client = self._cached(server_id)
|
||||
if client is not None:
|
||||
self._check_cfg(server_id, server_cfg)
|
||||
return client
|
||||
async with self._locks.setdefault(server_id, asyncio.Lock()):
|
||||
client = self._cached(server_id) # re-check: a racer may have connected
|
||||
if client is not None:
|
||||
self._check_cfg(server_id, server_cfg)
|
||||
return client
|
||||
return await self._open(server_id, server_cfg)
|
||||
|
||||
def _check_cfg(self, server_id: str, server_cfg: dict) -> None:
|
||||
"""Warn when a caller asks for a server with other connection settings than
|
||||
the pooled client was opened with. The client is reused as is: reconnecting
|
||||
on every mismatch would let one malformed write command drop the reads.
|
||||
Config changes from the slot evict the client first (ServerReader.shutdown)."""
|
||||
opened = self._configs.get(server_id, {})
|
||||
diff = [
|
||||
k for k in ('url', 'server_uri', 'use_security') if opened.get(k) != server_cfg.get(k)
|
||||
]
|
||||
if diff:
|
||||
self.logger.warning(
|
||||
f'OPC server {server_id}: reusing the pooled connection, but {diff} differ '
|
||||
f'from the settings it was opened with'
|
||||
)
|
||||
|
||||
def config_for(self, server_id: str) -> dict | None:
|
||||
"""Connection settings of the pooled client for `server_id`, or None when the
|
||||
server has no live pooled connection. Lets a writer reuse exactly the settings
|
||||
the read path opened the connection with."""
|
||||
server_id = str(server_id)
|
||||
if server_id not in self._connections:
|
||||
return None
|
||||
cfg = self._configs.get(server_id)
|
||||
return dict(cfg) if cfg is not None else None
|
||||
|
||||
def _cached(self, server_id: str) -> Client | None:
|
||||
entry = self._connections.get(server_id)
|
||||
if entry is None:
|
||||
@@ -98,10 +63,6 @@ class OpcConnectionPool(SientiaMonitoring):
|
||||
try:
|
||||
client = await self._connect(server_cfg)
|
||||
except Exception:
|
||||
if server_cfg.get('use_security'):
|
||||
# The cached certificate may be the cause (replaced in MinIO,
|
||||
# rejected by the server): download it again on the next attempt.
|
||||
self.cert_provider.invalidate()
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_CONNECTIONS_FAILED,
|
||||
tags={'pod_id': self.pod_id, 'server_name': server_id},
|
||||
@@ -109,23 +70,13 @@ class OpcConnectionPool(SientiaMonitoring):
|
||||
raise
|
||||
|
||||
self._connections[server_id] = (client, self.clock.elapsed())
|
||||
self._configs[server_id] = dict(server_cfg)
|
||||
await self._record_connected(server_id, server_cfg['url'])
|
||||
return client
|
||||
|
||||
async def _connect(self, server_cfg: dict) -> Client:
|
||||
client = Client(server_cfg['url'], timeout=10)
|
||||
# Identify the pod on the OPC server, as the legacy OpcManager did: asyncua
|
||||
# sends `name` as ApplicationName and "<description> Session<n>" as the
|
||||
# session name, so each pod's session is recognizable in the server console.
|
||||
client.name = self.pod_id
|
||||
client.description = self.pod_id
|
||||
pod_uri = self.pod_id.replace('-', ':')
|
||||
client.product_uri = pod_uri
|
||||
client.application_uri = pod_uri
|
||||
server_uri = server_cfg.get('server_uri')
|
||||
if server_uri:
|
||||
# Must match the certificate's SAN URI (KEPServer: BadCertificateUriInvalid).
|
||||
client.application_uri = server_uri
|
||||
if server_cfg.get('use_security'):
|
||||
cert, key = await self.cert_provider.fetch()
|
||||
@@ -145,39 +96,15 @@ class OpcConnectionPool(SientiaMonitoring):
|
||||
tags={'pod_id': self.pod_id, 'server_name': server_id, 'server_url': url},
|
||||
)
|
||||
|
||||
def pin(self, server_id: str) -> None:
|
||||
"""Mark one more active subscription on `server_id` (never evicted as idle)."""
|
||||
server_id = str(server_id)
|
||||
self._pins[server_id] = self._pins.get(server_id, 0) + 1
|
||||
|
||||
def unpin(self, server_id: str) -> None:
|
||||
"""Drop one active subscription mark from `server_id`."""
|
||||
server_id = str(server_id)
|
||||
count = self._pins.get(server_id, 0) - 1
|
||||
if count > 0:
|
||||
self._pins[server_id] = count
|
||||
else:
|
||||
self._pins.pop(server_id, None)
|
||||
|
||||
async def release_idle(self, max_idle_s: float):
|
||||
"""Disconnect and drop clients unused for longer than `max_idle_s`,
|
||||
except the ones pinned by an active subscription."""
|
||||
"""Disconnect and drop clients unused for longer than `max_idle_s`."""
|
||||
now = self.clock.elapsed()
|
||||
idle_ids = [
|
||||
sid
|
||||
for sid, (_, last_used) in self._connections.items()
|
||||
if now - last_used > max_idle_s and not self._pins.get(sid)
|
||||
sid for sid, (_, last_used) in self._connections.items() if now - last_used > max_idle_s
|
||||
]
|
||||
for server_id in idle_ids:
|
||||
await self._evict(server_id)
|
||||
|
||||
async def evict(self, server_id: str):
|
||||
"""Disconnect `server_id` now (lost server or changed config); the next
|
||||
get() reconnects. No-op when the server has no pooled client."""
|
||||
server_id = str(server_id)
|
||||
if server_id in self._connections:
|
||||
await self._evict(server_id)
|
||||
|
||||
async def close_all(self):
|
||||
"""Disconnect every pooled client (e.g. on shutdown)."""
|
||||
# list() snapshots the keys: _evict() pops from self._connections,
|
||||
@@ -187,7 +114,6 @@ class OpcConnectionPool(SientiaMonitoring):
|
||||
|
||||
async def _evict(self, server_id: str):
|
||||
client, _ = self._connections.pop(server_id)
|
||||
self._configs.pop(server_id, None)
|
||||
await self._disconnect_with_retries(client, server_id)
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_DISCONNECTIONS_TOTAL,
|
||||
@@ -196,7 +122,7 @@ class OpcConnectionPool(SientiaMonitoring):
|
||||
|
||||
async def _disconnect_with_retries(self, client: Client, server_id: str) -> None:
|
||||
"""Tries 5 times to disconnect, with a delay of 100ms x try (mirrors
|
||||
the legacy OpcManager.disconnection_fallback)."""
|
||||
OpcManager.disconnection_fallback)."""
|
||||
for i in range(5):
|
||||
try:
|
||||
await client.disconnect()
|
||||
|
||||
@@ -1,120 +0,0 @@
|
||||
"""Connection settings of the OPC server a write command targets (SIENTIAPDE-1975 GW1).
|
||||
|
||||
The Laborious only knows the `server_id` of a write tag (pipeline `write_tags`); the
|
||||
connection (url, server URI, security) belongs to the gateway, which owns every OPC
|
||||
session. It is resolved with the same rule the orchestrator uses to build the read slots
|
||||
(SIENTIAPDE-1974): the tag database `connectors` first, the legacy `opc-servers` otherwise.
|
||||
"""
|
||||
|
||||
from typing import Any
|
||||
|
||||
from sientia_do.observability.logger import Logger
|
||||
|
||||
from ingestor.connection.clock import MonotonicClock
|
||||
from ingestor.connection.pool import OpcConnectionPool
|
||||
|
||||
CONNECTORS = 'connectors'
|
||||
OPC_SERVERS = 'opc-servers'
|
||||
|
||||
|
||||
def _id_forms(server_id: str) -> list[Any]:
|
||||
"""`server_id` as stored: the tag database keeps ints, opc-servers strings or ints."""
|
||||
forms: list[Any] = [server_id]
|
||||
if server_id.isdigit():
|
||||
forms.append(int(server_id))
|
||||
return forms
|
||||
|
||||
|
||||
def _connector_cfg(connector: dict[str, Any]) -> dict[str, Any] | None:
|
||||
if not connector.get('url'):
|
||||
return None
|
||||
return {
|
||||
'url': connector['url'],
|
||||
'server_uri': connector.get('server_uri'),
|
||||
'use_security': bool(connector.get('use_security', False)),
|
||||
}
|
||||
|
||||
|
||||
def _opc_server_cfg(server: dict[str, Any]) -> dict[str, Any] | None:
|
||||
if not server.get('url'):
|
||||
return None
|
||||
return {
|
||||
'url': server['url'],
|
||||
'server_uri': server.get('uri'),
|
||||
# The certificate comes from MinIO now: a cert_path only says the server uses
|
||||
# security, as in the orchestrator's slot (SIENTIAPDE-1974 RF4).
|
||||
'use_security': bool(server.get('cert_path')),
|
||||
}
|
||||
|
||||
|
||||
class ServerConfigResolver:
|
||||
"""Resolve and cache the connection settings of an OPC server by `server_id`.
|
||||
|
||||
Order: settings carried by the command itself (an explicit override; the C2
|
||||
command of the Laborious carries none), the live pooled connection (what the read
|
||||
path opened, so writes never fight the reads over settings), then MongoDB. Lookups
|
||||
are cached for `ttl_s`; a server that is not found, for `missing_ttl_s`, so a
|
||||
pipeline pointing at a removed server does not query MongoDB on every command."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
pool: OpcConnectionPool,
|
||||
mongo,
|
||||
logger: Logger,
|
||||
ttl_s: float = 60.0,
|
||||
missing_ttl_s: float = 10.0,
|
||||
metadata: dict | None = None,
|
||||
):
|
||||
self.pool = pool
|
||||
self.mongo = mongo
|
||||
self.logger = logger
|
||||
self.ttl_s = ttl_s
|
||||
self.missing_ttl_s = missing_ttl_s
|
||||
self.metadata = metadata or {}
|
||||
self.clock = MonotonicClock()
|
||||
self._cache: dict[str, tuple[dict[str, Any] | None, float]] = {}
|
||||
|
||||
async def resolve(self, server_id: str, cmd: dict) -> dict[str, Any] | None:
|
||||
"""Connection settings for `server_id`, or None when no source knows it.
|
||||
|
||||
Raises:
|
||||
Exception: When MongoDB cannot be read (the caller reports the write as
|
||||
failed).
|
||||
"""
|
||||
if cmd.get('url'):
|
||||
return {
|
||||
'url': cmd['url'],
|
||||
'server_uri': cmd.get('server_uri'),
|
||||
'use_security': cmd.get('use_security', False),
|
||||
}
|
||||
server_id = str(server_id)
|
||||
pooled = self.pool.config_for(server_id)
|
||||
if pooled is not None:
|
||||
return pooled
|
||||
now = self.clock.elapsed()
|
||||
cached = self._cache.get(server_id)
|
||||
if cached is not None:
|
||||
cfg, cached_at = cached
|
||||
if now - cached_at < (self.ttl_s if cfg is not None else self.missing_ttl_s):
|
||||
return dict(cfg) if cfg is not None else None
|
||||
cfg = await self._lookup(server_id)
|
||||
self._cache[server_id] = (cfg, now)
|
||||
return dict(cfg) if cfg is not None else None
|
||||
|
||||
async def _lookup(self, server_id: str) -> dict[str, Any] | None:
|
||||
ids = {'$in': _id_forms(server_id)}
|
||||
connector = await self.mongo.find_one(
|
||||
CONNECTORS,
|
||||
{'connector_id': ids, 'type': 'opc_ua', 'status': 'Active'},
|
||||
self.metadata,
|
||||
)
|
||||
cfg = _connector_cfg(connector) if connector else None
|
||||
if cfg is not None:
|
||||
return cfg
|
||||
server = await self.mongo.find_one(OPC_SERVERS, {'id': ids}, self.metadata)
|
||||
cfg = _opc_server_cfg(server) if server else None
|
||||
if cfg is None:
|
||||
self.logger.warning(
|
||||
f'OPC server {server_id} not found in {CONNECTORS} nor {OPC_SERVERS}'
|
||||
)
|
||||
return cfg
|
||||
@@ -35,9 +35,6 @@ class SubscriptionEngine(SientiaMonitoring):
|
||||
self.subscriptions: dict = {}
|
||||
self._clients: dict = {}
|
||||
self._servers: dict = {}
|
||||
# Addresses each subscription monitors, so unsubscribe can forget them
|
||||
# (otherwise check_cycles keeps alarming on tags nobody reads anymore).
|
||||
self._sub_nodes: dict[str, set[str]] = {}
|
||||
self.non_receive_count = 0
|
||||
|
||||
SientiaMonitoring.__init__(
|
||||
@@ -60,7 +57,6 @@ class SubscriptionEngine(SientiaMonitoring):
|
||||
raise
|
||||
self._clients[name] = client
|
||||
self._servers[name] = server_id
|
||||
self.pool.pin(server_id)
|
||||
self.logger.info(f'Subscription {name} created on {server_id}.')
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_SUBSCRIPTIONS_CREATED,
|
||||
@@ -78,7 +74,6 @@ class SubscriptionEngine(SientiaMonitoring):
|
||||
server_id = self._servers[subscription]
|
||||
addr_nodes = [client.get_node(n) for n in nodes]
|
||||
self.nodes.update(nodes)
|
||||
self._sub_nodes.setdefault(subscription, set()).update(nodes)
|
||||
self._init_cycle_rules(nodes, collect_period, server_id)
|
||||
|
||||
await self.subscriptions[subscription].subscribe_data_change(addr_nodes)
|
||||
@@ -101,35 +96,11 @@ class SubscriptionEngine(SientiaMonitoring):
|
||||
if not self.subscriptions.get(subscription):
|
||||
self.logger.warning(f"Subscription '{subscription}' not found. Cannot unsubscribe.")
|
||||
return
|
||||
sub = self.subscriptions.pop(subscription)
|
||||
await self.subscriptions[subscription].delete()
|
||||
del self.subscriptions[subscription]
|
||||
self._clients.pop(subscription, None)
|
||||
server_id = self._servers.pop(subscription, None)
|
||||
if server_id is not None:
|
||||
self.pool.unpin(server_id)
|
||||
self._forget_nodes(subscription)
|
||||
try:
|
||||
await sub.delete()
|
||||
except Exception as e:
|
||||
# The connection may already be gone (lost server); the subscription is
|
||||
# dropped from the books either way so the slot can be re-subscribed.
|
||||
self.logger.warning(f'Failed to delete subscription {subscription}: {e}')
|
||||
self._servers.pop(subscription, None)
|
||||
self.logger.info(f'Unsubscribed from {subscription}.')
|
||||
if server_id is not None:
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
|
||||
method='set',
|
||||
value=len(self.nodes),
|
||||
tags={'pod_id': self.pod_id, 'server_name': server_id},
|
||||
)
|
||||
|
||||
def _forget_nodes(self, subscription: str) -> None:
|
||||
"""Drop the subscription's addresses, except those another subscription
|
||||
of this engine still monitors."""
|
||||
nodes = self._sub_nodes.pop(subscription, set())
|
||||
still_monitored = set().union(*self._sub_nodes.values())
|
||||
for node in nodes:
|
||||
if node not in still_monitored:
|
||||
self.nodes.pop(node, None)
|
||||
|
||||
async def datachange_notification(self, node, _val, data):
|
||||
"""asyncua subscription callback: fan the new value out to every name
|
||||
|
||||
@@ -18,7 +18,6 @@ from sientia_do.observability.sientia_monitoring import SientiaMonitoring
|
||||
import ingestor.metrics as metrics
|
||||
from ingestor.connection.clock import MonotonicClock
|
||||
from ingestor.connection.pool import OpcConnectionPool
|
||||
from ingestor.connection.servers import ServerConfigResolver
|
||||
from ingestor.converters import ConversionError, to_typed_value
|
||||
from ingestor.io.data_sink import DataSink
|
||||
|
||||
@@ -78,13 +77,9 @@ class WriteEngine(SientiaMonitoring):
|
||||
write_timestamp: bool = True,
|
||||
write_quality: bool = True,
|
||||
min_pause_s: float = 0.0,
|
||||
servers: ServerConfigResolver | None = None,
|
||||
):
|
||||
self.pool = pool
|
||||
self.sink = sink
|
||||
# Resolves the server connection by server_id (SIENTIAPDE-1975 GW1). Without it,
|
||||
# only the connection carried by the command itself is used.
|
||||
self.servers = servers
|
||||
self.write_timestamp = write_timestamp
|
||||
self.write_quality = write_quality
|
||||
self.min_pause_s = min_pause_s
|
||||
@@ -110,24 +105,11 @@ class WriteEngine(SientiaMonitoring):
|
||||
except ConversionError as e:
|
||||
return await self._emit_error(cmd, 'conversion', str(e), 'error')
|
||||
|
||||
try:
|
||||
dv = self._build_datavalue(cmd, typed)
|
||||
except (KeyError, TypeError, ValueError) as e:
|
||||
# write_timestamp on with a missing or non-ISO timestamp.
|
||||
return await self._emit_error(cmd, 'bad_timestamp', repr(e), 'error')
|
||||
|
||||
try:
|
||||
server_cfg = await self._server_cfg(cmd)
|
||||
except Exception as e:
|
||||
return await self._emit_error(cmd, 'server_lookup', str(e), 'error')
|
||||
if server_cfg is None:
|
||||
return await self._emit_error(
|
||||
cmd, 'server_not_found', f'OPC server {cmd["server_id"]} is not configured', 'error'
|
||||
)
|
||||
dv = self._build_datavalue(cmd, typed)
|
||||
|
||||
start = self.clock.elapsed()
|
||||
try:
|
||||
client = await self.pool.get(cmd['server_id'], server_cfg)
|
||||
client = await self.pool.get(cmd['server_id'], self._server_cfg(cmd))
|
||||
node = client.get_node(cmd['addr'])
|
||||
await node.write_value(dv)
|
||||
except Exception as e:
|
||||
@@ -145,14 +127,10 @@ class WriteEngine(SientiaMonitoring):
|
||||
now = self.clock.elapsed()
|
||||
self._last_write_at = now
|
||||
|
||||
async def _server_cfg(self, cmd: dict) -> dict | None:
|
||||
"""Connection settings of the target server: resolved by server_id (connection
|
||||
carried by the command, live pooled connection, connectors, opc-servers), or only
|
||||
the ones carried by the command when no resolver is wired."""
|
||||
if self.servers is not None:
|
||||
return await self.servers.resolve(str(cmd['server_id']), cmd)
|
||||
if not cmd.get('url'):
|
||||
return None
|
||||
def _server_cfg(self, cmd: dict) -> dict:
|
||||
# ponytail: minimal cfg built straight from the C2 message. Resolving
|
||||
# the full connector config (url/security) from the connectors DB by
|
||||
# server_id is F4 wiring, not this task — pool is mocked in tests.
|
||||
return {
|
||||
'url': cmd.get('url'),
|
||||
'use_security': cmd.get('use_security', False),
|
||||
@@ -178,12 +156,6 @@ class WriteEngine(SientiaMonitoring):
|
||||
return isinstance(exc, UaStatusCodeError) and type(exc).__name__ in _SESSION_BAD_NAMES
|
||||
|
||||
async def _emit_error(self, cmd: dict, error_type: str, message: str, status: str):
|
||||
# The reason only lives here: the row gets write_status alone (SIENTIAPDE-1975).
|
||||
self.logger.error(
|
||||
f'OPC write failed ({status}, {error_type}): server={cmd.get("server_id")} '
|
||||
f'addr={cmd.get("addr")} model_id={cmd.get("model_id")} '
|
||||
f'timestamp={cmd.get("timestamp")}: {message}'
|
||||
)
|
||||
tags = {'pod_id': self.pod_id, 'server_name': cmd.get('server_id', 'unknown')}
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_ERROR_BY_TYPE, tags={**tags, 'error_type': error_type}
|
||||
@@ -192,17 +164,6 @@ class WriteEngine(SientiaMonitoring):
|
||||
metric_object=metrics.OPC_WRITE_TOTAL,
|
||||
tags={**tags, 'model_id': cmd.get('model_id', 'unknown'), 'result': 'error'},
|
||||
)
|
||||
# Same series as a success (status='ok'), so it always shows the last outcome.
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_WRITE_STATUS_BY_MODEL,
|
||||
method='set',
|
||||
value=0,
|
||||
tags={
|
||||
'pod_id': self.pod_id,
|
||||
'model_id': cmd.get('model_id', 'unknown'),
|
||||
'status': 'ok',
|
||||
},
|
||||
)
|
||||
result = WriteResult(ok=False, status=status, opc_status_code=error_type, message=message)
|
||||
await self.sink.update_write_status(cmd, result)
|
||||
return result
|
||||
|
||||
@@ -2,27 +2,31 @@
|
||||
object graph and wires it into the app's lifecycle.
|
||||
|
||||
Reuses the existing ingestor's logger/notification_handler/metrics_controller and
|
||||
its already-connected DataManager. The OpcConnectionPool (and its CertProvider,
|
||||
MinIO + Mongo) belongs to the read path in IngestorManager (SIENTIAPDE-1974):
|
||||
reads start in prepare_ingestor, before this runtime exists, and must keep running
|
||||
if anything here fails. The runtime only borrows the pool for writes and gRPC.
|
||||
its already-connected DataManager (so CertProvider shares the existing Mongo
|
||||
connection instead of opening a second one -- MongoDBRepository.find_one has the
|
||||
exact (collection_name, filters, metadata) signature CertProvider.fetch calls).
|
||||
|
||||
# ponytail: scheduled_tick + gRPC slot_lookup wiring land in the next 1974 RC.
|
||||
# ponytail: subscription v2 swap + scheduled_tick wiring happen at 1974
|
||||
# integration; when swapping, build ONE SubscriptionEngine per OPC server (its
|
||||
# non_receive_count is engine-wide -- see plan ledger F1 carry-forward).
|
||||
"""
|
||||
|
||||
from os import getenv
|
||||
|
||||
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
|
||||
from sientia_do.notifications.models import NotificationLevel
|
||||
from sientia_do.observability.logger import Logger
|
||||
from sientia_do.observability.metrics_controller import MetricsController
|
||||
from sientia_do.observability.sientia_monitoring import SientiaMonitoring
|
||||
from sientia_do.repository.minio_repository import MinioRepository
|
||||
from sientia_do.temporal.activities.postgres import Postgres
|
||||
from sientia_do.utils.connectors_config import build_postgres_config
|
||||
|
||||
from ingestor.adapters.grpc_server import GrpcServer
|
||||
from ingestor.adapters.nats_consumer import NatsWriteConsumer
|
||||
from ingestor.config import GatewayConfig
|
||||
from ingestor.config import GatewayConfig, _env_bool
|
||||
from ingestor.connection.certs import CertProvider
|
||||
from ingestor.connection.pool import OpcConnectionPool
|
||||
from ingestor.connection.servers import ServerConfigResolver
|
||||
from ingestor.engines.active_read import ActiveReadEngine
|
||||
from ingestor.engines.write import WriteEngine
|
||||
from ingestor.io.data_sink import DataSink
|
||||
@@ -40,7 +44,6 @@ class GatewayRuntime(SientiaMonitoring):
|
||||
self,
|
||||
config: GatewayConfig,
|
||||
data_manager: DataManager,
|
||||
pool: OpcConnectionPool,
|
||||
logger: Logger,
|
||||
notification_handler: NotificationHandler,
|
||||
metrics_controller: MetricsController,
|
||||
@@ -52,7 +55,26 @@ class GatewayRuntime(SientiaMonitoring):
|
||||
self._nats_started = False
|
||||
self._nats_notified = False # notify once per outage, not per 5s retry
|
||||
|
||||
self.pool = pool # owned by IngestorManager (read path); closed there
|
||||
minio = MinioRepository(
|
||||
endpoint=getenv('MINIO_ENDPOINT', 'localhost:9000'),
|
||||
access_key=getenv('MINIO_ACCESS_KEY', 'minioadmin'),
|
||||
secret_key=getenv('MINIO_SECRET_KEY', 'minioadmin'),
|
||||
secure=_env_bool('MINIO_SECURE', True),
|
||||
bucket=config.cert_bucket,
|
||||
logger=logger,
|
||||
notification_handler=notification_handler,
|
||||
metrics_controller=metrics_controller,
|
||||
)
|
||||
cert_provider = CertProvider(
|
||||
mongo=data_manager.mongo_repository,
|
||||
minio=minio,
|
||||
cache_dir=config.cert_cache_dir,
|
||||
bucket=config.cert_bucket,
|
||||
metadata=self.metadata,
|
||||
)
|
||||
self.pool = OpcConnectionPool(
|
||||
cert_provider, logger, notification_handler, metrics_controller
|
||||
)
|
||||
|
||||
postgres = Postgres(
|
||||
**build_postgres_config(),
|
||||
@@ -62,12 +84,6 @@ class GatewayRuntime(SientiaMonitoring):
|
||||
)
|
||||
self.sink = DataSink(data_manager, postgres)
|
||||
|
||||
# Write commands carry only the server_id (SIENTIAPDE-1975): the connection is
|
||||
# resolved here, from the live pooled connection or the connectors/opc-servers
|
||||
# documents, with the same rule the orchestrator uses for the read slots.
|
||||
self.servers = ServerConfigResolver(
|
||||
self.pool, data_manager.mongo_repository, logger, metadata=self.metadata
|
||||
)
|
||||
self.write_engine = WriteEngine(
|
||||
self.pool,
|
||||
self.sink,
|
||||
@@ -77,7 +93,6 @@ class GatewayRuntime(SientiaMonitoring):
|
||||
write_timestamp=config.write_timestamp,
|
||||
write_quality=config.write_quality,
|
||||
min_pause_s=config.write_min_pause_s,
|
||||
servers=self.servers,
|
||||
)
|
||||
self.nats_consumer = NatsWriteConsumer(
|
||||
config.nats_url,
|
||||
@@ -113,16 +128,10 @@ class GatewayRuntime(SientiaMonitoring):
|
||||
if self._nats_started:
|
||||
await self.nats_consumer.stop()
|
||||
self._nats_started = False
|
||||
# The pool is not closed here: IngestorManager.shutdown() owns it and runs
|
||||
# after this, once its readers have dropped their subscriptions.
|
||||
await self.pool.close_all()
|
||||
|
||||
async def idle_sweep(self) -> None:
|
||||
await self.pool.release_idle(self.config.idle_conn_max_s)
|
||||
if self._nats_started and self.nats_consumer.closed:
|
||||
# Reconnection is unbounded once connected; a closed connection means it
|
||||
# was closed for good (e.g. by the server), so subscribe again.
|
||||
self.logger.warning('GatewayRuntime: NATS connection closed; starting again')
|
||||
self._nats_started = False
|
||||
if not self._nats_started:
|
||||
await self._start_nats()
|
||||
|
||||
|
||||
@@ -268,14 +268,12 @@ class Ingestor(SientiaMonitoring):
|
||||
|
||||
overleases = list(self.ingestor_manager.managed_tags.keys())[1:]
|
||||
|
||||
# Unsubscribe before giving the leases back: another pod may take a released
|
||||
# slot right away, and must not find this pod still reading it
|
||||
await self.ingestor_manager.drop_slot_leases(overleases)
|
||||
|
||||
for lease in overleases:
|
||||
await self.ingestor_manager.unsubscribe_slot(lease)
|
||||
self.ingestor_manager.managed_tags.pop(lease)
|
||||
|
||||
await self.ingestor_manager.drop_slot_leases(overleases)
|
||||
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.SLOTS_MANAGED,
|
||||
method='set',
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
import asyncio
|
||||
import re
|
||||
|
||||
from sqlalchemy import Select, Update, case, column, select, table, update
|
||||
from sqlalchemy import text
|
||||
|
||||
import ingestor.metrics as metrics
|
||||
|
||||
@@ -9,34 +8,6 @@ import ingestor.metrics as metrics
|
||||
# match before a trailing '\n', which is not what we want for a SQL identifier.
|
||||
_IDENTIFIER_RE = re.compile(r'\w+', re.ASCII)
|
||||
|
||||
# Worst-wins precedence of predictions.write_status (SIENTIAPDE-1975), from best to worst;
|
||||
# NULL, 'pending' and unknown values rank 0. The gateway is the only writer that updates the
|
||||
# column: the Laborious only sets the value of the rows it inserts ('pending', PI outcome).
|
||||
STATUS_RANK = {'ok': 1, 'error': 2, 'session_bad': 3}
|
||||
|
||||
|
||||
def write_status_statements(
|
||||
schema: str, table_name: str, status: str, timestamp, model_id
|
||||
) -> tuple[Update, Select]:
|
||||
"""The UPDATE that records `status` unless the row has a worse one, and the SELECT that
|
||||
tells whether the row exists. Built with SQLAlchemy Core: identifiers and values are
|
||||
never formatted into the SQL text."""
|
||||
predictions = table(
|
||||
table_name,
|
||||
column('write_status'),
|
||||
column('timestamp'),
|
||||
column('model_id'),
|
||||
schema=schema,
|
||||
)
|
||||
row = (predictions.c.timestamp == timestamp) & (predictions.c.model_id == model_id)
|
||||
current_rank = case(STATUS_RANK, value=predictions.c.write_status, else_=0)
|
||||
return (
|
||||
update(predictions)
|
||||
.where(row, current_rank <= STATUS_RANK.get(status, 0))
|
||||
.values(write_status=status),
|
||||
select(1).select_from(predictions).where(row),
|
||||
)
|
||||
|
||||
|
||||
class DataSink:
|
||||
"""Gateway egress wrapper: fans a reading out to every configured name/collection."""
|
||||
@@ -50,16 +21,15 @@ class DataSink:
|
||||
await self.data_manager.emit_metric(**kwargs)
|
||||
|
||||
async def publish_reading(self, tag: str, value, timestamp, names: list[dict]) -> None:
|
||||
"""Publish one MongoDB document per name entry and record its tag value metric.
|
||||
|
||||
`model_id` is only written when the name carries one: names converted from a
|
||||
slot v1 have none, and their documents must stay identical to the legacy ones."""
|
||||
"""Publish one MongoDB document per name entry and record its tag value metric."""
|
||||
for name in names:
|
||||
doc = {'tag': tag, 'name': name['tag_name']}
|
||||
if 'model_id' in name:
|
||||
doc['model_id'] = name['model_id']
|
||||
doc['timestamp'] = timestamp
|
||||
doc['value'] = value
|
||||
doc = {
|
||||
'tag': tag,
|
||||
'name': name['tag_name'],
|
||||
'model_id': name['model_id'],
|
||||
'timestamp': timestamp,
|
||||
'value': value,
|
||||
}
|
||||
await self.data_manager.publish(name['collection'], doc)
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_TAG_VALUE,
|
||||
@@ -68,76 +38,37 @@ class DataSink:
|
||||
tags={
|
||||
'pod_id': self.data_manager.pod_id,
|
||||
'tag_name': name['tag_name'],
|
||||
'model_id': name.get('model_id', ''),
|
||||
'model_id': name['model_id'],
|
||||
},
|
||||
)
|
||||
|
||||
async def update_write_status(self, cmd: dict, result) -> str:
|
||||
"""Record a write outcome on the prediction row (C3), keeping the worst one.
|
||||
async def update_write_status(self, cmd: dict, result) -> None: # NOSONAR
|
||||
"""UPDATE the per-model prediction row's write_status (C3), by timestamp+model_id.
|
||||
|
||||
The row is found by timestamp + model_id. A prediction is written to several
|
||||
tags, each answered by its own command, so a better outcome never replaces a
|
||||
worse one: ``ok`` < ``error`` < ``session_bad``, and ``pending``/NULL are
|
||||
replaced by any answer, so the order of the answers does not matter. Only the
|
||||
gateway updates the column (SIENTIAPDE-1975): the Laborious publishes the commands
|
||||
and sets ``pending`` (merged with its PI Web API outcome) on the row it inserts.
|
||||
|
||||
Never raises: a failure here must not reach the NATS callback nor hide the OPC
|
||||
write itself, which already happened. The outcome is logged and counted instead.
|
||||
|
||||
Return:
|
||||
str: ``updated``, ``kept`` (the row already has a worse outcome),
|
||||
``row_not_found`` (e.g. the Laborious INSERT failed), ``invalid`` (bad or
|
||||
missing schema/table/timestamp/model_id) or ``error``.
|
||||
schema/table_name are per-model SQL identifiers, not values — they can't be
|
||||
bound params, so they're checked against an allowlist before interpolation.
|
||||
status/timestamp/model_id ARE bound params, never f-stringed.
|
||||
"""
|
||||
try:
|
||||
schema = self._validate_identifier(cmd.get('schema'))
|
||||
table = self._validate_identifier(cmd.get('table_name'))
|
||||
if cmd.get('timestamp') is None or cmd.get('model_id') is None:
|
||||
raise ValueError('missing timestamp or model_id')
|
||||
except ValueError as e:
|
||||
return await self._status_outcome('invalid', cmd, f'write_status not recorded: {e}')
|
||||
|
||||
statements = write_status_statements(
|
||||
schema, table, result.status, cmd['timestamp'], cmd['model_id']
|
||||
schema = self._validate_identifier(cmd['schema'])
|
||||
table = self._validate_identifier(cmd['table_name'])
|
||||
# schema/table are allowlist-validated identifiers above; status/timestamp/model_id
|
||||
# are bound params below, never interpolated — not a real injection vector.
|
||||
stmt = text(
|
||||
f'UPDATE {schema}.{table} SET write_status = :status ' # noqa: S608 # nosec B608
|
||||
'WHERE timestamp = :timestamp AND model_id = :model_id'
|
||||
)
|
||||
try:
|
||||
# The engine is synchronous (no async pg driver here): run it off the event
|
||||
# loop, which also serves the subscriptions of the read path.
|
||||
outcome = await asyncio.to_thread(self._update_sync, *statements)
|
||||
except Exception as e:
|
||||
return await self._status_outcome('error', cmd, f'write_status UPDATE failed: {e}')
|
||||
detail = {
|
||||
'updated': f'write_status={result.status} recorded',
|
||||
'kept': f'write_status={result.status} not recorded: the row has a worse outcome',
|
||||
'row_not_found': f'write_status={result.status} not recorded: no prediction row',
|
||||
}[outcome]
|
||||
return await self._status_outcome(outcome, cmd, detail)
|
||||
|
||||
def _update_sync(self, update_stmt: Update, exists_stmt: Select) -> str:
|
||||
params = {
|
||||
'status': result.status,
|
||||
'timestamp': cmd['timestamp'],
|
||||
'model_id': cmd['model_id'],
|
||||
}
|
||||
# ponytail: sync engine call inside async def — same pattern sientia_do's own
|
||||
# Postgres activities use (e.g. export_data_to_postgres); no async pg driver here.
|
||||
with self.postgres.engine.begin() as connection:
|
||||
if connection.execute(update_stmt).rowcount:
|
||||
return 'updated'
|
||||
row = connection.execute(exists_stmt).first()
|
||||
return 'kept' if row is not None else 'row_not_found'
|
||||
|
||||
async def _status_outcome(self, outcome: str, cmd: dict, detail: str) -> str:
|
||||
message = (
|
||||
f'{detail} (server={cmd.get("server_id")} addr={cmd.get("addr")} '
|
||||
f'model_id={cmd.get("model_id")} timestamp={cmd.get("timestamp")})'
|
||||
)
|
||||
if outcome in ('updated', 'kept'):
|
||||
self.data_manager.logger.debug(message)
|
||||
else:
|
||||
self.data_manager.logger.warning(message)
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_WRITE_STATUS_UPDATE_TOTAL,
|
||||
tags={'pod_id': self.data_manager.pod_id, 'result': outcome},
|
||||
)
|
||||
return outcome
|
||||
connection.execute(stmt, params)
|
||||
|
||||
@staticmethod
|
||||
def _validate_identifier(name: object) -> str:
|
||||
def _validate_identifier(name: str) -> str:
|
||||
if not isinstance(name, str) or not _IDENTIFIER_RE.fullmatch(name):
|
||||
raise ValueError(f'invalid SQL identifier: {name!r}')
|
||||
return name
|
||||
|
||||
@@ -1,47 +1,25 @@
|
||||
"""Parse the slot written by the Orchestrator to Redis into the slot v2 structure (contract C1).
|
||||
|
||||
Both shapes are accepted so the gateway and the Orchestrator can be deployed in
|
||||
any order (SIENTIAPDE-1974):
|
||||
|
||||
- v2: each tag carries `names` (one entry per collection / tag name / model);
|
||||
- v1 (legacy): each tag carries `tag_name` + `topics`, and the server carries the
|
||||
file-based `cert_path`. v1 is converted to v2: one `names` entry per topic, and
|
||||
`use_security` is true when the server had a certificate path (the certificate
|
||||
itself now always comes from MinIO via CertProvider, never from that path).
|
||||
"""
|
||||
"""Parse the slot v2 structure written by the Orchestrator to Redis (contract C1)."""
|
||||
|
||||
SERVER_FIELDS = ('server_id', 'url', 'server_uri', 'use_security', 'subscription_period_ms')
|
||||
# What a connection/subscription needs (the pool keys clients by server_id itself).
|
||||
CONNECTION_FIELDS = tuple(f for f in SERVER_FIELDS if f != 'server_id')
|
||||
|
||||
|
||||
def _parse_names(address: str, tag: dict) -> list[dict]:
|
||||
names = tag.get('names')
|
||||
if names:
|
||||
return names
|
||||
topics = tag.get('topics')
|
||||
if topics and tag.get('tag_name'):
|
||||
return [{'collection': topic, 'tag_name': tag['tag_name']} for topic in topics]
|
||||
raise ValueError(f'tag {address!r} has no names')
|
||||
|
||||
|
||||
def _parse_tag(address: str, tag: dict) -> dict:
|
||||
names = tag.get('names')
|
||||
if not names:
|
||||
raise ValueError(f'tag {address!r} has no names')
|
||||
return {
|
||||
'frequency': tag['frequency'],
|
||||
'names': _parse_names(address, tag),
|
||||
'names': names,
|
||||
'active_read': tag.get('active_read', {'enabled': False}),
|
||||
}
|
||||
|
||||
|
||||
def _parse_server(name: str, server: dict) -> dict:
|
||||
def _parse_server(server: dict) -> dict:
|
||||
parsed = {field: server[field] for field in SERVER_FIELDS if field in server}
|
||||
parsed['name'] = server.get('name', name)
|
||||
if 'use_security' not in parsed:
|
||||
parsed['use_security'] = bool(server.get('cert_path'))
|
||||
parsed['tags'] = {addr: _parse_tag(addr, tag) for addr, tag in server['tags'].items()}
|
||||
return parsed
|
||||
|
||||
|
||||
def parse_slot(raw: dict) -> dict:
|
||||
"""Normalize a raw slot (v1 or v2) into `{server_name: {..., tags: {addr: tag}}}`."""
|
||||
return {name: _parse_server(name, server) for name, server in raw.items()}
|
||||
"""Normalize a raw slot v2 dict into `{server_name: {..., tags: {addr: tag}}}`."""
|
||||
return {name: _parse_server(server) for name, server in raw.items()}
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import traceback
|
||||
from copy import deepcopy
|
||||
from os import getenv
|
||||
|
||||
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
|
||||
from sientia_do.notifications.models import NotificationLevel
|
||||
@@ -9,21 +8,9 @@ from sientia_do.observability.metrics_controller import MetricsController
|
||||
from sientia_do.observability.sientia_monitoring import SientiaMonitoring
|
||||
|
||||
import ingestor.metrics as metrics
|
||||
from ingestor.config import GatewayConfig
|
||||
from ingestor.connection.certs import build_cert_provider
|
||||
from ingestor.connection.pool import OpcConnectionPool
|
||||
from ingestor.io.data_sink import DataSink
|
||||
from ingestor.io.slots import parse_slot
|
||||
from ingestor.managers.data_manager import DataManager
|
||||
from ingestor.managers.opc_manager import OpcManager
|
||||
from ingestor.managers.resource_manager import ResourceManager
|
||||
from ingestor.managers.server_reader import ServerReader
|
||||
|
||||
# READ_ENGINE (SIENTIAPDE-1974): 'v2' = ServerReader/SubscriptionEngine over the shared
|
||||
# pool, certificate from MinIO (default). 'legacy' = plan B: the untouched rc3
|
||||
# OpcManager, slot v1 only, certificate files from the Gitea clone (cert_path). Kept
|
||||
# for validation until the end of 1974; remove once unused and the tests pass without it.
|
||||
READ_ENGINES = ('v2', 'legacy')
|
||||
|
||||
|
||||
class IngestorManager(SientiaMonitoring):
|
||||
@@ -60,10 +47,7 @@ class IngestorManager(SientiaMonitoring):
|
||||
|
||||
Attributes:
|
||||
data_manager (DataManager): Manages data persistence
|
||||
opc_managers (dict): ServerReader per OPC server, keyed by server name
|
||||
pool (OpcConnectionPool): OPC connections shared by reads, writes and gRPC;
|
||||
owned here because reads start in prepare_ingestor, before GatewayRuntime
|
||||
sink (DataSink): read egress (fan-out to every slot v2 `names` entry)
|
||||
opc_managers (dict): Dictionary of OPC managers keyed by server name
|
||||
resource_manager (ResourceManager): Manages Redis-based resource coordination
|
||||
number_of_slots (int): Total number of slots configured in the system
|
||||
poll_interval (int): Main loop polling interval
|
||||
@@ -84,8 +68,6 @@ class IngestorManager(SientiaMonitoring):
|
||||
logger: Logger,
|
||||
notification_handler: NotificationHandler,
|
||||
metrics_controller: MetricsController,
|
||||
gateway_config: GatewayConfig | None = None,
|
||||
read_engine: str | None = None,
|
||||
):
|
||||
redis_host: str = redis_data['host']
|
||||
redis_port: int = int(redis_data['port'])
|
||||
@@ -108,26 +90,6 @@ class IngestorManager(SientiaMonitoring):
|
||||
metrics_controller=metrics_controller,
|
||||
)
|
||||
self.opc_managers: dict = {}
|
||||
# Read path (SIENTIAPDE-1974): pooled connections with the certificate from
|
||||
# MinIO. Nothing here touches the network at construction (MinIO is lazy), so
|
||||
# a MinIO/Postgres/NATS outage can't stop the reads before they start.
|
||||
config = gateway_config or GatewayConfig.certs_from_env()
|
||||
self.cert_provider = build_cert_provider(
|
||||
mongo=self.data_manager.mongo_repository,
|
||||
config=config,
|
||||
logger=logger,
|
||||
notification_handler=notification_handler,
|
||||
metrics_controller=metrics_controller,
|
||||
metadata=metadata,
|
||||
)
|
||||
self.pool = OpcConnectionPool(
|
||||
self.cert_provider, logger, notification_handler, metrics_controller
|
||||
)
|
||||
self.sink = DataSink(self.data_manager, None)
|
||||
self._server_cert_warned: set[str] = set()
|
||||
self.read_engine = self._resolve_read_engine(
|
||||
read_engine or getenv('READ_ENGINE') or 'v2', logger
|
||||
)
|
||||
self.resource_manager = ResourceManager(
|
||||
host=redis_host,
|
||||
port=redis_port,
|
||||
@@ -147,19 +109,36 @@ class IngestorManager(SientiaMonitoring):
|
||||
|
||||
self.metadata = metadata
|
||||
|
||||
@staticmethod
|
||||
def _resolve_read_engine(value: str, logger: Logger) -> str:
|
||||
engine = value.strip().lower()
|
||||
if engine not in READ_ENGINES:
|
||||
logger.warning(f'Invalid READ_ENGINE {value!r}, using v2 ({"|".join(READ_ENGINES)})')
|
||||
engine = 'v2'
|
||||
logger.info(f'OPC read engine: {engine}')
|
||||
return engine
|
||||
async def initialize_opc_from_config(self, server_config: dict) -> OpcManager | None:
|
||||
"""
|
||||
Initializes an OPC Manager instance using the provided server configuration.
|
||||
|
||||
def _build_reader(self, server_config: dict) -> ServerReader | OpcManager:
|
||||
if self.read_engine == 'legacy':
|
||||
# Plan B: the rc3 OpcManager, unchanged (own connection, cert files).
|
||||
return OpcManager(
|
||||
This method creates and configures an OPC Manager for a specific OPC UA server,
|
||||
establishing the connection and preparing it for tag subscriptions.
|
||||
|
||||
Args:
|
||||
server_config (dict): A dictionary containing the OPC server configuration.
|
||||
Expected keys include:
|
||||
- 'name' (str): The name of the OPC server.
|
||||
- 'url' (str): The URL of the OPC server.
|
||||
- 'server_uri' (str): The URI of the OPC server.
|
||||
- 'cert_path' (str, optional): Path to the client certificate file.
|
||||
- 'private_key_path' (str, optional): Path to the private key file.
|
||||
- 'server_cert_path' (str, optional): Path to the server certificate file.
|
||||
|
||||
Returns:
|
||||
OpcManager | None: An initialized OpcManager instance if successful,
|
||||
otherwise None if an error occurs during initialization.
|
||||
|
||||
Raises:
|
||||
Exception: If OPC manager initialization fails, the error is logged and
|
||||
a notification is sent, but the method returns None to allow
|
||||
the system to continue operating with other servers.
|
||||
"""
|
||||
|
||||
try:
|
||||
self.logger.info(f'Initializing OpcManager at {server_config["url"]}')
|
||||
manager = OpcManager(
|
||||
name=server_config['name'],
|
||||
url=server_config['url'],
|
||||
subscription_period_ms=server_config['subscription_period_ms'],
|
||||
@@ -173,41 +152,6 @@ class IngestorManager(SientiaMonitoring):
|
||||
server_cert_path=server_config.get('server_cert_path'),
|
||||
metrics_controller=self.metrics_controller,
|
||||
)
|
||||
return ServerReader(
|
||||
name=server_config['name'],
|
||||
server_config=server_config,
|
||||
pool=self.pool,
|
||||
sink=self.sink,
|
||||
logger=self.logger,
|
||||
notification_handler=self.notification_handler,
|
||||
metrics_controller=self.metrics_controller,
|
||||
metadata=self.metadata,
|
||||
)
|
||||
|
||||
async def initialize_opc_from_config(
|
||||
self, server_config: dict
|
||||
) -> ServerReader | OpcManager | None:
|
||||
"""
|
||||
Initializes the reader of one OPC UA server from its slot configuration.
|
||||
|
||||
With READ_ENGINE=v2 (default) creates a ServerReader and opens its pooled
|
||||
connection; with `use_security`, the certificate comes from MinIO. With
|
||||
READ_ENGINE=legacy (plan B) creates the rc3 OpcManager, which connects on its
|
||||
own with the certificate files of the slot v1 (`cert_path`).
|
||||
|
||||
Args:
|
||||
server_config (dict): The server configuration of the slot, without the
|
||||
'tags' key. v2: 'name', 'server_id', 'url', 'server_uri',
|
||||
'use_security', 'subscription_period_ms'. legacy: the slot v1 fields.
|
||||
|
||||
Returns:
|
||||
ServerReader | OpcManager | None: The connected reader, or None if the
|
||||
connection fails (logged and notified, so the other servers keep working).
|
||||
"""
|
||||
|
||||
try:
|
||||
self.logger.info(f'Initializing OPC reader at {server_config["url"]}')
|
||||
manager = self._build_reader(server_config)
|
||||
|
||||
manager.config = server_config
|
||||
await manager.connect()
|
||||
@@ -232,33 +176,23 @@ class IngestorManager(SientiaMonitoring):
|
||||
Gracefully shuts down the IngestorManager and all its components.
|
||||
|
||||
This method ensures proper cleanup of:
|
||||
- All OPC readers, their subscriptions and the pooled connections
|
||||
- All OPC manager instances and their connections
|
||||
- Data manager connections and resources
|
||||
- Active subscriptions and server connections
|
||||
|
||||
The shutdown process is performed asynchronously to allow proper cleanup
|
||||
of all managed resources before termination.
|
||||
"""
|
||||
for _server_name, server in self.opc_managers.items():
|
||||
await server.shutdown()
|
||||
await self.pool.close_all()
|
||||
self.data_manager.shutdown()
|
||||
|
||||
async def remove_server(self, server: str, evict: bool = True):
|
||||
async def remove_server(self, server: str):
|
||||
"""
|
||||
Removes an OPC server from the ingestor.
|
||||
|
||||
Args:
|
||||
server (str): The server name.
|
||||
evict (bool): Also drop the pooled connection (lost server, failed
|
||||
subscription). False when the server merely left this pod's slots:
|
||||
the shared connection stays for writes/gRPC until the idle sweep.
|
||||
"""
|
||||
if server in self.opc_managers:
|
||||
if self.read_engine == 'legacy':
|
||||
# OpcManager owns its connection: shutdown always disconnects it.
|
||||
await self.opc_managers[server].shutdown()
|
||||
else:
|
||||
await self.opc_managers[server].shutdown(evict=evict)
|
||||
await self.opc_managers[server].shutdown()
|
||||
del self.opc_managers[server]
|
||||
for slot, _config in self.managed_tags.items():
|
||||
self.managed_tags[slot].pop(server, None)
|
||||
@@ -324,10 +258,9 @@ class IngestorManager(SientiaMonitoring):
|
||||
for server in servers:
|
||||
if server not in registered_servers:
|
||||
self.logger.warning(
|
||||
f'Server {server} not found in managed tags. Unsubscribing from server.'
|
||||
f'Server {server} not found in managed tags. Desconnecting from server.'
|
||||
)
|
||||
# Slot moved away: keep the shared connection for writes/gRPC.
|
||||
await self.remove_server(server, evict=False)
|
||||
await self.remove_server(server)
|
||||
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_MANAGERS_ACTIVE,
|
||||
@@ -486,18 +419,14 @@ class IngestorManager(SientiaMonitoring):
|
||||
|
||||
Notes:
|
||||
- If a slot is leased but its details cannot be retrieved
|
||||
(i.e., `get_tag_slot` returns None) or are malformed, that slot is skipped.
|
||||
- Slot details are normalized to the slot v2 structure (parse_slot).
|
||||
(i.e., `get_tag_slot` returns None), that slot is skipped.
|
||||
"""
|
||||
|
||||
acquired = {}
|
||||
for i in range(1, self.number_of_slots + 1):
|
||||
if await self.resource_manager.lease_tag(str(i)):
|
||||
self.logger.info(f'Leased slot {i}')
|
||||
raw = await self.resource_manager.get_tag_slot(str(i))
|
||||
if raw is None:
|
||||
continue
|
||||
slots = self.parse_slot_config(str(i), raw)
|
||||
slots = await self.resource_manager.get_tag_slot(str(i))
|
||||
if slots is None:
|
||||
continue
|
||||
acquired[str(i)] = slots
|
||||
@@ -538,84 +467,26 @@ class IngestorManager(SientiaMonitoring):
|
||||
|
||||
async def unsubscribe_slot(self, slot: str):
|
||||
"""
|
||||
Unsubscribes a specific slot from every OPC reader that holds a subscription for it.
|
||||
Unsubscribes a specific slot from all associated OPC servers.
|
||||
|
||||
The readers' own subscriptions are the source of truth, not managed_tags: the
|
||||
main loop removes a slot from managed_tags before update_ingestor_manager asks
|
||||
to unsubscribe it, both when manage_leases drops an extra lease (which already
|
||||
unsubscribed it) and when update_slot_config finds the slot gone from Redis
|
||||
(which did not). Looking the slot up in managed_tags raised KeyError in both
|
||||
cases, which stopped the main loop and restarted the container; the second case
|
||||
also left the subscription open on the OPC server. Unsubscribing a slot that no
|
||||
reader holds is a no-op.
|
||||
|
||||
A reader that fails to unsubscribe is handled by _unsubscribe_reader: the other
|
||||
readers are still unsubscribed and the main loop goes on.
|
||||
This method removes all subscriptions for a given slot across all
|
||||
OPC servers that were managing it. It ensures clean cleanup of
|
||||
resources when slots are released or reconfigured.
|
||||
|
||||
Args:
|
||||
slot (str): The name of the slot to unsubscribe.
|
||||
|
||||
Raises:
|
||||
KeyError: If the specified slot does not exist in the managed tags.
|
||||
|
||||
Side Effects:
|
||||
- Removes the slot's subscription from every OPC reader that has one
|
||||
- Removes subscriptions from all OPC servers for the specified slot
|
||||
- Cleans up subscription resources on the OPC servers
|
||||
"""
|
||||
|
||||
# list() snapshots the readers: the loop awaits, so a reader added or removed
|
||||
# meanwhile must not break the iteration. SonarQube flags it as an unnecessary
|
||||
# list() (S7504); the issue is accepted in SonarQube for this reason.
|
||||
for server, reader in list(self.opc_managers.items()):
|
||||
if slot in reader.subscriptions:
|
||||
await self._unsubscribe_reader(server, reader, slot)
|
||||
|
||||
async def _unsubscribe_reader(
|
||||
self, server: str, reader: ServerReader | OpcManager, slot: str
|
||||
) -> None:
|
||||
"""
|
||||
Unsubscribe one reader from a slot without letting a failure stop the main loop.
|
||||
|
||||
A failure (e.g. the legacy OpcManager, whose delete() raises when the server
|
||||
session is gone) increments OPC_SUBSCRIPTION_ERRORS, sends
|
||||
OPC_UNSUBSCRIBE_ERROR_<slot>:<server> and is logged. The slot is dropped from the
|
||||
reader's books, as the v2 engine already does, so it can be subscribed again.
|
||||
|
||||
Known limitation: if delete() failed while the session is still alive (e.g. a
|
||||
timeout), the server-side subscription may keep delivering data until that
|
||||
session ends (reconnection or pod restart).
|
||||
|
||||
Args:
|
||||
server (str): The reader's key in opc_managers
|
||||
reader (ServerReader | OpcManager): The OPC reader
|
||||
slot (str): The slot to unsubscribe
|
||||
"""
|
||||
try:
|
||||
await reader.unsubscribe(slot)
|
||||
except Exception as e:
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_SUBSCRIPTION_ERRORS,
|
||||
tags={
|
||||
'pod_id': self.pod_id,
|
||||
'server': server,
|
||||
'slot': slot,
|
||||
},
|
||||
)
|
||||
trace = traceback.format_exc()
|
||||
await self.send_notification_async(
|
||||
metadata=self.metadata,
|
||||
notification_id=f'OPC_UNSUBSCRIBE_ERROR_{slot}:{server}',
|
||||
message=f'Failed to unsubscribe slot {slot} from {server}: {e}',
|
||||
block='opc_manager',
|
||||
level=NotificationLevel.ERROR,
|
||||
attachment_content=trace,
|
||||
)
|
||||
self.logger.error(trace)
|
||||
reader.subscriptions.pop(slot, None)
|
||||
|
||||
async def _unsubscribe_removed_slots(self, slots: list[str]) -> None:
|
||||
"""Unsubscribe and forget slots whose configuration is gone from Redis."""
|
||||
for slot in slots:
|
||||
self.logger.warning(f'Slot {slot} is no longer configured; unsubscribing it')
|
||||
await self.unsubscribe_slot(slot)
|
||||
self.managed_tags.pop(slot, None)
|
||||
for server in self.managed_tags[slot].keys():
|
||||
if server in self.opc_managers:
|
||||
await self.opc_managers[server].unsubscribe(slot)
|
||||
|
||||
async def update_slot_config(self):
|
||||
"""
|
||||
@@ -625,7 +496,7 @@ class IngestorManager(SientiaMonitoring):
|
||||
This method performs the following steps:
|
||||
1. Renews the lease for each managed slot using the resource manager.
|
||||
2. Fetches the latest configuration for each slot.
|
||||
3. Logs, unsubscribes and removes slots whose configurations are no longer available.
|
||||
3. Logs and removes slots whose configurations are no longer available.
|
||||
4. Updates the configuration of slots if changes are detected.
|
||||
5. Unsubscribes and re-subscribes to slots with updated configurations.
|
||||
6. Removes slots from the managed tags if they are no longer valid.
|
||||
@@ -647,75 +518,15 @@ class IngestorManager(SientiaMonitoring):
|
||||
removed_slots: list[str] = []
|
||||
for slot, _slot_config in self.managed_tags.items():
|
||||
await self.resource_manager.renew_tag_lease(slot)
|
||||
raw = await self.resource_manager.get_tag_slot(slot)
|
||||
if raw is None:
|
||||
update = await self.resource_manager.get_tag_slot(slot)
|
||||
if update is None:
|
||||
removed_slots.append(slot)
|
||||
continue
|
||||
|
||||
update = self.parse_slot_config(slot, raw)
|
||||
if update is None:
|
||||
# Malformed config: keep reading with the last good one.
|
||||
continue
|
||||
self.managed_tags[slot] = update
|
||||
|
||||
# Unsubscribe right away: leaving it to update_ingestor_manager's snapshot diff
|
||||
# would leak the subscription if anything in between failed
|
||||
await self._unsubscribe_removed_slots(removed_slots)
|
||||
|
||||
def parse_slot_config(self, slot: str, raw: dict) -> dict | None:
|
||||
"""
|
||||
Normalizes the raw slot read from Redis to the slot v2 structure.
|
||||
|
||||
Accepts slot v1 (`topics`) and v2 (`names`), so the gateway and the
|
||||
Orchestrator can be deployed in any order (SIENTIAPDE-1974). With
|
||||
READ_ENGINE=legacy the raw slot v1 is kept as is (what OpcManager reads), and
|
||||
a slot v2 is rejected.
|
||||
|
||||
Returns:
|
||||
dict | None: The parsed slot, or None when the config is malformed
|
||||
(logged and notified).
|
||||
"""
|
||||
try:
|
||||
if self.read_engine == 'legacy':
|
||||
self._validate_slot_v1(raw)
|
||||
return raw
|
||||
parsed = parse_slot(raw)
|
||||
except (AttributeError, KeyError, TypeError, ValueError) as e:
|
||||
self.send_notification(
|
||||
metadata=self.metadata,
|
||||
notification_id=f'OPC_SLOT_CONFIG_ERROR_{slot}',
|
||||
message=f'Slot {slot} has an invalid configuration: {e!r}',
|
||||
block='ingestor_manager',
|
||||
level=NotificationLevel.ERROR,
|
||||
)
|
||||
self.logger.error(f'Slot {slot} has an invalid configuration: {e!r}')
|
||||
return None
|
||||
self._warn_server_cert_path(raw)
|
||||
return parsed
|
||||
|
||||
@staticmethod
|
||||
def _validate_slot_v1(raw: dict) -> None:
|
||||
"""READ_ENGINE=legacy: OpcManager only understands slot v1 (`tag_name` +
|
||||
`topics` per tag); a v2 slot (`names`) would fail in its datachange callback."""
|
||||
for name, server in raw.items():
|
||||
for address, tag in server['tags'].items():
|
||||
if 'topics' not in tag or 'tag_name' not in tag or 'frequency' not in tag:
|
||||
raise ValueError(
|
||||
f'READ_ENGINE=legacy needs a slot v1 (tag_name + topics); '
|
||||
f'server {name!r}, tag {address!r} is not'
|
||||
)
|
||||
|
||||
def _warn_server_cert_path(self, raw: dict) -> None:
|
||||
"""Logs once per server when a slot still pins a server certificate
|
||||
(`server_cert_path`): connections now come from the pool with the platform
|
||||
certificate from MinIO, and that file-based pin is no longer applied."""
|
||||
for name, server in raw.items():
|
||||
if server.get('server_cert_path') and name not in self._server_cert_warned:
|
||||
self._server_cert_warned.add(name)
|
||||
self.logger.warning(
|
||||
f'OPC server {name}: server_cert_path is set in the slot but is no '
|
||||
f'longer applied (server certificate validation is disabled)'
|
||||
)
|
||||
for slot in removed_slots:
|
||||
self.managed_tags.pop(slot, None)
|
||||
|
||||
async def drop_slot_leases(self, ids: list[str]) -> None:
|
||||
"""
|
||||
@@ -823,7 +634,7 @@ class IngestorManager(SientiaMonitoring):
|
||||
self.logger.error(trace)
|
||||
|
||||
self.logger.warning(f'Removing subscription from server {server} for slot {slot}')
|
||||
await self._unsubscribe_reader(server, self.opc_managers[server], slot)
|
||||
await self.opc_managers[server].unsubscribe(slot)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
@@ -1,107 +0,0 @@
|
||||
"""Per-server read path driven by the IngestorManager (SIENTIAPDE-1974).
|
||||
|
||||
Replaces the legacy OpcManager behind the same interface the IngestorManager
|
||||
already uses (connect / create_subscription / subscribe / unsubscribe /
|
||||
check_cycles / check_opc_listenning / shutdown), so the slot lease loop is
|
||||
unchanged. The difference is underneath:
|
||||
|
||||
- the connection comes from the shared OpcConnectionPool, so the certificate is
|
||||
fetched from MinIO by CertProvider (Basic256Sha256) — never from a file shipped
|
||||
in the Gitea repository;
|
||||
- readings go through SubscriptionEngine, which fans each value out to every
|
||||
`names` entry of the slot v2 (one document per collection / tag name / model).
|
||||
"""
|
||||
|
||||
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
|
||||
from sientia_do.observability.logger import Logger
|
||||
from sientia_do.observability.metrics_controller import MetricsController
|
||||
|
||||
import ingestor.metrics as metrics
|
||||
from ingestor.connection.pool import OpcConnectionPool
|
||||
from ingestor.engines.subscription import SubscriptionEngine
|
||||
from ingestor.io.data_sink import DataSink
|
||||
from ingestor.io.slots import CONNECTION_FIELDS
|
||||
|
||||
|
||||
class ServerReader:
|
||||
"""Reads one OPC server: one SubscriptionEngine (its `non_receive_count` is
|
||||
engine-wide, hence one per server) with one subscription per leased slot."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
name: str,
|
||||
server_config: dict,
|
||||
pool: OpcConnectionPool,
|
||||
sink: DataSink,
|
||||
logger: Logger,
|
||||
notification_handler: NotificationHandler,
|
||||
metrics_controller: MetricsController,
|
||||
metadata: dict | None = None,
|
||||
):
|
||||
self.name = name
|
||||
self.server_id = str(server_config['server_id'])
|
||||
self.url = server_config['url']
|
||||
self.server_cfg = {k: server_config[k] for k in CONNECTION_FIELDS if k in server_config}
|
||||
self.pool = pool
|
||||
self.logger = logger
|
||||
self.config: dict | None = None
|
||||
self.engine = SubscriptionEngine(
|
||||
pool, sink, logger, notification_handler, metrics_controller, metadata
|
||||
)
|
||||
|
||||
def __str__(self):
|
||||
return f'ServerReader(name={self.name}, server_id={self.server_id}, url={self.url})'
|
||||
|
||||
@property
|
||||
def subscriptions(self) -> dict:
|
||||
return self.engine.subscriptions
|
||||
|
||||
async def connect(self):
|
||||
"""Open (or reuse) the pooled connection; raises if the server is unreachable
|
||||
or the certificate can't be fetched, like OpcManager.connect did."""
|
||||
await self.pool.get(self.server_id, self.server_cfg)
|
||||
|
||||
async def create_subscription(self, slot: str):
|
||||
await self.engine.create_subscription(self.server_id, self.server_cfg, slot)
|
||||
|
||||
async def subscribe(self, slot: str, nodes: dict, collect_period: int):
|
||||
await self.engine.subscribe(slot, nodes, collect_period)
|
||||
|
||||
async def unsubscribe(self, slot: str):
|
||||
await self.engine.unsubscribe(slot)
|
||||
|
||||
async def check_cycles(self):
|
||||
await self.engine.check_cycles()
|
||||
|
||||
async def check_opc_listenning(self) -> bool:
|
||||
return await self.engine.check_opc_listenning(self.server_id)
|
||||
|
||||
async def shutdown(self, evict: bool = True):
|
||||
"""Drop every subscription of this server.
|
||||
|
||||
With `evict` (lost server or changed server config) the pooled connection is
|
||||
also dropped, so the next initialization reconnects. Without it (the slot just
|
||||
moved to another pod) the connection stays for writes/gRPC: the connection is
|
||||
shared, and the idle sweep closes it if nobody uses it anymore."""
|
||||
# list() snapshots the keys: unsubscribe() pops from engine.subscriptions, so
|
||||
# iterating the dict directly would raise "changed size during iteration".
|
||||
for slot in list(self.engine.subscriptions): # NOSONAR
|
||||
await self.engine.unsubscribe(slot)
|
||||
if evict:
|
||||
await self.pool.evict(self.server_id)
|
||||
await self.engine.emit_metric(
|
||||
metric_object=metrics.OPC_CONNECTION_STATUS,
|
||||
method='set',
|
||||
value=0,
|
||||
tags={
|
||||
'pod_id': self.engine.pod_id,
|
||||
'server_name': self.server_id,
|
||||
'server_url': self.url,
|
||||
},
|
||||
)
|
||||
await self.engine.emit_metric(
|
||||
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
|
||||
method='set',
|
||||
value=0,
|
||||
tags={'pod_id': self.engine.pod_id, 'server_name': self.server_id},
|
||||
)
|
||||
@@ -93,7 +93,7 @@ OPC_MANAGERS_ACTIVE = Gauge(
|
||||
)
|
||||
OPC_SUBSCRIPTION_ERRORS = Counter(
|
||||
'ingestor_opc_subscription_errors_total',
|
||||
'Errors when trying to subscribe to or unsubscribe from OPC tags',
|
||||
'Errors when trying to subscribe to OPC tags',
|
||||
['pod_id', 'server', 'slot'],
|
||||
)
|
||||
|
||||
@@ -156,12 +156,6 @@ OPC_WRITE_STATUS_BY_MODEL = Gauge(
|
||||
'Last write status per model (1=ok,0=error)',
|
||||
['pod_id', 'model_id', 'status'],
|
||||
)
|
||||
OPC_WRITE_STATUS_UPDATE_TOTAL = Counter(
|
||||
'opc_write_status_update_total',
|
||||
'write_status UPDATEs on the prediction rows',
|
||||
# result: updated|kept|row_not_found|invalid|error
|
||||
['pod_id', 'result'],
|
||||
)
|
||||
# --- Gateway: Active read metrics ---
|
||||
OPC_ACTIVE_READ_TOTAL = Counter(
|
||||
'opc_active_read_total',
|
||||
|
||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "ingestor"
|
||||
version = "2.2.0-rc1"
|
||||
version = "2.0.0-rc3"
|
||||
description = "Sientia DataOps Ingestor - OPC Tag Ingestor"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
|
||||
@@ -4,9 +4,6 @@ redis
|
||||
# (generic Mongo find_one/distinct/aggregate methods) — not in 1.12.2 yet.
|
||||
# Local dev uses an editable install of the local checkout (feature/SIENTIAPDE-1590).
|
||||
sientia_do==1.12.2
|
||||
# TEST ONLY (SIENTIAPDE-1975 dev test): SQLAlchemy 2.1 makes psycopg (v3) the default driver for
|
||||
# postgresql:// URLs; sientia_do 1.12.2 builds that URL and ships only psycopg2-binary.
|
||||
sqlalchemy<2.1
|
||||
prometheus_client
|
||||
pymongo
|
||||
nats-py>=2.6
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
import asyncio
|
||||
import inspect
|
||||
import json
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
@@ -39,99 +37,20 @@ async def test_handle_invalid_json_drops_message_and_logs_error():
|
||||
consumer.logger.error.assert_called_once()
|
||||
|
||||
|
||||
CLIENT = 'ingestor.adapters.nats_consumer.Client'
|
||||
|
||||
|
||||
def _fake_nc(connect=None):
|
||||
nc = MagicMock()
|
||||
nc.connect = connect or AsyncMock()
|
||||
nc.subscribe = AsyncMock()
|
||||
nc.close = AsyncMock()
|
||||
nc.drain = AsyncMock()
|
||||
nc.is_closed = False
|
||||
return nc
|
||||
|
||||
|
||||
async def _hang(*_args, **_kwargs):
|
||||
await asyncio.sleep(10)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_start_subscribes_with_subject_and_queue():
|
||||
consumer = make_consumer(subject='opc.write', queue='gateway-writers')
|
||||
nc = _fake_nc()
|
||||
mock_nc = AsyncMock()
|
||||
mock_nc.subscribe = AsyncMock()
|
||||
|
||||
with patch(CLIENT, return_value=nc):
|
||||
with patch('ingestor.adapters.nats_consumer.nats') as mock_nats:
|
||||
mock_nats.connect = AsyncMock(return_value=mock_nc)
|
||||
await consumer.start()
|
||||
|
||||
kwargs = nc.connect.await_args.kwargs
|
||||
assert kwargs['servers'] == 'nats://localhost:4222'
|
||||
# Reconnection is unbounded once connected: an outage never closes it for good.
|
||||
assert kwargs['max_reconnect_attempts'] == -1
|
||||
nc.subscribe.assert_awaited_once_with('opc.write', queue='gateway-writers', cb=consumer._handle)
|
||||
assert consumer.closed is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_start_gives_up_after_the_connect_timeout():
|
||||
"""start() runs on the main loop: an unreachable NATS must not block it."""
|
||||
consumer = make_consumer(connect_timeout_s=0.1)
|
||||
nc = _fake_nc(connect=AsyncMock(side_effect=_hang))
|
||||
# Outer guard: fails the test instead of hanging it if start() ignores its timeout.
|
||||
start = asyncio.wait_for(consumer.start(), timeout=5)
|
||||
|
||||
with patch(CLIENT, return_value=nc):
|
||||
with pytest.raises(TimeoutError):
|
||||
await start
|
||||
|
||||
nc.close.assert_awaited_once()
|
||||
assert consumer.closed is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_start_closes_the_connection_when_subscribe_fails():
|
||||
consumer = make_consumer()
|
||||
nc = _fake_nc()
|
||||
nc.subscribe.side_effect = RuntimeError('permissions violation')
|
||||
|
||||
with patch(CLIENT, return_value=nc), pytest.raises(RuntimeError):
|
||||
await consumer.start()
|
||||
|
||||
nc.close.assert_awaited_once()
|
||||
assert consumer.closed is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handle_unexpected_engine_error_is_logged_not_raised():
|
||||
consumer = make_consumer()
|
||||
consumer.write_engine.write.side_effect = RuntimeError('bug')
|
||||
msg = MagicMock(data=json.dumps({'server_id': '1'}).encode())
|
||||
|
||||
await consumer._handle(msg)
|
||||
|
||||
assert 'bug' in consumer.logger.error.call_args.args[0]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_connection_events_are_logged_in_one_line():
|
||||
consumer = make_consumer()
|
||||
nc = _fake_nc()
|
||||
with patch(CLIENT, return_value=nc):
|
||||
await consumer.start()
|
||||
callbacks = nc.connect.await_args.kwargs
|
||||
# nats-py raises InvalidCallbackTypeError on connect for a non-coroutine callback.
|
||||
for name in ('error_cb', 'disconnected_cb', 'reconnected_cb'):
|
||||
assert inspect.iscoroutinefunction(callbacks[name]), name
|
||||
|
||||
await callbacks['error_cb'](OSError('connection refused'))
|
||||
await callbacks['disconnected_cb']()
|
||||
await callbacks['reconnected_cb']()
|
||||
|
||||
warnings = [c.args[0] for c in consumer.logger.warning.call_args_list]
|
||||
assert warnings == [
|
||||
'NATS error: OSError connection refused',
|
||||
'Disconnected from NATS; reconnecting',
|
||||
]
|
||||
mock_nats.connect.assert_awaited_once_with(servers='nats://localhost:4222')
|
||||
mock_nc.subscribe.assert_awaited_once_with(
|
||||
'opc.write', queue='gateway-writers', cb=consumer._handle
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -144,28 +63,14 @@ async def test_stop_before_start_does_not_raise():
|
||||
@pytest.mark.asyncio
|
||||
async def test_stop_after_start_drains_and_unsubscribes():
|
||||
consumer = make_consumer()
|
||||
nc = _fake_nc()
|
||||
sub = AsyncMock()
|
||||
nc.subscribe = AsyncMock(return_value=sub)
|
||||
mock_nc = AsyncMock()
|
||||
mock_sub = AsyncMock()
|
||||
mock_nc.subscribe = AsyncMock(return_value=mock_sub)
|
||||
|
||||
with patch(CLIENT, return_value=nc):
|
||||
with patch('ingestor.adapters.nats_consumer.nats') as mock_nats:
|
||||
mock_nats.connect = AsyncMock(return_value=mock_nc)
|
||||
await consumer.start()
|
||||
await consumer.stop()
|
||||
|
||||
sub.unsubscribe.assert_awaited_once()
|
||||
nc.drain.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stop_on_a_connection_closed_for_good_does_not_raise():
|
||||
consumer = make_consumer()
|
||||
nc = _fake_nc()
|
||||
with patch(CLIENT, return_value=nc):
|
||||
await consumer.start()
|
||||
nc.is_closed = True
|
||||
consumer.sub.unsubscribe = AsyncMock(side_effect=RuntimeError('connection closed'))
|
||||
|
||||
await consumer.stop() # no raise
|
||||
|
||||
nc.drain.assert_not_awaited()
|
||||
assert consumer.closed is True
|
||||
mock_sub.unsubscribe.assert_awaited_once()
|
||||
mock_nc.drain.assert_awaited_once()
|
||||
|
||||
@@ -1,20 +0,0 @@
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
from pytest import fixture
|
||||
|
||||
|
||||
@fixture
|
||||
def reader_double():
|
||||
"""Factory of OPC reader doubles holding subscriptions for the given slots.
|
||||
|
||||
Like the real readers (ServerReader, OpcManager), `subscriptions` is keyed by slot
|
||||
and `unsubscribe` drops the slot from it.
|
||||
"""
|
||||
|
||||
def make(*slots):
|
||||
reader = MagicMock()
|
||||
reader.subscriptions = {slot: MagicMock() for slot in slots}
|
||||
reader.unsubscribe = AsyncMock(side_effect=lambda slot: reader.subscriptions.pop(slot))
|
||||
return reader
|
||||
|
||||
return make
|
||||
@@ -26,123 +26,3 @@ async def test_fetch_returns_local_paths(tmp_path):
|
||||
)
|
||||
assert (tmp_path / 'client_cert_c.pem').read_bytes() == b'cert-bytes'
|
||||
assert (tmp_path / 'private_key_k.pem').read_bytes() == b'key-bytes'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lazy_minio_builds_client_once_on_first_download():
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from ingestor.connection.certs import LazyMinio
|
||||
|
||||
client = AsyncMock()
|
||||
client.download_file.return_value = b'bytes'
|
||||
factory = MagicMock(return_value=client)
|
||||
minio = LazyMinio(factory)
|
||||
|
||||
factory.assert_not_called() # no network at construction
|
||||
assert await minio.download_file('a.pem', 'bucket', {}) == b'bytes'
|
||||
await minio.download_file('b.pem', 'bucket', {})
|
||||
|
||||
factory.assert_called_once()
|
||||
assert client.download_file.await_count == 2
|
||||
|
||||
|
||||
def test_build_cert_provider_is_lazy_and_uses_config(monkeypatch):
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from ingestor.config import GatewayConfig
|
||||
from ingestor.connection import certs
|
||||
|
||||
minio_cls = MagicMock()
|
||||
monkeypatch.setattr(certs, 'MinioRepository', minio_cls)
|
||||
mongo = MagicMock()
|
||||
|
||||
provider = certs.build_cert_provider(
|
||||
mongo=mongo,
|
||||
config=GatewayConfig(cert_bucket='streamlit-connectors', cert_cache_dir='certs-cache'),
|
||||
logger=MagicMock(),
|
||||
notification_handler=MagicMock(),
|
||||
metrics_controller=MagicMock(),
|
||||
)
|
||||
|
||||
assert provider.mongo is mongo
|
||||
assert provider.bucket == 'streamlit-connectors'
|
||||
assert provider.cache_dir == 'certs-cache'
|
||||
assert isinstance(provider.minio, certs.LazyMinio)
|
||||
minio_cls.assert_not_called() # MinIO is only reached when a cert is needed
|
||||
|
||||
|
||||
def _provider(tmp_path, doc=None, data=b'pem'):
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
mongo = MagicMock()
|
||||
mongo.find_one = AsyncMock(
|
||||
return_value=doc
|
||||
if doc is not None
|
||||
else {
|
||||
'client_cert': {'minio_object_name': 'platform_certificates/c.pem'},
|
||||
'private_key': {'minio_object_name': 'platform_certificates/k.pem'},
|
||||
}
|
||||
)
|
||||
minio = MagicMock()
|
||||
minio.download_file = AsyncMock(return_value=data)
|
||||
return CertProvider(mongo=mongo, minio=minio, cache_dir=str(tmp_path), bucket='b'), minio
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
'doc',
|
||||
[
|
||||
None,
|
||||
{},
|
||||
{'client_cert': {'minio_object_name': 'c.pem'}},
|
||||
{'client_cert': None, 'private_key': {}},
|
||||
],
|
||||
)
|
||||
async def test_fetch_without_platform_certificate_raises_clear_error(tmp_path, doc):
|
||||
from ingestor.connection.certs import CertificateNotConfiguredError
|
||||
|
||||
provider, minio = _provider(tmp_path)
|
||||
provider.mongo.find_one.return_value = doc
|
||||
|
||||
with pytest.raises(CertificateNotConfiguredError, match='platform_certificates'):
|
||||
await provider.fetch()
|
||||
minio.download_file.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_is_cached_until_invalidated(tmp_path):
|
||||
provider, minio = _provider(tmp_path)
|
||||
|
||||
first = await provider.fetch()
|
||||
second = await provider.fetch()
|
||||
assert first == second
|
||||
assert minio.download_file.await_count == 2 # cert + key, once
|
||||
|
||||
provider.invalidate()
|
||||
await provider.fetch()
|
||||
assert minio.download_file.await_count == 4
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_concurrent_fetches_download_once(tmp_path):
|
||||
import asyncio
|
||||
|
||||
provider, minio = _provider(tmp_path)
|
||||
|
||||
results = await asyncio.gather(*(provider.fetch() for _ in range(5)))
|
||||
|
||||
assert len(set(results)) == 1
|
||||
assert minio.download_file.await_count == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_replaces_file_atomically(tmp_path):
|
||||
provider, minio = _provider(tmp_path, data=b'new')
|
||||
(tmp_path / 'c.pem').write_bytes(b'old')
|
||||
|
||||
cert_path, _ = await provider.fetch()
|
||||
|
||||
assert (tmp_path / 'c.pem').read_bytes() == b'new'
|
||||
assert cert_path == str(tmp_path / 'c.pem')
|
||||
assert not list(tmp_path.glob('*.tmp'))
|
||||
|
||||
@@ -182,183 +182,3 @@ async def test_concurrent_get_same_server_connects_once():
|
||||
assert c1 is c2 is clients[0]
|
||||
assert ctor.call_count == 1
|
||||
assert clients[0].connect.await_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_release_idle_keeps_pinned_connection():
|
||||
"""A subscribed connection only gets callbacks (never get()), so it looks idle:
|
||||
the pin keeps release_idle from dropping it (SIENTIAPDE-1974 G4)."""
|
||||
pool = make_pool()
|
||||
fake_client = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
|
||||
pool.pin('srv')
|
||||
await pool.release_idle(max_idle_s=0)
|
||||
|
||||
assert 'srv' in pool._connections
|
||||
fake_client.disconnect.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_release_idle_evicts_after_last_unpin():
|
||||
pool = make_pool()
|
||||
fake_client = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
|
||||
pool.pin('srv')
|
||||
pool.pin('srv') # two slots subscribed on the same server
|
||||
pool.unpin('srv')
|
||||
await pool.release_idle(max_idle_s=0)
|
||||
assert 'srv' in pool._connections # one subscription still active
|
||||
|
||||
pool.unpin('srv')
|
||||
await pool.release_idle(max_idle_s=0)
|
||||
assert 'srv' not in pool._connections
|
||||
fake_client.disconnect.assert_awaited_once()
|
||||
|
||||
|
||||
def test_unpin_never_goes_negative():
|
||||
pool = make_pool()
|
||||
|
||||
pool.unpin('srv')
|
||||
pool.pin('srv')
|
||||
|
||||
assert pool._pins == {'srv': 1}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_evict_disconnects_now_and_next_get_reconnects():
|
||||
pool = make_pool()
|
||||
first, second = AsyncMock(), AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', side_effect=[first, second]):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.evict('srv')
|
||||
client = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
|
||||
first.disconnect.assert_awaited_once()
|
||||
assert client is second
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_evict_unknown_server_is_noop():
|
||||
pool = make_pool()
|
||||
|
||||
await pool.evict('nope')
|
||||
|
||||
pool.emit_metric.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_normalizes_server_id_to_str():
|
||||
pool = make_pool()
|
||||
fake_client = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
c1 = await pool.get(1, {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
c2 = await pool.get('1', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
|
||||
assert c1 is c2
|
||||
assert fake_client.connect.await_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_with_different_cfg_reuses_client_and_warns():
|
||||
pool = make_pool()
|
||||
fake_client = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
client = await pool.get('srv', {'url': 'opc.tcp://10.0.0.5:4840', 'use_security': False})
|
||||
|
||||
assert client is fake_client
|
||||
assert fake_client.connect.await_count == 1
|
||||
pool.logger.warning.assert_called_once()
|
||||
assert 'url' in pool.logger.warning.call_args.args[0]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_secured_connect_failure_invalidates_cert_cache():
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
cert_provider = MagicMock()
|
||||
cert_provider.fetch = AsyncMock(return_value=('cert.pem', 'key.pem'))
|
||||
pool = make_pool(cert_provider=cert_provider)
|
||||
fake_client = AsyncMock()
|
||||
fake_client.connect.side_effect = RuntimeError('BadCertificateUntrusted')
|
||||
with (
|
||||
patch('ingestor.connection.pool.Client', return_value=fake_client),
|
||||
patch('ingestor.connection.pool.apply_security', AsyncMock()),
|
||||
):
|
||||
with pytest.raises(RuntimeError):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': True})
|
||||
|
||||
cert_provider.invalidate.assert_called_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_insecure_connect_failure_keeps_cert_cache():
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
cert_provider = MagicMock()
|
||||
pool = make_pool(cert_provider=cert_provider)
|
||||
fake_client = AsyncMock()
|
||||
fake_client.connect.side_effect = RuntimeError('unreachable')
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
with pytest.raises(RuntimeError):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
|
||||
cert_provider.invalidate.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_connect_identifies_the_pod_on_the_server():
|
||||
"""Session/application name = pod id, as the legacy OpcManager did, so each pod's
|
||||
session is recognizable in the KEPServer console."""
|
||||
pool = make_pool()
|
||||
fake_client = MagicMock(connect=AsyncMock())
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
|
||||
assert fake_client.name == pool.pod_id
|
||||
assert fake_client.description == pool.pod_id
|
||||
assert fake_client.product_uri == pool.pod_id.replace('-', ':')
|
||||
assert fake_client.application_uri == pool.pod_id.replace('-', ':')
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_connect_server_uri_still_wins_for_application_uri():
|
||||
pool = make_pool()
|
||||
fake_client = MagicMock(connect=AsyncMock())
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get(
|
||||
'srv',
|
||||
{
|
||||
'url': 'opc.tcp://h:4840',
|
||||
'use_security': False,
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
},
|
||||
)
|
||||
|
||||
# must match the certificate SAN URI (BadCertificateUriInvalid otherwise)
|
||||
assert fake_client.application_uri == 'urn:sientia:opc-ingestor'
|
||||
assert fake_client.name == pool.pod_id
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_config_for_returns_a_copy_of_the_live_connection_settings():
|
||||
pool = make_pool()
|
||||
cfg = {'url': 'opc.tcp://h:4840', 'server_uri': 'urn:h', 'use_security': False}
|
||||
with patch('ingestor.connection.pool.Client', return_value=AsyncMock()):
|
||||
await pool.get(1, cfg)
|
||||
|
||||
found = pool.config_for('1')
|
||||
assert found == cfg
|
||||
found['url'] = 'mutated'
|
||||
assert pool.config_for(1)['url'] == 'opc.tcp://h:4840'
|
||||
|
||||
await pool.evict('1')
|
||||
assert pool.config_for('1') is None
|
||||
|
||||
|
||||
def test_config_for_unknown_server_is_none():
|
||||
assert make_pool().config_for('missing') is None
|
||||
|
||||
@@ -1,151 +0,0 @@
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from ingestor.connection.servers import CONNECTORS, OPC_SERVERS, ServerConfigResolver
|
||||
|
||||
CONNECTOR = {
|
||||
'connector_id': 1,
|
||||
'type': 'opc_ua',
|
||||
'status': 'Active',
|
||||
'name': 'kepserver',
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'server_uri': 'urn:kepserver',
|
||||
'use_security': True,
|
||||
}
|
||||
OPC_SERVER = {
|
||||
'id': '1',
|
||||
'server_name': 'kepserver',
|
||||
'url': 'opc.tcp://legacy:49320',
|
||||
'uri': 'urn:legacy',
|
||||
'cert_path': '/app/certs/cert.pem',
|
||||
}
|
||||
|
||||
|
||||
def make_resolver(connector=None, opc_server=None, pooled=None, ttl_s=60.0):
|
||||
pool = MagicMock()
|
||||
pool.config_for = MagicMock(return_value=pooled)
|
||||
mongo = MagicMock()
|
||||
|
||||
async def find_one(collection, _filters, _metadata):
|
||||
return {CONNECTORS: connector, OPC_SERVERS: opc_server}[collection]
|
||||
|
||||
mongo.find_one = AsyncMock(side_effect=find_one)
|
||||
return ServerConfigResolver(pool, mongo, MagicMock(), ttl_s=ttl_s)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_command_connection_overrides_every_other_source():
|
||||
resolver = make_resolver(connector=CONNECTOR)
|
||||
|
||||
cfg = await resolver.resolve('1', {'url': 'opc.tcp://h:1', 'server_uri': 'urn:h'})
|
||||
|
||||
assert cfg == {'url': 'opc.tcp://h:1', 'server_uri': 'urn:h', 'use_security': False}
|
||||
resolver.mongo.find_one.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_live_pooled_connection_wins_over_mongo():
|
||||
"""Writes reuse exactly the settings the read path opened the connection with."""
|
||||
pooled = {'url': 'opc.tcp://read:1', 'server_uri': 'urn:read', 'use_security': True}
|
||||
resolver = make_resolver(connector=CONNECTOR, pooled=pooled)
|
||||
|
||||
assert await resolver.resolve('1', {}) == pooled
|
||||
resolver.pool.config_for.assert_called_once_with('1')
|
||||
resolver.mongo.find_one.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_tag_database_connector_first():
|
||||
resolver = make_resolver(connector=CONNECTOR, opc_server=OPC_SERVER)
|
||||
|
||||
cfg = await resolver.resolve('1', {})
|
||||
|
||||
assert cfg == {
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'server_uri': 'urn:kepserver',
|
||||
'use_security': True,
|
||||
}
|
||||
collection, filters, _ = resolver.mongo.find_one.await_args_list[0].args
|
||||
assert collection == CONNECTORS
|
||||
# connector_id is an int in the tag database; the command carries a string.
|
||||
assert filters == {'connector_id': {'$in': ['1', 1]}, 'type': 'opc_ua', 'status': 'Active'}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_opc_servers_fallback_without_tag_database():
|
||||
resolver = make_resolver(connector=None, opc_server=OPC_SERVER)
|
||||
|
||||
cfg = await resolver.resolve('1', {})
|
||||
|
||||
# Same rule as the orchestrator's read slot: a cert_path means security.
|
||||
assert cfg == {
|
||||
'url': 'opc.tcp://legacy:49320',
|
||||
'server_uri': 'urn:legacy',
|
||||
'use_security': True,
|
||||
}
|
||||
collection, filters, _ = resolver.mongo.find_one.await_args_list[1].args
|
||||
assert collection == OPC_SERVERS
|
||||
assert filters == {'id': {'$in': ['1', 1]}}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_opc_server_without_cert_path_is_insecure():
|
||||
resolver = make_resolver(opc_server={**OPC_SERVER, 'cert_path': ''})
|
||||
|
||||
assert (await resolver.resolve('1', {}))['use_security'] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_non_numeric_server_id_is_looked_up_as_is():
|
||||
resolver = make_resolver(opc_server={**OPC_SERVER, 'id': 'kep-a'})
|
||||
|
||||
await resolver.resolve('kep-a', {})
|
||||
|
||||
_, filters, _ = resolver.mongo.find_one.await_args_list[0].args
|
||||
assert filters['connector_id'] == {'$in': ['kep-a']}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unknown_server_is_cached_briefly():
|
||||
resolver = make_resolver(ttl_s=60.0)
|
||||
resolver.missing_ttl_s = 10.0
|
||||
resolver.clock = MagicMock()
|
||||
resolver.clock.elapsed.side_effect = [0.0, 5.0, 11.0]
|
||||
|
||||
assert await resolver.resolve('9', {}) is None
|
||||
assert await resolver.resolve('9', {}) is None # within missing_ttl_s: no lookup
|
||||
assert await resolver.resolve('9', {}) is None # expired: looked up again
|
||||
assert resolver.mongo.find_one.await_count == 4 # 2 lookups (connectors + opc-servers)
|
||||
assert resolver.logger.warning.call_count == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_documents_without_url_are_ignored():
|
||||
resolver = make_resolver(connector={**CONNECTOR, 'url': ''}, opc_server=OPC_SERVER)
|
||||
|
||||
assert (await resolver.resolve('1', {}))['url'] == 'opc.tcp://legacy:49320'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lookups_are_cached_for_the_ttl():
|
||||
resolver = make_resolver(connector=CONNECTOR, ttl_s=60.0)
|
||||
resolver.clock = MagicMock()
|
||||
resolver.clock.elapsed.side_effect = [0.0, 30.0, 61.0]
|
||||
|
||||
first = await resolver.resolve('1', {})
|
||||
first['url'] = 'mutated by the caller'
|
||||
second = await resolver.resolve('1', {}) # within the TTL: cached, and a copy
|
||||
await resolver.resolve('1', {}) # expired: looked up again
|
||||
|
||||
assert second['url'] == 'opc.tcp://kepserver:49320'
|
||||
assert resolver.mongo.find_one.await_count == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mongo_errors_propagate_to_the_caller():
|
||||
resolver = make_resolver()
|
||||
resolver.mongo.find_one = AsyncMock(side_effect=RuntimeError('mongo down'))
|
||||
|
||||
with pytest.raises(RuntimeError, match='mongo down'):
|
||||
await resolver.resolve('1', {})
|
||||
@@ -201,100 +201,3 @@ async def test_check_opc_listenning_triggers_reconnect_at_15():
|
||||
|
||||
assert result is True
|
||||
assert eng.send_notification_async.call_count == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_subscription_pins_pool_connection():
|
||||
eng = make_engine()
|
||||
eng.pool = MagicMock(get=AsyncMock(return_value=AsyncMock()))
|
||||
|
||||
await eng.create_subscription('1', {'subscription_period_ms': 500}, 'slot1')
|
||||
|
||||
eng.pool.pin.assert_called_once_with('1')
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_subscription_failure_does_not_pin():
|
||||
eng = make_engine()
|
||||
client = AsyncMock()
|
||||
client.create_subscription.side_effect = RuntimeError('boom')
|
||||
eng.pool = MagicMock(get=AsyncMock(return_value=client))
|
||||
|
||||
with pytest.raises(RuntimeError):
|
||||
await eng.create_subscription('1', {'subscription_period_ms': 500}, 'slot1')
|
||||
|
||||
eng.pool.pin.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unsubscribe_unpins_pool_connection():
|
||||
eng = make_engine()
|
||||
eng.pool = MagicMock(get=AsyncMock(return_value=AsyncMock()))
|
||||
await eng.create_subscription('1', {'subscription_period_ms': 500}, 'slot1')
|
||||
|
||||
await eng.unsubscribe('slot1')
|
||||
|
||||
eng.pool.unpin.assert_called_once_with('1')
|
||||
assert 'slot1' not in eng.subscriptions
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unsubscribe_tolerates_dead_connection():
|
||||
"""A lost server's subscription can't be deleted on the server side; it must
|
||||
still be dropped locally (and unpinned) so the slot can be re-subscribed."""
|
||||
eng = make_engine()
|
||||
sub = AsyncMock()
|
||||
sub.delete.side_effect = ConnectionError('closed')
|
||||
client = AsyncMock()
|
||||
client.create_subscription.return_value = sub
|
||||
eng.pool = MagicMock(get=AsyncMock(return_value=client))
|
||||
await eng.create_subscription('1', {'subscription_period_ms': 500}, 'slot1')
|
||||
|
||||
await eng.unsubscribe('slot1') # must not raise
|
||||
|
||||
assert 'slot1' not in eng.subscriptions
|
||||
eng.pool.unpin.assert_called_once_with('1')
|
||||
eng.logger.warning.assert_called_once()
|
||||
|
||||
|
||||
async def _subscribed_engine(slots):
|
||||
"""Engine with one subscription per slot: {slot: [addresses]}."""
|
||||
eng = make_engine()
|
||||
eng.pool = MagicMock(
|
||||
get=AsyncMock(
|
||||
return_value=MagicMock(create_subscription=AsyncMock(return_value=AsyncMock()))
|
||||
)
|
||||
)
|
||||
for slot, addresses in slots.items():
|
||||
await eng.create_subscription('1', {'subscription_period_ms': 500}, slot)
|
||||
nodes = {
|
||||
a: {'frequency': '1000', 'names': [{'collection': 'c', 'tag_name': a}]}
|
||||
for a in addresses
|
||||
}
|
||||
await eng.subscribe(slot, nodes, 10)
|
||||
return eng
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unsubscribe_forgets_the_slot_nodes():
|
||||
eng = await _subscribed_engine({'s1': ['ns=2;i=1', 'ns=2;i=2'], 's2': ['ns=2;i=3']})
|
||||
|
||||
await eng.unsubscribe('s1')
|
||||
|
||||
# no more cycle checks / tag counting for addresses nobody reads anymore
|
||||
assert set(eng.nodes) == {'ns=2;i=3'}
|
||||
eng.emit_metric.assert_any_await(
|
||||
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
|
||||
method='set',
|
||||
value=1,
|
||||
tags={'pod_id': eng.pod_id, 'server_name': '1'},
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unsubscribe_keeps_node_shared_with_another_slot():
|
||||
eng = await _subscribed_engine({'s1': ['ns=2;i=1', 'ns=2;i=2'], 's2': ['ns=2;i=2']})
|
||||
|
||||
await eng.unsubscribe('s1')
|
||||
|
||||
assert set(eng.nodes) == {'ns=2;i=2'}
|
||||
|
||||
@@ -7,17 +7,8 @@ from asyncua import ua
|
||||
import ingestor.metrics as metrics
|
||||
from ingestor.engines.write import WriteEngine
|
||||
|
||||
SERVER_CFG = {'url': 'opc.tcp://kepserver:49320', 'server_uri': 'urn:kep', 'use_security': True}
|
||||
|
||||
|
||||
def make_resolver(cfg=SERVER_CFG):
|
||||
resolver = MagicMock()
|
||||
resolver.resolve = AsyncMock(return_value=cfg)
|
||||
return resolver
|
||||
|
||||
|
||||
def make_engine(**kwargs):
|
||||
kwargs.setdefault('servers', make_resolver())
|
||||
eng = WriteEngine(
|
||||
pool=AsyncMock(),
|
||||
sink=AsyncMock(),
|
||||
@@ -266,104 +257,3 @@ async def test_no_rate_limit_when_min_pause_is_zero():
|
||||
await eng.write(base_cmd())
|
||||
await eng.write(base_cmd())
|
||||
sleep_mock.assert_not_awaited()
|
||||
|
||||
|
||||
# --- SIENTIAPDE-1975: server resolution (GW1) and error reporting (GW4) ---
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_uses_the_resolved_server_connection():
|
||||
"""The C2 command of the Laborious carries only the server_id."""
|
||||
eng = make_engine()
|
||||
eng.pool.get.return_value = make_client()
|
||||
|
||||
await eng.write(base_cmd())
|
||||
|
||||
eng.servers.resolve.assert_awaited_once_with('srv1', base_cmd())
|
||||
eng.pool.get.assert_awaited_once_with('srv1', SERVER_CFG)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_unknown_server_records_error_without_connecting():
|
||||
eng = make_engine(servers=make_resolver(cfg=None))
|
||||
|
||||
result = await eng.write(base_cmd())
|
||||
|
||||
assert result.status == 'error'
|
||||
assert result.opc_status_code == 'server_not_found'
|
||||
eng.pool.get.assert_not_awaited()
|
||||
eng.sink.update_write_status.assert_awaited_once_with(base_cmd(), result)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_server_lookup_failure_records_error():
|
||||
resolver = make_resolver()
|
||||
resolver.resolve.side_effect = RuntimeError('mongo down')
|
||||
eng = make_engine(servers=resolver)
|
||||
|
||||
result = await eng.write(base_cmd())
|
||||
|
||||
assert result.status == 'error'
|
||||
assert result.opc_status_code == 'server_lookup'
|
||||
assert 'mongo down' in result.message
|
||||
eng.pool.get.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_without_resolver_uses_the_command_connection():
|
||||
eng = make_engine(servers=None)
|
||||
eng.pool.get.return_value = make_client()
|
||||
|
||||
await eng.write(base_cmd(url='opc.tcp://h:4840', server_uri='urn:h', use_security=True))
|
||||
|
||||
eng.pool.get.assert_awaited_once_with(
|
||||
'srv1', {'url': 'opc.tcp://h:4840', 'use_security': True, 'server_uri': 'urn:h'}
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_without_resolver_or_url_is_server_not_found():
|
||||
eng = make_engine(servers=None)
|
||||
|
||||
result = await eng.write(base_cmd())
|
||||
|
||||
assert result.opc_status_code == 'server_not_found'
|
||||
eng.pool.get.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_error_logs_the_reason_and_sets_the_model_status():
|
||||
eng = make_engine()
|
||||
node = make_node()
|
||||
node.write_value.side_effect = RuntimeError('BadWriteNotSupported')
|
||||
eng.pool.get.return_value = make_client(node)
|
||||
|
||||
await eng.write(base_cmd())
|
||||
|
||||
logged = eng.logger.error.call_args.args[0]
|
||||
assert 'RuntimeError' in logged and 'BadWriteNotSupported' in logged
|
||||
assert 'addr=ns=2;i=5' in logged and 'model_id=m1' in logged
|
||||
status_call = next(
|
||||
c
|
||||
for c in eng.emit_metric.await_args_list
|
||||
if c.kwargs['metric_object'] is metrics.OPC_WRITE_STATUS_BY_MODEL
|
||||
)
|
||||
# Same series as a success, so it shows the last outcome.
|
||||
assert status_call.kwargs['value'] == 0
|
||||
assert status_call.kwargs['tags'] == {'pod_id': eng.pod_id, 'model_id': 'm1', 'status': 'ok'}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize('timestamp', [None, 'not-a-timestamp'])
|
||||
async def test_write_bad_timestamp_records_error_without_raising(timestamp):
|
||||
eng = make_engine(write_timestamp=True)
|
||||
cmd = base_cmd(timestamp=timestamp)
|
||||
if timestamp is None:
|
||||
del cmd['timestamp']
|
||||
|
||||
result = await eng.write(cmd)
|
||||
|
||||
assert result.status == 'error'
|
||||
assert result.opc_status_code == 'bad_timestamp'
|
||||
eng.pool.get.assert_not_awaited()
|
||||
eng.sink.update_write_status.assert_awaited_once()
|
||||
|
||||
@@ -67,26 +67,3 @@ async def test_publish_reading_emits_tag_value_metric_per_name():
|
||||
value=42,
|
||||
tags={'pod_id': dm.pod_id, 'tag_name': 'Counter', 'model_id': '1'},
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_publish_reading_without_model_id_keeps_legacy_doc():
|
||||
"""Names converted from a slot v1 carry no model_id: the document must be the
|
||||
same as the legacy OpcManager one (SIENTIAPDE-1974 NF2)."""
|
||||
sink, dm = make_sink()
|
||||
names = [{'collection': 'raw_x', 'tag_name': 'Counter'}]
|
||||
|
||||
await sink.publish_reading(
|
||||
tag='ns=2;i=2', value=42, timestamp='2026-07-13 00:00:00+0000', names=names
|
||||
)
|
||||
|
||||
collection, doc = dm.publish.await_args_list[0].args
|
||||
assert collection == 'raw_x'
|
||||
assert doc == {
|
||||
'tag': 'ns=2;i=2',
|
||||
'name': 'Counter',
|
||||
'timestamp': '2026-07-13 00:00:00+0000',
|
||||
'value': 42,
|
||||
}
|
||||
_, kwargs = sink.emit_metric.await_args
|
||||
assert kwargs['tags']['model_id'] == ''
|
||||
|
||||
@@ -1,149 +1,89 @@
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
import ingestor.metrics as metrics
|
||||
from ingestor.engines.write import WriteResult
|
||||
from ingestor.io.data_sink import DataSink, write_status_statements
|
||||
|
||||
CMD = {
|
||||
'server_id': '1',
|
||||
'addr': 'ns=2;s=Pred',
|
||||
'schema': 'sientia_data',
|
||||
'table_name': 'pred_m1',
|
||||
'model_id': '1',
|
||||
'timestamp': '2026-07-13T00:00:00Z',
|
||||
}
|
||||
from ingestor.io.data_sink import DataSink
|
||||
|
||||
|
||||
def _result(status):
|
||||
return WriteResult(ok=status == 'ok', status=status, opc_status_code='x', message='')
|
||||
|
||||
|
||||
def make_sink(rowcount=1, existing_row=True):
|
||||
def make_sink():
|
||||
postgres = MagicMock()
|
||||
connection = MagicMock()
|
||||
connection.execute.return_value.rowcount = rowcount
|
||||
connection.execute.return_value.first.return_value = (1,) if existing_row else None
|
||||
postgres.engine.begin.return_value.__enter__.return_value = connection
|
||||
data_manager = MagicMock()
|
||||
data_manager.emit_metric = AsyncMock()
|
||||
sink = DataSink(data_manager=data_manager, postgres=postgres)
|
||||
sink = DataSink(data_manager=MagicMock(), postgres=postgres)
|
||||
return sink, connection
|
||||
|
||||
|
||||
def _outcome_metric(sink):
|
||||
calls = [
|
||||
c
|
||||
for c in sink.data_manager.emit_metric.await_args_list
|
||||
if c.kwargs['metric_object'] is metrics.OPC_WRITE_STATUS_UPDATE_TOTAL
|
||||
]
|
||||
assert len(calls) == 1
|
||||
return calls[0].kwargs['tags']['result']
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_write_status_targets_model_row_with_precedence():
|
||||
async def test_update_write_status_targets_model_row():
|
||||
sink, connection = make_sink()
|
||||
cmd = {
|
||||
'schema': 'sientia_data',
|
||||
'table_name': 'pred_m1',
|
||||
'model_id': '1',
|
||||
'timestamp': '2026-07-13T00:00:00Z',
|
||||
}
|
||||
result = WriteResult(ok=True, status='ok', opc_status_code='Good', message='')
|
||||
|
||||
outcome = await sink.update_write_status(CMD, _result('ok'))
|
||||
await sink.update_write_status(cmd, result)
|
||||
|
||||
assert outcome == 'updated'
|
||||
assert connection.execute.call_count == 1
|
||||
(stmt,) = connection.execute.call_args[0]
|
||||
compiled = stmt.compile(dialect=postgresql.dialect())
|
||||
sql = str(compiled)
|
||||
assert sql.startswith('UPDATE sientia_data.pred_m1 SET write_status=')
|
||||
assert 'CASE sientia_data.pred_m1.write_status WHEN' in sql
|
||||
stmt, params = connection.execute.call_args[0]
|
||||
sql = str(stmt)
|
||||
assert 'update' in sql.lower() and 'write_status' in sql.lower()
|
||||
assert 'sientia_data.pred_m1' in sql
|
||||
# values must be bound params, never interpolated into the SQL text itself
|
||||
assert '2026-07-13T00:00:00Z' not in sql and "'ok'" not in sql
|
||||
assert compiled.params['write_status'] == 'ok'
|
||||
assert '2026-07-13T00:00:00Z' in compiled.params.values()
|
||||
assert _outcome_metric(sink) == 'updated'
|
||||
assert params == {'status': 'ok', 'timestamp': '2026-07-13T00:00:00Z', 'model_id': '1'}
|
||||
assert 'ok' not in sql
|
||||
assert '2026-07-13T00:00:00Z' not in sql
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_write_status_kept_when_the_row_has_a_worse_outcome():
|
||||
sink, _ = make_sink(rowcount=0, existing_row=True)
|
||||
|
||||
assert await sink.update_write_status(CMD, _result('ok')) == 'kept'
|
||||
assert _outcome_metric(sink) == 'kept'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_write_status_row_not_found_is_logged():
|
||||
"""E.g. the Laborious INSERT failed and it wrote to OPC anyway (SIENTIAPDE-1975 D10)."""
|
||||
sink, _ = make_sink(rowcount=0, existing_row=False)
|
||||
|
||||
assert await sink.update_write_status(CMD, _result('error')) == 'row_not_found'
|
||||
assert 'no prediction row' in sink.data_manager.logger.warning.call_args.args[0]
|
||||
assert _outcome_metric(sink) == 'row_not_found'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_write_status_database_error_is_not_raised():
|
||||
async def test_update_write_status_rejects_bad_schema():
|
||||
sink, connection = make_sink()
|
||||
connection.execute.side_effect = RuntimeError('column "write_status" does not exist')
|
||||
cmd = {
|
||||
'schema': 'a; DROP TABLE x',
|
||||
'table_name': 'pred_m1',
|
||||
'model_id': '1',
|
||||
'timestamp': '2026-07-13T00:00:00Z',
|
||||
}
|
||||
result = WriteResult(ok=False, status='error', opc_status_code='Bad', message='x')
|
||||
|
||||
assert await sink.update_write_status(CMD, _result('ok')) == 'error'
|
||||
assert 'write_status' in sink.data_manager.logger.warning.call_args.args[0]
|
||||
assert _outcome_metric(sink) == 'error'
|
||||
with pytest.raises(ValueError):
|
||||
await sink.update_write_status(cmd, result)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
('field', 'value'),
|
||||
[
|
||||
('schema', 'a; DROP TABLE x'),
|
||||
('schema', 'sientia_data\n'),
|
||||
('table_name', 'pred_m1; DROP TABLE x'),
|
||||
('schema', None),
|
||||
('timestamp', None),
|
||||
('model_id', None),
|
||||
],
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_write_status_rejects_invalid_commands(field, value):
|
||||
sink, connection = make_sink()
|
||||
|
||||
outcome = await sink.update_write_status({**CMD, field: value}, _result('error'))
|
||||
|
||||
assert outcome == 'invalid'
|
||||
connection.execute.assert_not_called()
|
||||
assert _outcome_metric(sink) == 'invalid'
|
||||
|
||||
|
||||
def _apply(initial, answers):
|
||||
"""Run the real statements on SQLite for each answer; return the final status."""
|
||||
engine = create_engine('sqlite://')
|
||||
with engine.begin() as connection:
|
||||
connection.execute(
|
||||
text('CREATE TABLE pred (model_id INTEGER, timestamp TEXT, write_status TEXT)')
|
||||
)
|
||||
connection.execute(
|
||||
text('INSERT INTO pred VALUES (1, :ts, :status)'),
|
||||
{'ts': '2026-07-13T00:00:00Z', 'status': initial},
|
||||
)
|
||||
for answer in answers:
|
||||
update_stmt, exists_stmt = write_status_statements(
|
||||
None, 'pred', answer, '2026-07-13T00:00:00Z', 1
|
||||
)
|
||||
connection.execute(update_stmt)
|
||||
assert connection.execute(exists_stmt).first() is not None
|
||||
return connection.execute(text('SELECT write_status FROM pred')).scalar()
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_write_status_rejects_trailing_newline_identifier():
|
||||
sink, connection = make_sink()
|
||||
cmd = {
|
||||
'schema': 'sientia_data\n',
|
||||
'table_name': 'pred_m1',
|
||||
'model_id': '1',
|
||||
'timestamp': '2026-07-13T00:00:00Z',
|
||||
}
|
||||
result = WriteResult(ok=True, status='ok', opc_status_code='Good', message='')
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
await sink.update_write_status(cmd, result)
|
||||
|
||||
connection.execute.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
('initial', 'answers', 'expected'),
|
||||
[
|
||||
('pending', ['ok'], 'ok'),
|
||||
('pending', ['error', 'ok'], 'error'),
|
||||
('pending', ['ok', 'error'], 'error'),
|
||||
('pending', ['session_bad', 'error', 'ok'], 'session_bad'),
|
||||
('error', ['ok'], 'error'),
|
||||
(None, ['ok'], 'ok'),
|
||||
],
|
||||
)
|
||||
def test_update_statement_keeps_the_worst_answer(initial, answers, expected):
|
||||
assert _apply(initial, answers) == expected
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_write_status_rejects_bad_table_name():
|
||||
sink, connection = make_sink()
|
||||
cmd = {
|
||||
'schema': 'sientia_data',
|
||||
'table_name': 'pred_m1; DROP TABLE x',
|
||||
'model_id': '1',
|
||||
'timestamp': '2026-07-13T00:00:00Z',
|
||||
}
|
||||
result = WriteResult(ok=False, status='error', opc_status_code='Bad', message='x')
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
await sink.update_write_status(cmd, result)
|
||||
|
||||
connection.execute.assert_not_called()
|
||||
|
||||
@@ -62,102 +62,3 @@ def test_parse_slot_rejects_tag_without_names():
|
||||
|
||||
with pytest.raises((KeyError, ValueError)):
|
||||
parse_slot(raw)
|
||||
|
||||
|
||||
RAW_V1 = {
|
||||
'kepserver': {
|
||||
'name': 'kepserver',
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert_path': '/app/certs/opc-ingestor-cert.pem',
|
||||
'private_key_path': '/app/certs/opc-ingestor-key.pem',
|
||||
'server_cert_path': None,
|
||||
'subscription_period_ms': 15000.0,
|
||||
'tags': {
|
||||
'ns=2;s=A.TargetSin': {
|
||||
'aggr_func': 'lts',
|
||||
'frequency': 30000,
|
||||
'server_id': '1',
|
||||
'tag_address': 'ns=2;s=A.TargetSin',
|
||||
'tag_name': 'TargetSin',
|
||||
'topics': ['raw_scouter-sin-approx', 'raw_other-model'],
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def test_parse_slot_converts_v1_topics_to_names():
|
||||
"""Slot v1 (current Orchestrator) is read as v2: one name per topic, no model_id,
|
||||
so the documents stay identical to the legacy ones (SIENTIAPDE-1974 G1)."""
|
||||
server = parse_slot(RAW_V1)['kepserver']
|
||||
tag = server['tags']['ns=2;s=A.TargetSin']
|
||||
|
||||
assert tag['names'] == [
|
||||
{'collection': 'raw_scouter-sin-approx', 'tag_name': 'TargetSin'},
|
||||
{'collection': 'raw_other-model', 'tag_name': 'TargetSin'},
|
||||
]
|
||||
assert tag['frequency'] == 30000
|
||||
assert tag['active_read'] == {'enabled': False}
|
||||
|
||||
|
||||
def test_parse_slot_v1_cert_path_means_security_but_path_is_dropped():
|
||||
server = parse_slot(RAW_V1)['kepserver']
|
||||
|
||||
assert server['use_security'] is True
|
||||
assert server['name'] == 'kepserver'
|
||||
assert server['server_id'] == '1'
|
||||
assert server['subscription_period_ms'] == 15000.0
|
||||
# the certificate always comes from MinIO now: no file path survives the parse
|
||||
assert 'cert_path' not in server
|
||||
assert 'private_key_path' not in server
|
||||
|
||||
|
||||
def test_parse_slot_v1_without_cert_path_is_insecure():
|
||||
raw = {
|
||||
's': {
|
||||
'server_id': '2',
|
||||
'url': 'u',
|
||||
'cert_path': None,
|
||||
'tags': {'ns=2;i=1': {'tag_name': 't', 'frequency': '1000', 'topics': ['raw_c']}},
|
||||
}
|
||||
}
|
||||
|
||||
server = parse_slot(raw)['s']
|
||||
|
||||
assert server['use_security'] is False
|
||||
assert server['name'] == 's' # falls back to the slot key
|
||||
|
||||
|
||||
def test_parse_slot_v2_explicit_use_security_wins():
|
||||
raw = {
|
||||
's': {
|
||||
'server_id': '1',
|
||||
'url': 'u',
|
||||
'use_security': False,
|
||||
'cert_path': '/legacy/path',
|
||||
'tags': {
|
||||
'ns=2;i=1': {
|
||||
'frequency': '1000',
|
||||
'names': [{'collection': 'c', 'tag_name': 't', 'model_id': '1'}],
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
assert parse_slot(raw)['s']['use_security'] is False
|
||||
|
||||
|
||||
def test_parse_slot_rejects_v1_tag_without_tag_name():
|
||||
raw = {
|
||||
's': {
|
||||
'server_id': '1',
|
||||
'url': 'u',
|
||||
'tags': {'ns=2;i=1': {'frequency': '1', 'topics': ['c']}},
|
||||
}
|
||||
}
|
||||
import pytest
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
parse_slot(raw)
|
||||
|
||||
@@ -1,16 +1,10 @@
|
||||
import gc
|
||||
from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
|
||||
from unittest.mock import AsyncMock, MagicMock, call, patch
|
||||
|
||||
from pytest import fixture, mark
|
||||
from sientia_do.notifications.models import NotificationLevel
|
||||
|
||||
import ingestor.metrics as metrics
|
||||
from ingestor.config import GatewayConfig
|
||||
from ingestor.connection.pool import OpcConnectionPool
|
||||
from ingestor.io.slots import parse_slot
|
||||
from ingestor.managers.ingestor_manager import IngestorManager
|
||||
from ingestor.managers.opc_manager import OpcManager
|
||||
from ingestor.managers.server_reader import ServerReader
|
||||
|
||||
metadata = {
|
||||
'metadata': {
|
||||
@@ -22,21 +16,6 @@ metadata = {
|
||||
}
|
||||
|
||||
|
||||
def raw_slot_v1(tag_name='Counter', address='ns=2;i=2'):
|
||||
"""A slot as the current (v1) Orchestrator writes it to Redis."""
|
||||
return {
|
||||
'server1': {
|
||||
'name': 'server1',
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://localhost:4840',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert_path': '/app/certs/opc-ingestor-cert.pem',
|
||||
'subscription_period_ms': 500.0,
|
||||
'tags': {address: {'tag_name': tag_name, 'frequency': '1000', 'topics': ['raw_a']}},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@fixture(autouse=True)
|
||||
def _collect_garbage_after_test():
|
||||
# ponytail: IngestorManager.__del__ calls asyncio.run(self.shutdown()); if a
|
||||
@@ -73,12 +52,12 @@ def ingestor_manager(data_manager_mock, resource_manager_mock):
|
||||
return ingestor
|
||||
|
||||
|
||||
@patch('ingestor.managers.ingestor_manager.ServerReader')
|
||||
@patch('ingestor.managers.ingestor_manager.OpcManager')
|
||||
@patch('ingestor.managers.ingestor_manager.DataManager')
|
||||
@patch('ingestor.managers.ingestor_manager.ResourceManager')
|
||||
@patch('ingestor.managers.ingestor_manager.NotificationHandler')
|
||||
def test___init__(
|
||||
notification_handler_mock, resource_manager_mock, data_manager_mock, server_reader_mock
|
||||
notification_handler_mock, resource_manager_mock, data_manager_mock, opc_manager_mock
|
||||
):
|
||||
ingestor = IngestorManager(
|
||||
redis_data={'host': 'localhost', 'port': 6379},
|
||||
@@ -93,7 +72,7 @@ def test___init__(
|
||||
metrics_controller=MagicMock(),
|
||||
)
|
||||
|
||||
server_reader_mock.assert_not_called()
|
||||
opc_manager_mock.assert_not_called()
|
||||
data_manager_mock.assert_called_once_with(
|
||||
mongo_connection_string='mongodb://localhost:27017',
|
||||
mongo_database='sientia',
|
||||
@@ -121,86 +100,59 @@ def test___init__(
|
||||
assert ingestor.data_manager == data_manager_mock.return_value
|
||||
assert ingestor.resource_manager == resource_manager_mock.return_value
|
||||
|
||||
# Read path (SIENTIAPDE-1974): pool + read sink built here, nothing connected yet.
|
||||
assert isinstance(ingestor.pool, OpcConnectionPool)
|
||||
assert ingestor.pool.cert_provider is ingestor.cert_provider
|
||||
assert ingestor.cert_provider.mongo is data_manager_mock.return_value.mongo_repository
|
||||
assert ingestor.sink.data_manager is data_manager_mock.return_value
|
||||
assert ingestor.sink.postgres is None
|
||||
|
||||
|
||||
@patch('ingestor.managers.ingestor_manager.DataManager')
|
||||
@patch('ingestor.managers.ingestor_manager.ResourceManager')
|
||||
def test___init___cert_provider_uses_gateway_config(resource_manager_mock, data_manager_mock):
|
||||
ingestor = IngestorManager(
|
||||
redis_data={'host': 'localhost', 'port': 6379},
|
||||
lease_ttl=60,
|
||||
heartbeat_ttl=60,
|
||||
poll_interval=5,
|
||||
mongo_connection_string='mongodb://localhost:27017',
|
||||
mongo_database='sientia',
|
||||
logger=MagicMock(),
|
||||
notification_handler=MagicMock(),
|
||||
metadata=metadata['metadata'],
|
||||
metrics_controller=MagicMock(),
|
||||
gateway_config=GatewayConfig(cert_bucket='my-bucket', cert_cache_dir='certs-cache'),
|
||||
)
|
||||
|
||||
assert ingestor.cert_provider.bucket == 'my-bucket'
|
||||
assert ingestor.cert_provider.cache_dir == 'certs-cache'
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('ingestor.managers.ingestor_manager.ServerReader')
|
||||
async def test_initialize_opc_from_config(server_reader, ingestor_manager):
|
||||
@patch('ingestor.managers.ingestor_manager.OpcManager')
|
||||
async def test_initialize_opc_from_config(opc_manager, ingestor_manager):
|
||||
server_config = {
|
||||
'name': 'server1',
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://localhost:4840',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'use_security': True,
|
||||
'subscription_period_ms': 1000,
|
||||
'server_uri': 'http://opcua-server.simulator',
|
||||
'cert_path': '/path/to/cert',
|
||||
'private_key_path': '/path/to/private_key',
|
||||
'server_cert_path': '/path/to/server_cert',
|
||||
'pod_id': 'test_pod',
|
||||
}
|
||||
|
||||
server_reader.return_value = MagicMock(connect=AsyncMock())
|
||||
opc_manager.return_value = MagicMock(connect=AsyncMock())
|
||||
result = await ingestor_manager.initialize_opc_from_config(server_config)
|
||||
|
||||
server_reader.assert_called_once_with(
|
||||
opc_manager.assert_called_once_with(
|
||||
name=server_config['name'],
|
||||
server_config=server_config,
|
||||
pool=ingestor_manager.pool,
|
||||
sink=ingestor_manager.sink,
|
||||
url=server_config['url'],
|
||||
data_manager=ingestor_manager.data_manager,
|
||||
logger=ingestor_manager.logger,
|
||||
subscription_period_ms=server_config['subscription_period_ms'],
|
||||
server_uri=server_config['server_uri'],
|
||||
notification_handler=ingestor_manager.notification_handler,
|
||||
metrics_controller=ingestor_manager.metrics_controller,
|
||||
cert_path=server_config['cert_path'],
|
||||
private_key_path=server_config['private_key_path'],
|
||||
server_cert_path=server_config['server_cert_path'],
|
||||
metadata=metadata['metadata'],
|
||||
metrics_controller=ingestor_manager.metrics_controller,
|
||||
)
|
||||
|
||||
assert result == server_reader.return_value
|
||||
assert result.config == server_config
|
||||
result.connect.assert_awaited_once()
|
||||
assert result == opc_manager.return_value
|
||||
result.connect.assert_called_once()
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('ingestor.managers.ingestor_manager.ServerReader')
|
||||
@patch('ingestor.managers.ingestor_manager.OpcManager')
|
||||
@patch('ingestor.managers.ingestor_manager.traceback')
|
||||
async def test_initialize_opc_from_config_exception(
|
||||
traceback_mock, server_reader, ingestor_manager
|
||||
):
|
||||
async def test_initialize_opc_from_config_exception(traceback_mock, opc_manager, ingestor_manager):
|
||||
server_config = {
|
||||
'name': 'server1',
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://localhost:4840',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'use_security': True,
|
||||
'subscription_period_ms': 1000,
|
||||
'server_uri': 'http://opcua-server.simulator',
|
||||
'cert_path': '/path/to/cert',
|
||||
'private_key_path': '/path/to/private_key',
|
||||
'server_cert_path': '/path/to/server_cert',
|
||||
}
|
||||
|
||||
ingestor_manager.logger.error = MagicMock()
|
||||
# e.g. the server is unreachable or the certificate can't be fetched from MinIO
|
||||
server_reader.return_value = MagicMock(
|
||||
connect=AsyncMock(side_effect=Exception('Initialization error'))
|
||||
)
|
||||
opc_manager.side_effect = Exception('Initialization error')
|
||||
|
||||
result = await ingestor_manager.initialize_opc_from_config(server_config)
|
||||
|
||||
@@ -222,14 +174,11 @@ async def test_shutdown(ingestor_manager):
|
||||
ingestor_manager.opc_managers = {'server1': AsyncMock(), 'server2': AsyncMock()}
|
||||
|
||||
ingestor_manager.data_manager.shutdown = MagicMock()
|
||||
ingestor_manager.pool = MagicMock(close_all=AsyncMock())
|
||||
|
||||
await ingestor_manager.shutdown()
|
||||
|
||||
ingestor_manager.opc_managers['server1'].shutdown.assert_called_once()
|
||||
ingestor_manager.opc_managers['server2'].shutdown.assert_called_once()
|
||||
# The pool is owned by the read path: closed after the readers unsubscribe.
|
||||
ingestor_manager.pool.close_all.assert_awaited_once()
|
||||
|
||||
ingestor_manager.data_manager.shutdown.assert_called_once()
|
||||
|
||||
@@ -269,8 +218,9 @@ async def test_remove_server_not_found(ingestor_manager):
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('ingestor.managers.ingestor_manager.OpcManager')
|
||||
@patch('ingestor.managers.ingestor_manager.metrics')
|
||||
async def test_update_opc_servers(metrics, ingestor_manager):
|
||||
async def test_update_opc_servers(metrics, opc_manager, ingestor_manager):
|
||||
manager1 = MagicMock(config={'config': 'config1'})
|
||||
manager2 = MagicMock(config={'config': 'config2'})
|
||||
manager3 = MagicMock(config={'config': 'config3'})
|
||||
@@ -317,9 +267,7 @@ async def test_update_opc_servers(metrics, ingestor_manager):
|
||||
assert ingestor_manager.opc_managers['server2'].config == {'config': 'config2'}
|
||||
assert ingestor_manager.opc_managers['server3'].config == {'config': 'config3'}
|
||||
|
||||
# server4 left this pod's slots: unsubscribe only, the connection stays for writes
|
||||
ingestor_manager.remove_server.assert_any_call('server4', evict=False)
|
||||
# server5 failed to initialize: removed with the default (evict)
|
||||
ingestor_manager.remove_server.assert_any_call('server4')
|
||||
ingestor_manager.remove_server.assert_any_call('server5')
|
||||
assert ingestor_manager.remove_server.call_count == 2
|
||||
|
||||
@@ -483,47 +431,25 @@ async def test_get_number_of_slots_empty(metrics, ingestor_manager):
|
||||
@mark.asyncio
|
||||
async def test_get_slot_leases_1_success(ingestor_manager):
|
||||
ingestor_manager.resource_manager.lease_tag = AsyncMock(return_value=True)
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(return_value=raw_slot_v1())
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(return_value={'tags': ['tag1']})
|
||||
|
||||
ingestor_manager.number_of_slots = 1
|
||||
result = await ingestor_manager.get_slot_leases()
|
||||
|
||||
# Stored already normalized to slot v2 (v1 converted by parse_slot).
|
||||
assert result == {'1': parse_slot(raw_slot_v1())}
|
||||
assert ingestor_manager.managed_tags == result
|
||||
assert result == {'1': {'tags': ['tag1']}}
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_get_slot_leases_2_success(ingestor_manager):
|
||||
ingestor_manager.resource_manager.lease_tag = AsyncMock(side_effect=[True, True])
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
|
||||
side_effect=[raw_slot_v1('Counter'), raw_slot_v1('Ramp', 'ns=2;i=3')]
|
||||
side_effect=[{'tags': ['tag1']}, {'tags': ['tag2']}]
|
||||
)
|
||||
|
||||
ingestor_manager.number_of_slots = 2
|
||||
result = await ingestor_manager.get_slot_leases(max_slots=2)
|
||||
|
||||
assert result == {
|
||||
'1': parse_slot(raw_slot_v1('Counter')),
|
||||
'2': parse_slot(raw_slot_v1('Ramp', 'ns=2;i=3')),
|
||||
}
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_get_slot_leases_skips_malformed_slot(ingestor_manager):
|
||||
ingestor_manager.resource_manager.lease_tag = AsyncMock(return_value=True)
|
||||
# tag without names/topics/frequency
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
|
||||
return_value={'server1': {'server_id': '1', 'url': 'u', 'tags': {'ns=2;i=2': {}}}}
|
||||
)
|
||||
|
||||
ingestor_manager.number_of_slots = 1
|
||||
result = await ingestor_manager.get_slot_leases()
|
||||
|
||||
assert result == {}
|
||||
_, kwargs = ingestor_manager.send_notification.call_args
|
||||
assert kwargs['notification_id'] == 'OPC_SLOT_CONFIG_ERROR_1'
|
||||
assert kwargs['level'] == NotificationLevel.ERROR
|
||||
assert result == {'1': {'tags': ['tag1']}, '2': {'tags': ['tag2']}}
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@@ -551,172 +477,40 @@ async def test_get_slot_leases_1_failure(ingestor_manager):
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_unsubscribe_slot(ingestor_manager, reader_double):
|
||||
async def test_unsubscribe_slot(ingestor_manager):
|
||||
ingestor_manager.managed_tags = {
|
||||
'slot1': {'server1': {'tags': 'config1'}, 'server2': {'tags': 'config2'}},
|
||||
'slot2': {'server3': {'tags': 'config3'}, 'server1': {'tags': 'config1'}},
|
||||
}
|
||||
ingestor_manager.opc_managers = {
|
||||
'server1': reader_double('slot1', 'slot2'),
|
||||
'server2': reader_double('slot1'),
|
||||
'server3': reader_double('slot2'),
|
||||
'server1': AsyncMock(),
|
||||
'server2': AsyncMock(),
|
||||
'server3': AsyncMock(),
|
||||
}
|
||||
await ingestor_manager.unsubscribe_slot('slot1')
|
||||
|
||||
ingestor_manager.opc_managers['server1'].unsubscribe.assert_called_once_with('slot1')
|
||||
ingestor_manager.opc_managers['server2'].unsubscribe.assert_called_once_with('slot1')
|
||||
ingestor_manager.opc_managers['server3'].unsubscribe.assert_not_called()
|
||||
assert list(ingestor_manager.opc_managers['server1'].subscriptions) == ['slot2']
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_unsubscribe_slot_not_in_managed_tags(ingestor_manager, reader_double):
|
||||
"""The slot is already gone from managed_tags: the readers still holding it are unsubscribed"""
|
||||
ingestor_manager.managed_tags = {'slot1': {'server1': {'tags': 'config1'}}}
|
||||
ingestor_manager.opc_managers = {
|
||||
'server1': reader_double('slot1', 'slot3'),
|
||||
'server2': reader_double('slot3'),
|
||||
}
|
||||
|
||||
await ingestor_manager.unsubscribe_slot('slot3')
|
||||
|
||||
ingestor_manager.opc_managers['server1'].unsubscribe.assert_called_once_with('slot3')
|
||||
ingestor_manager.opc_managers['server2'].unsubscribe.assert_called_once_with('slot3')
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_unsubscribe_slot_twice_is_a_no_op(ingestor_manager, reader_double):
|
||||
"""A second unsubscribe of the same slot (lease already dropped) does nothing and never raises"""
|
||||
ingestor_manager.managed_tags = {}
|
||||
ingestor_manager.opc_managers = {'server1': reader_double('slot2')}
|
||||
|
||||
await ingestor_manager.unsubscribe_slot('slot2')
|
||||
await ingestor_manager.unsubscribe_slot('slot2')
|
||||
|
||||
ingestor_manager.opc_managers['server1'].unsubscribe.assert_called_once_with('slot2')
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_unsubscribe_slot_reader_failure_does_not_raise(ingestor_manager, reader_double):
|
||||
"""A reader that fails to unsubscribe is notified and forgotten; the others still go"""
|
||||
broken = reader_double('slot1')
|
||||
broken.unsubscribe = AsyncMock(side_effect=ConnectionError('session closed'))
|
||||
healthy = reader_double('slot1')
|
||||
ingestor_manager.opc_managers = {'server1': broken, 'server2': healthy}
|
||||
|
||||
await ingestor_manager.unsubscribe_slot('slot1')
|
||||
|
||||
healthy.unsubscribe.assert_called_once_with('slot1')
|
||||
assert 'slot1' not in broken.subscriptions
|
||||
ingestor_manager.emit_metric.assert_called_once_with(
|
||||
metric_object=metrics.OPC_SUBSCRIPTION_ERRORS,
|
||||
tags={'pod_id': ingestor_manager.pod_id, 'server': 'server1', 'slot': 'slot1'},
|
||||
)
|
||||
ingestor_manager.send_notification_async.assert_called_once_with(
|
||||
metadata=ingestor_manager.metadata,
|
||||
notification_id='OPC_UNSUBSCRIBE_ERROR_slot1:server1',
|
||||
message='Failed to unsubscribe slot slot1 from server1: session closed',
|
||||
block='opc_manager',
|
||||
level=NotificationLevel.ERROR,
|
||||
attachment_content=ANY,
|
||||
)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_unsubscribe_slot_with_real_readers(ingestor_manager):
|
||||
"""Contract: both real readers keep `subscriptions` keyed by slot, and
|
||||
unsubscribe_slot closes the slot's subscription on each of them"""
|
||||
v2_subscription = AsyncMock()
|
||||
client = MagicMock(create_subscription=AsyncMock(return_value=v2_subscription))
|
||||
pool = MagicMock(get=AsyncMock(return_value=client), evict=AsyncMock())
|
||||
server_reader = ServerReader(
|
||||
name='kepserver',
|
||||
server_config={
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'subscription_period_ms': 1000.0,
|
||||
},
|
||||
pool=pool,
|
||||
sink=AsyncMock(),
|
||||
logger=MagicMock(),
|
||||
notification_handler=MagicMock(),
|
||||
metrics_controller=AsyncMock(),
|
||||
metadata={},
|
||||
)
|
||||
server_reader.engine.emit_metric = AsyncMock()
|
||||
await server_reader.create_subscription('slot1')
|
||||
|
||||
with patch('ingestor.managers.opc_manager.metrics'):
|
||||
legacy_reader = OpcManager(
|
||||
name='legacy',
|
||||
url='opc.tcp://legacy:4840',
|
||||
data_manager=AsyncMock(),
|
||||
subscription_period_ms=1000,
|
||||
logger=MagicMock(),
|
||||
server_uri='opc.tcp://legacy:4840',
|
||||
notification_handler=MagicMock(),
|
||||
metadata={},
|
||||
metrics_controller=AsyncMock(),
|
||||
)
|
||||
legacy_reader.emit_metric = AsyncMock()
|
||||
legacy_subscription = AsyncMock()
|
||||
legacy_reader.client = MagicMock(
|
||||
create_subscription=AsyncMock(return_value=legacy_subscription)
|
||||
)
|
||||
await legacy_reader.create_subscription('slot1')
|
||||
|
||||
assert 'slot1' in server_reader.subscriptions
|
||||
assert 'slot1' in legacy_reader.subscriptions
|
||||
ingestor_manager.opc_managers = {'kepserver': server_reader, 'legacy': legacy_reader}
|
||||
|
||||
await ingestor_manager.unsubscribe_slot('slot1')
|
||||
|
||||
v2_subscription.delete.assert_awaited_once()
|
||||
legacy_subscription.delete.assert_awaited_once()
|
||||
assert 'slot1' not in server_reader.subscriptions
|
||||
assert 'slot1' not in legacy_reader.subscriptions
|
||||
pool.pin.assert_called_once_with('1')
|
||||
pool.unpin.assert_called_once_with('1')
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_update_slot_config_unsubscribes_removed_slot(ingestor_manager, reader_double):
|
||||
"""A slot deleted from Redis is unsubscribed right away, not left to the main loop"""
|
||||
ingestor_manager.managed_tags = {
|
||||
'slot1': parse_slot(raw_slot_v1('Same')),
|
||||
'slot2': parse_slot(raw_slot_v1('Same')),
|
||||
}
|
||||
reader = reader_double('slot1', 'slot2')
|
||||
ingestor_manager.opc_managers = {'test_server': reader}
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
|
||||
side_effect=lambda slot: raw_slot_v1('Same') if slot == 'slot1' else None
|
||||
)
|
||||
ingestor_manager.resource_manager.renew_tag_lease = AsyncMock()
|
||||
|
||||
await ingestor_manager.update_slot_config()
|
||||
|
||||
reader.unsubscribe.assert_called_once_with('slot2')
|
||||
assert list(ingestor_manager.managed_tags) == ['slot1']
|
||||
assert list(reader.subscriptions) == ['slot1']
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_update_slot_config(ingestor_manager):
|
||||
ingestor_manager.managed_tags = {
|
||||
'slot1': parse_slot(raw_slot_v1('Old')),
|
||||
'slot2': parse_slot(raw_slot_v1('Same')),
|
||||
'slot3': parse_slot(raw_slot_v1('Old')),
|
||||
'slot1': {'config': 'old_config'},
|
||||
'slot2': {'config': 'new_config'},
|
||||
'slot3': {'config': 'old_config'},
|
||||
}
|
||||
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
|
||||
side_effect=[raw_slot_v1('Updated'), raw_slot_v1('Same'), None]
|
||||
side_effect=[{'config': 'updated_config'}, {'config': 'new_config'}, None]
|
||||
)
|
||||
ingestor_manager.resource_manager.renew_tag_lease = AsyncMock()
|
||||
|
||||
await ingestor_manager.update_slot_config()
|
||||
|
||||
assert ingestor_manager.managed_tags['slot1'] == parse_slot(raw_slot_v1('Updated'))
|
||||
assert ingestor_manager.managed_tags['slot2'] == parse_slot(raw_slot_v1('Same'))
|
||||
assert ingestor_manager.managed_tags['slot1'] == {'config': 'updated_config'}
|
||||
assert ingestor_manager.managed_tags['slot2'] == {'config': 'new_config'}
|
||||
assert 'slot3' not in ingestor_manager.managed_tags
|
||||
|
||||
ingestor_manager.resource_manager.renew_tag_lease.assert_any_call('slot1')
|
||||
@@ -781,30 +575,6 @@ async def test_manage_server_create_subscription_failure(ingestor_manager):
|
||||
ingestor_manager.opc_managers['server1'].subscribe.assert_not_called()
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_manage_server_subscribe_and_unsubscribe_failure(ingestor_manager):
|
||||
"""subscribe() fails and so does the cleanup unsubscribe (legacy reader, session
|
||||
gone): manage_server reports the failure and nothing escapes to the main loop"""
|
||||
reader = AsyncMock()
|
||||
reader.subscriptions = {}
|
||||
reader.subscribe.side_effect = Exception('Subscription error')
|
||||
reader.unsubscribe.side_effect = ConnectionError('session closed')
|
||||
ingestor_manager.opc_managers = {'server1': reader}
|
||||
server_config = {'tags': {'tag1': 'config1'}}
|
||||
|
||||
result = await ingestor_manager.manage_server('slot1', 'server1', server_config, server_config)
|
||||
|
||||
assert result == 2
|
||||
reader.unsubscribe.assert_called_once_with('slot1')
|
||||
notified = [
|
||||
c.kwargs['notification_id'] for c in ingestor_manager.send_notification_async.call_args_list
|
||||
]
|
||||
assert notified == [
|
||||
'OPC_SUBSCRIPTION_ERROR_slot1:server1',
|
||||
'OPC_UNSUBSCRIBE_ERROR_slot1:server1',
|
||||
]
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_manage_server(ingestor_manager):
|
||||
ingestor_manager.opc_managers = {'server1': AsyncMock(), 'server2': AsyncMock()}
|
||||
@@ -882,202 +652,3 @@ async def test_subscribe_to_tags(ingestor_manager):
|
||||
assert ingestor_manager.manage_server.call_count == 3
|
||||
|
||||
ingestor_manager.managed_tags['slot1'].pop.assert_called_once_with('server3', None)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_update_slot_config_keeps_last_good_config_when_malformed(ingestor_manager):
|
||||
good = parse_slot(raw_slot_v1('Counter'))
|
||||
ingestor_manager.managed_tags = {'slot1': good}
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
|
||||
return_value={'server1': {'server_id': '1', 'url': 'u', 'tags': {'ns=2;i=2': {}}}}
|
||||
)
|
||||
ingestor_manager.resource_manager.renew_tag_lease = AsyncMock()
|
||||
|
||||
await ingestor_manager.update_slot_config()
|
||||
|
||||
# Still reading with the last good config; the bad one is notified, not applied.
|
||||
assert ingestor_manager.managed_tags == {'slot1': good}
|
||||
_, kwargs = ingestor_manager.send_notification.call_args
|
||||
assert kwargs['notification_id'] == 'OPC_SLOT_CONFIG_ERROR_slot1'
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_remove_server_forwards_evict(ingestor_manager):
|
||||
reader = AsyncMock()
|
||||
ingestor_manager.opc_managers = {'server1': reader}
|
||||
ingestor_manager.managed_tags = {'slot1': {'server1': {}}}
|
||||
|
||||
await ingestor_manager.remove_server('server1', evict=False)
|
||||
|
||||
reader.shutdown.assert_awaited_once_with(evict=False)
|
||||
assert 'server1' not in ingestor_manager.opc_managers
|
||||
assert ingestor_manager.managed_tags == {'slot1': {}}
|
||||
|
||||
|
||||
@patch('ingestor.managers.ingestor_manager.DataManager')
|
||||
@patch('ingestor.managers.ingestor_manager.ResourceManager')
|
||||
def test___init___ignores_malformed_write_env(
|
||||
resource_manager_mock, data_manager_mock, monkeypatch
|
||||
):
|
||||
"""A malformed write/gRPC-only env var must not stop the read path (review #1)."""
|
||||
monkeypatch.setenv('GRPC_PORT', '')
|
||||
monkeypatch.setenv('WRITE_MIN_PAUSE_S', '0,5')
|
||||
monkeypatch.setenv('CERT_BUCKET', 'streamlit-connectors')
|
||||
|
||||
ingestor = IngestorManager(
|
||||
redis_data={'host': 'localhost', 'port': 6379},
|
||||
lease_ttl=60,
|
||||
heartbeat_ttl=60,
|
||||
poll_interval=5,
|
||||
mongo_connection_string='mongodb://localhost:27017',
|
||||
mongo_database='sientia',
|
||||
logger=MagicMock(),
|
||||
notification_handler=MagicMock(),
|
||||
metadata=metadata['metadata'],
|
||||
metrics_controller=MagicMock(),
|
||||
)
|
||||
|
||||
assert ingestor.cert_provider.bucket == 'streamlit-connectors'
|
||||
|
||||
|
||||
def test_parse_slot_config_warns_once_about_server_cert_path(ingestor_manager):
|
||||
raw = raw_slot_v1()
|
||||
raw['server1']['server_cert_path'] = '/app/certs/kepserver.der'
|
||||
|
||||
ingestor_manager.parse_slot_config('1', raw)
|
||||
ingestor_manager.parse_slot_config('1', raw) # every loop refreshes the slot
|
||||
|
||||
warnings = [
|
||||
c for c in ingestor_manager.logger.warning.call_args_list if 'server_cert_path' in str(c)
|
||||
]
|
||||
assert len(warnings) == 1
|
||||
|
||||
|
||||
# --- READ_ENGINE=legacy: plan B with the untouched rc3 OpcManager -------------------
|
||||
|
||||
|
||||
def make_manager(read_engine=None):
|
||||
with (
|
||||
patch('ingestor.managers.ingestor_manager.DataManager'),
|
||||
patch('ingestor.managers.ingestor_manager.ResourceManager'),
|
||||
):
|
||||
manager = IngestorManager(
|
||||
redis_data={'host': 'localhost', 'port': 6379},
|
||||
lease_ttl=60,
|
||||
heartbeat_ttl=60,
|
||||
poll_interval=5,
|
||||
mongo_connection_string='mongodb://localhost:27017',
|
||||
mongo_database='sientia',
|
||||
logger=MagicMock(),
|
||||
notification_handler=MagicMock(),
|
||||
metadata=metadata['metadata'],
|
||||
metrics_controller=MagicMock(),
|
||||
read_engine=read_engine,
|
||||
)
|
||||
manager.send_notification = MagicMock()
|
||||
manager.emit_metric = AsyncMock()
|
||||
return manager
|
||||
|
||||
|
||||
def test_read_engine_defaults_to_v2(monkeypatch):
|
||||
monkeypatch.delenv('READ_ENGINE', raising=False)
|
||||
|
||||
assert make_manager().read_engine == 'v2'
|
||||
|
||||
|
||||
def test_read_engine_from_env(monkeypatch):
|
||||
monkeypatch.setenv('READ_ENGINE', 'Legacy ')
|
||||
|
||||
assert make_manager().read_engine == 'legacy'
|
||||
|
||||
|
||||
def test_invalid_read_engine_falls_back_to_v2(monkeypatch):
|
||||
monkeypatch.setenv('READ_ENGINE', 'v3')
|
||||
|
||||
manager = make_manager()
|
||||
|
||||
assert manager.read_engine == 'v2'
|
||||
assert any('Invalid READ_ENGINE' in str(c) for c in manager.logger.warning.call_args_list)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('ingestor.managers.ingestor_manager.ServerReader')
|
||||
@patch('ingestor.managers.ingestor_manager.OpcManager')
|
||||
async def test_legacy_initialize_builds_rc3_opc_manager(opc_manager, server_reader):
|
||||
manager = make_manager('legacy')
|
||||
server_config = {
|
||||
'name': 'server1',
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://localhost:4840',
|
||||
'subscription_period_ms': 1000,
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert_path': '/app/certs/opc-ingestor-cert.pem',
|
||||
'private_key_path': '/app/certs/opc-ingestor-key.pem',
|
||||
'server_cert_path': None,
|
||||
}
|
||||
opc_manager.return_value = MagicMock(connect=AsyncMock())
|
||||
|
||||
result = await manager.initialize_opc_from_config(server_config)
|
||||
|
||||
server_reader.assert_not_called()
|
||||
opc_manager.assert_called_once_with(
|
||||
name='server1',
|
||||
url='opc.tcp://localhost:4840',
|
||||
subscription_period_ms=1000,
|
||||
data_manager=manager.data_manager,
|
||||
logger=manager.logger,
|
||||
server_uri='urn:sientia:opc-ingestor',
|
||||
notification_handler=manager.notification_handler,
|
||||
metadata=metadata['metadata'],
|
||||
cert_path='/app/certs/opc-ingestor-cert.pem',
|
||||
private_key_path='/app/certs/opc-ingestor-key.pem',
|
||||
server_cert_path=None,
|
||||
metrics_controller=manager.metrics_controller,
|
||||
)
|
||||
assert result is opc_manager.return_value
|
||||
result.connect.assert_awaited_once()
|
||||
|
||||
|
||||
def test_legacy_keeps_raw_slot_v1():
|
||||
manager = make_manager('legacy')
|
||||
raw = raw_slot_v1()
|
||||
|
||||
# OpcManager reads tag_name/topics and the file cert paths straight from the slot
|
||||
assert manager.parse_slot_config('1', raw) is raw
|
||||
|
||||
|
||||
def test_legacy_rejects_slot_v2():
|
||||
manager = make_manager('legacy')
|
||||
raw_v2 = {
|
||||
'server1': {
|
||||
'name': 'server1',
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://localhost:4840',
|
||||
'use_security': True,
|
||||
'subscription_period_ms': 500,
|
||||
'tags': {
|
||||
'ns=2;i=2': {
|
||||
'frequency': '1000',
|
||||
'names': [{'collection': 'raw_a', 'tag_name': 'Counter', 'model_id': '1'}],
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
assert manager.parse_slot_config('1', raw_v2) is None
|
||||
_, kwargs = manager.send_notification.call_args
|
||||
assert kwargs['notification_id'] == 'OPC_SLOT_CONFIG_ERROR_1'
|
||||
assert 'READ_ENGINE=legacy' in kwargs['message']
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_legacy_remove_server_calls_plain_shutdown():
|
||||
manager = make_manager('legacy')
|
||||
opc_manager = MagicMock(shutdown=AsyncMock())
|
||||
manager.opc_managers = {'server1': opc_manager}
|
||||
manager.managed_tags = {'slot1': {'server1': {}}}
|
||||
|
||||
await manager.remove_server('server1', evict=False)
|
||||
|
||||
opc_manager.shutdown.assert_awaited_once_with()
|
||||
assert manager.opc_managers == {}
|
||||
|
||||
@@ -1,164 +0,0 @@
|
||||
from unittest.mock import AsyncMock, MagicMock, call
|
||||
|
||||
import pytest
|
||||
|
||||
import ingestor.metrics as metrics
|
||||
from ingestor.managers.server_reader import ServerReader
|
||||
|
||||
SERVER_CONFIG = {
|
||||
'name': 'kepserver',
|
||||
'server_id': 1, # the slot may carry it as int; the pool keys by str
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'use_security': True,
|
||||
'subscription_period_ms': 15000.0,
|
||||
}
|
||||
|
||||
|
||||
def make_reader():
|
||||
pool = MagicMock(get=AsyncMock(), evict=AsyncMock())
|
||||
reader = ServerReader(
|
||||
name='kepserver',
|
||||
server_config=SERVER_CONFIG,
|
||||
pool=pool,
|
||||
sink=AsyncMock(),
|
||||
logger=MagicMock(),
|
||||
notification_handler=MagicMock(),
|
||||
metrics_controller=AsyncMock(),
|
||||
metadata={},
|
||||
)
|
||||
reader.engine = MagicMock(
|
||||
subscriptions={},
|
||||
pod_id='pod',
|
||||
create_subscription=AsyncMock(),
|
||||
subscribe=AsyncMock(),
|
||||
unsubscribe=AsyncMock(),
|
||||
check_cycles=AsyncMock(),
|
||||
check_opc_listenning=AsyncMock(return_value=False),
|
||||
emit_metric=AsyncMock(),
|
||||
)
|
||||
return reader, pool
|
||||
|
||||
|
||||
def test_server_cfg_keeps_only_connection_fields():
|
||||
reader, _ = make_reader()
|
||||
|
||||
assert reader.server_id == '1'
|
||||
assert reader.server_cfg == {
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'use_security': True,
|
||||
'subscription_period_ms': 15000.0,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_connect_opens_pooled_connection():
|
||||
reader, pool = make_reader()
|
||||
|
||||
await reader.connect()
|
||||
|
||||
# use_security=True: the pool fetches the certificate from MinIO (CertProvider)
|
||||
pool.get.assert_awaited_once_with('1', reader.server_cfg)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_connect_propagates_failure():
|
||||
reader, pool = make_reader()
|
||||
pool.get.side_effect = ConnectionError('unreachable')
|
||||
|
||||
with pytest.raises(ConnectionError):
|
||||
await reader.connect()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_subscription_calls_delegate_to_engine_per_slot():
|
||||
reader, _ = make_reader()
|
||||
nodes = {'ns=2;i=2': {'frequency': '1000', 'names': [{'collection': 'raw_a', 'tag_name': 'a'}]}}
|
||||
|
||||
await reader.create_subscription('1')
|
||||
await reader.subscribe('1', nodes, 10)
|
||||
await reader.unsubscribe('1')
|
||||
|
||||
reader.engine.create_subscription.assert_awaited_once_with('1', reader.server_cfg, '1')
|
||||
reader.engine.subscribe.assert_awaited_once_with('1', nodes, 10)
|
||||
reader.engine.unsubscribe.assert_awaited_once_with('1')
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_supervision_delegates_to_engine():
|
||||
reader, _ = make_reader()
|
||||
reader.engine.check_opc_listenning.return_value = True
|
||||
|
||||
await reader.check_cycles()
|
||||
lost = await reader.check_opc_listenning()
|
||||
|
||||
reader.engine.check_cycles.assert_awaited_once()
|
||||
reader.engine.check_opc_listenning.assert_awaited_once_with('1')
|
||||
assert lost is True
|
||||
|
||||
|
||||
def test_subscriptions_are_the_engine_ones():
|
||||
reader, _ = make_reader()
|
||||
reader.engine.subscriptions['1'] = MagicMock()
|
||||
|
||||
assert reader.subscriptions is reader.engine.subscriptions
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_shutdown_unsubscribes_all_slots_evicts_and_zeroes_metrics():
|
||||
reader, pool = make_reader()
|
||||
reader.engine.subscriptions.update({'1': MagicMock(), '2': MagicMock()})
|
||||
|
||||
await reader.shutdown()
|
||||
|
||||
reader.engine.unsubscribe.assert_has_awaits([call('1'), call('2')])
|
||||
# Dropping the pooled connection makes the next initialization reconnect.
|
||||
pool.evict.assert_awaited_once_with('1')
|
||||
reader.engine.emit_metric.assert_any_await(
|
||||
metric_object=metrics.OPC_CONNECTION_STATUS,
|
||||
method='set',
|
||||
value=0,
|
||||
tags={'pod_id': 'pod', 'server_name': '1', 'server_url': 'opc.tcp://kepserver:49320'},
|
||||
)
|
||||
reader.engine.emit_metric.assert_any_await(
|
||||
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
|
||||
method='set',
|
||||
value=0,
|
||||
tags={'pod_id': 'pod', 'server_name': '1'},
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_shutdown_without_evict_keeps_the_shared_connection():
|
||||
"""Slot moved to another pod: drop the subscriptions, keep the pooled
|
||||
connection for NATS writes/gRPC (review #7)."""
|
||||
reader, pool = make_reader()
|
||||
reader.engine.subscriptions.update({'1': MagicMock()})
|
||||
|
||||
await reader.shutdown(evict=False)
|
||||
|
||||
reader.engine.unsubscribe.assert_awaited_once_with('1')
|
||||
pool.evict.assert_not_awaited()
|
||||
for c in reader.engine.emit_metric.await_args_list:
|
||||
assert c.kwargs['metric_object'] is not metrics.OPC_CONNECTION_STATUS
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_shutdown_unsubscribes_every_slot_while_engine_pops_them():
|
||||
"""The real unsubscribe() pops from engine.subscriptions: shutdown must iterate a
|
||||
snapshot (list()), or the loop breaks with "dictionary changed size during
|
||||
iteration" and the remaining slots stay subscribed (SonarQube S7504 does not apply)."""
|
||||
reader, pool = make_reader()
|
||||
subs = {'1': MagicMock(), '2': MagicMock(), '3': MagicMock()}
|
||||
reader.engine.subscriptions = subs
|
||||
|
||||
async def pop(slot):
|
||||
subs.pop(slot)
|
||||
|
||||
reader.engine.unsubscribe = AsyncMock(side_effect=pop)
|
||||
|
||||
await reader.shutdown()
|
||||
|
||||
assert subs == {}
|
||||
assert reader.engine.unsubscribe.await_count == 3
|
||||
@@ -139,7 +139,6 @@ async def test_main_successful_run_one_loop(mock_app_env, capsys):
|
||||
mock_app_env['GatewayRuntime'].assert_called_once_with(
|
||||
config=mock_app_env['GatewayConfig'].from_env.return_value,
|
||||
data_manager=mock_ingestor_instance.ingestor_manager.data_manager,
|
||||
pool=mock_ingestor_instance.ingestor_manager.pool,
|
||||
logger=mock_ingestor_instance.logger,
|
||||
notification_handler=mock_ingestor_instance.notification_handler,
|
||||
metrics_controller=mock_ingestor_instance.metrics_controller,
|
||||
@@ -351,9 +350,8 @@ async def test_main_prepare_ingestor_failure(mock_app_env, capsys):
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_main_gateway_runtime_init_failure_keeps_read_path_alive(mock_app_env):
|
||||
"""GatewayRuntime construction failing (e.g. Postgres config) must not kill the
|
||||
app: the read path lives in the ingestor_manager, so the main loop still runs and
|
||||
shuts down cleanly."""
|
||||
"""GatewayRuntime construction hitting the network (MinIO/Postgres) and failing
|
||||
must not kill the app: the main loop (read path) still runs and shuts down cleanly."""
|
||||
mock_ingestor_instance = mock_app_env['Ingestor'].return_value
|
||||
mock_exit_signal = mock_app_env['mock_exit_signal']
|
||||
|
||||
|
||||
@@ -31,7 +31,7 @@ def test_from_env_defaults():
|
||||
assert cfg.nats_queue == 'gateway-writers'
|
||||
assert cfg.grpc_enabled is False
|
||||
assert cfg.grpc_port == 50051
|
||||
assert cfg.cert_bucket == 'streamlit-connectors'
|
||||
assert cfg.cert_bucket == 'sientia-dataops'
|
||||
assert cfg.cert_cache_dir == '/tmp/opc-certs' # noqa: S108
|
||||
assert cfg.write_timestamp is True
|
||||
assert cfg.write_quality is True
|
||||
@@ -69,19 +69,3 @@ def test_bool_parsing_edge_cases(monkeypatch, value, expected):
|
||||
cfg = GatewayConfig.from_env()
|
||||
|
||||
assert cfg.grpc_enabled is expected
|
||||
|
||||
|
||||
def test_certs_from_env_ignores_malformed_numeric_vars(monkeypatch):
|
||||
import pytest
|
||||
|
||||
monkeypatch.setenv('GRPC_PORT', '')
|
||||
monkeypatch.setenv('IDLE_CONN_MAX_S', 'abc')
|
||||
monkeypatch.setenv('CERT_BUCKET', 'my-bucket')
|
||||
monkeypatch.setenv('CERT_CACHE_DIR', 'certs-cache')
|
||||
|
||||
cfg = GatewayConfig.certs_from_env()
|
||||
|
||||
assert cfg.cert_bucket == 'my-bucket'
|
||||
assert cfg.cert_cache_dir == 'certs-cache'
|
||||
with pytest.raises(ValueError):
|
||||
GatewayConfig.from_env() # the full config still fails, only the runtime uses it
|
||||
|
||||
@@ -6,6 +6,9 @@ from ingestor import gateway_runtime as gr
|
||||
from ingestor.config import GatewayConfig
|
||||
|
||||
_PATCHED_CLASSES = (
|
||||
'MinioRepository',
|
||||
'CertProvider',
|
||||
'OpcConnectionPool',
|
||||
'Postgres',
|
||||
'build_postgres_config',
|
||||
'DataSink',
|
||||
@@ -33,25 +36,19 @@ def mocks(monkeypatch):
|
||||
'min_connections': 5,
|
||||
'max_connections': 20,
|
||||
}
|
||||
# The pool is owned by IngestorManager (read path) and only borrowed here.
|
||||
patched['pool'] = MagicMock()
|
||||
patched['pool'].close_all = AsyncMock()
|
||||
patched['pool'].release_idle = AsyncMock()
|
||||
patched['OpcConnectionPool'].return_value.close_all = AsyncMock()
|
||||
patched['OpcConnectionPool'].return_value.release_idle = AsyncMock()
|
||||
patched['NatsWriteConsumer'].return_value.start = AsyncMock()
|
||||
patched['NatsWriteConsumer'].return_value.stop = AsyncMock()
|
||||
patched['NatsWriteConsumer'].return_value.closed = False
|
||||
patched['GrpcServer'].return_value.start = AsyncMock()
|
||||
patched['GrpcServer'].return_value.stop = AsyncMock()
|
||||
return patched
|
||||
|
||||
|
||||
def make_runtime(pool=None, **config_overrides):
|
||||
if pool is None:
|
||||
pool = MagicMock(close_all=AsyncMock(), release_idle=AsyncMock())
|
||||
def make_runtime(**config_overrides):
|
||||
return gr.GatewayRuntime(
|
||||
config=GatewayConfig(**config_overrides),
|
||||
data_manager=MagicMock(),
|
||||
pool=pool,
|
||||
logger=MagicMock(),
|
||||
notification_handler=MagicMock(),
|
||||
metrics_controller=MagicMock(),
|
||||
@@ -60,26 +57,26 @@ def make_runtime(pool=None, **config_overrides):
|
||||
|
||||
def test_builds_object_graph_with_config_flags(mocks):
|
||||
runtime = make_runtime(
|
||||
pool=mocks['pool'],
|
||||
write_timestamp=False,
|
||||
write_quality=False,
|
||||
write_min_pause_s=0.25,
|
||||
cert_bucket='my-bucket',
|
||||
cert_cache_dir='certs-cache',
|
||||
grpc_enabled=False,
|
||||
)
|
||||
|
||||
# Writes and active reads share the pool built by the read path.
|
||||
assert runtime.pool is mocks['pool']
|
||||
mocks['OpcConnectionPool'].assert_called_once()
|
||||
mocks['DataSink'].assert_called_once()
|
||||
|
||||
mocks['WriteEngine'].assert_called_once()
|
||||
args, kwargs = mocks['WriteEngine'].call_args
|
||||
assert args[0] is mocks['pool']
|
||||
_, kwargs = mocks['WriteEngine'].call_args
|
||||
assert kwargs['write_timestamp'] is False
|
||||
assert kwargs['write_quality'] is False
|
||||
assert kwargs['min_pause_s'] == 0.25
|
||||
|
||||
args, _ = mocks['ActiveReadEngine'].call_args
|
||||
assert args[0] is mocks['pool']
|
||||
_, cert_kwargs = mocks['CertProvider'].call_args
|
||||
assert cert_kwargs['bucket'] == 'my-bucket'
|
||||
assert cert_kwargs['cache_dir'] == 'certs-cache'
|
||||
|
||||
mocks['GrpcServer'].assert_not_called()
|
||||
assert runtime.grpc_server is None
|
||||
@@ -140,21 +137,20 @@ async def test_nats_start_failure_does_not_raise_and_idle_sweep_retries(mocks):
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stop_stops_started_components_but_not_the_pool(mocks):
|
||||
runtime = make_runtime(pool=mocks['pool'], grpc_enabled=True)
|
||||
async def test_stop_stops_started_components_and_closes_pool(mocks):
|
||||
runtime = make_runtime(grpc_enabled=True)
|
||||
await runtime.start()
|
||||
|
||||
await runtime.stop()
|
||||
|
||||
mocks['GrpcServer'].return_value.stop.assert_awaited_once()
|
||||
mocks['NatsWriteConsumer'].return_value.stop.assert_awaited_once()
|
||||
# IngestorManager.shutdown() closes it, after its readers unsubscribe.
|
||||
mocks['pool'].close_all.assert_not_called()
|
||||
mocks['OpcConnectionPool'].return_value.close_all.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stop_skips_nats_stop_if_never_started(mocks):
|
||||
runtime = make_runtime(pool=mocks['pool'])
|
||||
runtime = make_runtime()
|
||||
runtime.send_notification_async = AsyncMock()
|
||||
nats_mock = mocks['NatsWriteConsumer'].return_value
|
||||
nats_mock.start = AsyncMock(side_effect=Exception('boom'))
|
||||
@@ -163,58 +159,14 @@ async def test_stop_skips_nats_stop_if_never_started(mocks):
|
||||
await runtime.stop()
|
||||
|
||||
nats_mock.stop.assert_not_called()
|
||||
mocks['pool'].close_all.assert_not_called()
|
||||
mocks['OpcConnectionPool'].return_value.close_all.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_idle_sweep_releases_idle_connections(mocks):
|
||||
runtime = make_runtime(pool=mocks['pool'], idle_conn_max_s=42.0)
|
||||
runtime = make_runtime(idle_conn_max_s=42.0)
|
||||
runtime._nats_started = True # isolate from the retry path
|
||||
|
||||
await runtime.idle_sweep()
|
||||
|
||||
mocks['pool'].release_idle.assert_awaited_once_with(42.0)
|
||||
|
||||
|
||||
def test_write_engine_resolves_servers_from_the_pool_and_mongo(mocks):
|
||||
"""SIENTIAPDE-1975 GW1: write commands carry only the server_id."""
|
||||
data_manager = MagicMock()
|
||||
runtime = gr.GatewayRuntime(
|
||||
config=GatewayConfig(),
|
||||
data_manager=data_manager,
|
||||
pool=mocks['pool'],
|
||||
logger=MagicMock(),
|
||||
notification_handler=MagicMock(),
|
||||
metrics_controller=MagicMock(),
|
||||
)
|
||||
|
||||
_, kwargs = mocks['WriteEngine'].call_args
|
||||
assert kwargs['servers'] is runtime.servers
|
||||
assert runtime.servers.pool is mocks['pool']
|
||||
assert runtime.servers.mongo is data_manager.mongo_repository
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_idle_sweep_restarts_a_closed_nats_connection(mocks):
|
||||
runtime = make_runtime(pool=mocks['pool'])
|
||||
nats_mock = mocks['NatsWriteConsumer'].return_value
|
||||
await runtime.start()
|
||||
assert nats_mock.start.await_count == 1
|
||||
|
||||
nats_mock.closed = True # closed for good (e.g. by the server)
|
||||
await runtime.idle_sweep()
|
||||
|
||||
assert nats_mock.start.await_count == 2
|
||||
assert runtime._nats_started is True
|
||||
runtime.logger.warning.assert_called_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_idle_sweep_leaves_a_live_nats_connection_alone(mocks):
|
||||
runtime = make_runtime(pool=mocks['pool'])
|
||||
nats_mock = mocks['NatsWriteConsumer'].return_value
|
||||
await runtime.start()
|
||||
|
||||
await runtime.idle_sweep()
|
||||
|
||||
assert nats_mock.start.await_count == 1
|
||||
mocks['OpcConnectionPool'].return_value.release_idle.assert_awaited_once_with(42.0)
|
||||
|
||||
@@ -1,10 +1,8 @@
|
||||
from copy import deepcopy
|
||||
from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
|
||||
|
||||
from pytest import fixture, mark
|
||||
|
||||
from ingestor.ingestor import Ingestor
|
||||
from ingestor.managers.ingestor_manager import IngestorManager
|
||||
|
||||
|
||||
@patch('ingestor.ingestor.getenv')
|
||||
@@ -231,10 +229,6 @@ async def test_manage_leases_no_available_slots_extra_sltos(ingestor_manager_sta
|
||||
ingestor_manager_started.ingestor_manager.drop_slot_leases.assert_called_once_with(
|
||||
['tag2', 'tag3']
|
||||
)
|
||||
# The extra slots are unsubscribed before their leases are given back
|
||||
calls = [c[0] for c in ingestor_manager_started.ingestor_manager.mock_calls]
|
||||
assert calls.index('drop_slot_leases') > calls.index('unsubscribe_slot')
|
||||
assert list(ingestor_manager_started.ingestor_manager.managed_tags) == ['tag1']
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@@ -338,52 +332,3 @@ async def test_update_ingestor_manager_no_ingestor_manager(ingestor_manager_star
|
||||
ingestor_manager_started.ingestor_manager = None
|
||||
|
||||
await ingestor_manager_started.update_ingestor_manager({})
|
||||
|
||||
|
||||
def _manager_with_real_unsubscribe(managed_tags, readers):
|
||||
"""A mocked IngestorManager whose unsubscribe path (unsubscribe_slot and
|
||||
_unsubscribe_reader) is the real one."""
|
||||
manager = AsyncMock(update_opc_servers=AsyncMock(), subscribe_to_tags=AsyncMock())
|
||||
manager.managed_tags = managed_tags
|
||||
manager.opc_managers = readers
|
||||
manager.unsubscribe_slot = lambda slot: IngestorManager.unsubscribe_slot(manager, slot)
|
||||
manager._unsubscribe_reader = lambda server, reader, slot: IngestorManager._unsubscribe_reader(
|
||||
manager, server, reader, slot
|
||||
)
|
||||
return manager
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_dropped_extra_lease_does_not_stop_the_loop(ingestor_manager_started, reader_double):
|
||||
"""Regression: manage_leases drops slot 2, then update_ingestor_manager sees slot 2 in
|
||||
the snapshot only and unsubscribes it again. That used to raise KeyError and stop
|
||||
the main loop (container restart)."""
|
||||
reader = reader_double('1', '2')
|
||||
ingestor_manager_started.ingestor_manager = _manager_with_real_unsubscribe(
|
||||
{'1': {'kepserver': {}}, '2': {'kepserver': {}}}, {'kepserver': reader}
|
||||
)
|
||||
snapshot = deepcopy(ingestor_manager_started.ingestor_manager.managed_tags)
|
||||
|
||||
await ingestor_manager_started.manage_leases(0, 0, 1)
|
||||
await ingestor_manager_started.update_ingestor_manager(snapshot)
|
||||
|
||||
reader.unsubscribe.assert_called_once_with('2')
|
||||
assert list(reader.subscriptions) == ['1']
|
||||
assert list(ingestor_manager_started.ingestor_manager.managed_tags) == ['1']
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_slot_removed_from_redis_is_unsubscribed(ingestor_manager_started, reader_double):
|
||||
"""Regression: update_slot_config drops a slot deleted by the orchestrator from
|
||||
managed_tags without unsubscribing it; update_ingestor_manager must close its
|
||||
subscription instead of raising KeyError."""
|
||||
reader = reader_double('1', '3')
|
||||
ingestor_manager_started.ingestor_manager = _manager_with_real_unsubscribe(
|
||||
{'1': {'kepserver': {}}}, {'kepserver': reader}
|
||||
)
|
||||
snapshot = {'1': {'kepserver': {}}, '3': {'kepserver': {}}}
|
||||
|
||||
await ingestor_manager_started.update_ingestor_manager(snapshot)
|
||||
|
||||
reader.unsubscribe.assert_called_once_with('3')
|
||||
assert list(reader.subscriptions) == ['1']
|
||||
|
||||
Reference in New Issue
Block a user