SIENTIAPDE-1973
SIENTIAPDE-1973: Implement per-connector OPC client certificates and remove legacy read engine. This update overhauls certificate management, transitioning from a single platform certificate to individual client certificates defined in the slot configuration. The CertProvider now downloads certificates from MinIO based on certificate_id, applying dynamic security policies (mode, policy) from the slot. The legacy READ_ENGINE=legacy option and OpcManager have been entirely removed, standardizing on the ServerReader with strict slot v2 enforcement. Write command resolution (ServerConfigResolver) now consistently uses connectors joined with opc_certificates. The CERT_BUCKET environment variable is no longer used.
This commit is contained in:
@@ -21,7 +21,6 @@ NATS_WRITE_SUBJECT="opc.write"
|
||||
NATS_QUEUE="gateway-writers"
|
||||
GRPC_ENABLED="false"
|
||||
GRPC_PORT="50051"
|
||||
CERT_BUCKET="sientia-dataops"
|
||||
CERT_CACHE_DIR="/tmp/opc-certs"
|
||||
WRITE_TIMESTAMP="true"
|
||||
WRITE_QUALITY="true"
|
||||
|
||||
@@ -41,7 +41,6 @@ app.py (asyncio loop, signals, Prometheus server)
|
||||
-> IngestorManager (managers/ingestor_manager.py) — central coordinator
|
||||
-> ResourceManager (managers/resource_manager.py) — Redis slot leases, heartbeats, active-ingestor registry
|
||||
-> ServerReader (managers/server_reader.py), one per OPC server — SubscriptionEngine over the shared OpcConnectionPool
|
||||
(READ_ENGINE=legacy: OpcManager (managers/opc_manager.py), the rc3 reader kept as plan B until the end of SIENTIAPDE-1974)
|
||||
-> OpcConnectionPool (connection/pool.py) + CertProvider (connection/certs.py) — pooled connections, cert from MinIO
|
||||
-> DataManager (managers/data_manager.py) — MongoDB persistence + optional Kafka publish
|
||||
```
|
||||
@@ -58,9 +57,9 @@ app.py (asyncio loop, signals, Prometheus server)
|
||||
|
||||
Any change to slot/lease counts should be traced through `manage_leases`/`manage_no_slots` — that's where the rebalancing math lives, not in `ResourceManager` itself (which just does Redis primitives).
|
||||
|
||||
**ServerReader** (SIENTIAPDE-1974, default `READ_ENGINE=v2`; the legacy `OpcManager` stays as plan B with `READ_ENGINE=legacy` — slot v1 only, certificate files from the Gitea clone — until it is unused and the tests pass the same without it) reads one OPC server through a `SubscriptionEngine`: the connection comes from the shared `OpcConnectionPool` (also used by NATS writes and gRPC), which fetches the platform certificate from MinIO via `CertProvider` and uses `SecurityPolicyBasic256Sha256` — no certificate files in the repo/image. Slots are parsed by `io/slots.py` (`parse_slot`), which accepts slot v2 (`names[]`) and converts the legacy v1 (`topics`). `datachange_notification` fans each value out to every `names` entry via `DataSink`. Subscribed connections are pinned in the pool so the idle sweep never drops them. Tracks `non_receive_count` to detect silently-dead subscriptions (`check_cycles` / `check_opc_listenning`); a lost server is shut down (pool connection evicted) and re-initialized on the next loop.
|
||||
**ServerReader** (SIENTIAPDE-1974) reads one OPC server through a `SubscriptionEngine`: the connection comes from the shared `OpcConnectionPool` (also used by NATS writes and gRPC). Slots are slot v2 only, parsed by `io/slots.py` (`parse_slot`): each server carries `security` (`mode`, `policy`) and `certificate` (certificate_id, application_uri, MinIO `bucket`/`object_name` of cert and private_key, from `opc_certificates` via the Orchestrator; `None` = connect without security, D5). The pool applies security only with a certificate (`connection/security.py`: `is_secured`, `apply_security` maps mode/policy to asyncua, unknown → error), and `CertProvider` downloads each certificate into `<CERT_CACHE_DIR>/<certificate_id>/` (cache, lock and `invalidate` per certificate) — no certificate files in the repo/image, no default certificate. A change of `security`/`certificate` in the slot re-initializes the reader (pool connection evicted). `datachange_notification` fans each value out to every `names` entry via `DataSink`. Subscribed connections are pinned in the pool so the idle sweep never drops them. Tracks `non_receive_count` to detect silently-dead subscriptions (`check_cycles` / `check_opc_listenning`); a lost server is shut down (pool connection evicted) and re-initialized on the next loop.
|
||||
|
||||
**Writes** (SIENTIAPDE-1975): `NatsWriteConsumer` → `WriteEngine` → pool, then `DataSink.update_write_status`. Commands (C2) carry only `server_id`; `ServerConfigResolver` (`connection/servers.py`) resolves the connection (command override → live pooled connection → `connectors` → `opc-servers`). `write_status` keeps the worst outcome (`ok` < `error` < `session_bad`); the gateway is the only one that updates it (the Laborious inserts the row as `pending` and never records OPC outcomes). The NATS first connect is bounded (it runs on the main loop); reconnection after it is unbounded.
|
||||
**Writes** (SIENTIAPDE-1975): `NatsWriteConsumer` → `WriteEngine` → pool, then `DataSink.update_write_status`. Commands (C2) carry only `server_id`; `ServerConfigResolver` (`connection/servers.py`) resolves the connection: live pooled connection → `connectors` aggregation with a `$lookup` on `opc_certificates`, in the slot format. The certificate rule (active + client, `connection/certificates.py`) mirrors the Orchestrator's `tag_database.resolve_certificate` — keep both in sync (SIENTIAPDE-1974 D1; the Mongo fallback goes away when writes are routed to the pod that owns the connection). `write_status` keeps the worst outcome (`ok` < `error` < `session_bad`); the gateway is the only one that updates it (the Laborious inserts the row as `pending` and never records OPC outcomes). The NATS first connect is bounded (it runs on the main loop); reconnection after it is unbounded.
|
||||
|
||||
**Metrics** (`ingestor/metrics.py`): all Prometheus metrics defined here, always labeled by `pod_id`. Emitted via `SientiaMonitoring.emit_metric(...)` (from `sientia_do`), not the prometheus client directly — new metrics should follow that pattern for consistency with logging/notifications.
|
||||
|
||||
|
||||
115
README.md
115
README.md
@@ -63,8 +63,8 @@ Gateway for on-demand and command-driven interactions with OPC UA servers:
|
||||
- **Scheduled active reads**: a polling read engine (`ingestor/engines/active_read.py`)
|
||||
for tags that need periodic reads outside the subscription pipeline;
|
||||
orchestrator-side wiring lands in SIENTIAPDE-1974.
|
||||
- **Certificate management**: client certs for secure OPC connections are
|
||||
fetched from MinIO + a Mongo `platform_certificates` collection
|
||||
- **Certificate management**: each OPC connector has its own client certificate
|
||||
(`opc_certificates`), named in the slot and downloaded from MinIO
|
||||
(`ingestor/connection/certs.py`) rather than shipped in the image.
|
||||
|
||||
See `docs/deploy-notes.md` for the deployment-facing details (env vars,
|
||||
@@ -83,12 +83,12 @@ The Laborious publishes one command per tag (SIENTIAPDE-1975):
|
||||
```
|
||||
|
||||
- **Server connection**: the command carries only `server_id`. The connection is
|
||||
resolved by `ServerConfigResolver` (`ingestor/connection/servers.py`): `url`/
|
||||
`server_uri`/`use_security` in the command when present (override), else the
|
||||
live pooled connection opened by the reads, else the `connectors` document
|
||||
(`type: opc_ua`, `status: Active`), else the `opc-servers` document
|
||||
(`use_security = bool(cert_path)`) — the orchestrator's rule for the read slots.
|
||||
Lookups are cached for 60 s. An unknown server is a failed write (`server_not_found`).
|
||||
resolved by `ServerConfigResolver` (`ingestor/connection/servers.py`): the live
|
||||
pooled connection opened by the reads, else the `connectors` document (`type: opc_ua`,
|
||||
`status: Active`) joined with `opc_certificates`, with the orchestrator's rule for the
|
||||
read slots (`ingestor/connection/certificates.py`), in the slot format (`url`,
|
||||
`security`, `certificate`). Lookups are cached for 60 s. An unknown or incomplete
|
||||
server is a failed write (`server_not_found`).
|
||||
- **Outcome**: `UPDATE ... SET write_status` on the prediction row (by `timestamp` +
|
||||
`model_id`), keeping the worst outcome: `ok` < `error` < `session_bad`, and
|
||||
`pending`/NULL are replaced by any answer, so the order of the answers does not
|
||||
@@ -227,48 +227,42 @@ The feeder creates sample OPC tag configurations in Redis that the ingestor can
|
||||
|
||||
## 🔐 OPC Client Certificate
|
||||
|
||||
OPC UA servers that require secure connections validate the gateway with a single,
|
||||
shared platform X.509 client certificate. Every OPC connection — subscription reads,
|
||||
NATS writes and gRPC reads — uses it with `Basic256Sha256` (SIENTIAPDE-1974).
|
||||
Each OPC connector has its own X.509 client certificate (SIENTIAPDE-1974): the
|
||||
connector's `security.client_certificate_id` points to an `opc_certificates` document
|
||||
(`kind: client`, `status: active`). The Orchestrator joins them and writes, for each
|
||||
server of the slot, its `security` (`mode`, `policy`) and its `certificate`:
|
||||
|
||||
```json
|
||||
"certificate": {
|
||||
"certificate_id": 1,
|
||||
"application_uri": "urn:sientia:opc-ingestor",
|
||||
"cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-cert.pem"},
|
||||
"private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-key.pem"}
|
||||
}
|
||||
```
|
||||
|
||||
The certificate is **never** read from the repository or the image. On connect, the
|
||||
`CertProvider` (`ingestor/connection/certs.py`):
|
||||
1. reads the single document of the Mongo collection `platform_certificates`:
|
||||
`{"client_cert": {"minio_object_name": "..."}, "private_key": {"minio_object_name": "..."}}`;
|
||||
2. downloads both objects from the MinIO bucket `CERT_BUCKET` (default
|
||||
`streamlit-connectors`) into `CERT_CACHE_DIR` (default `/tmp/opc-certs`).
|
||||
`CertProvider` (`ingestor/connection/certs.py`) downloads both objects from MinIO into
|
||||
`CERT_CACHE_DIR/<certificate_id>/` (default `/tmp/opc-certs`), and caches them per
|
||||
certificate. When the slot names other objects for the same certificate (it was
|
||||
replaced), the server config changes, the connection is re-opened and the files are
|
||||
downloaded again. A failed secured connection also forces a new download.
|
||||
|
||||
Requirements for a server to accept the connection:
|
||||
- the server's slot config has `use_security: true` (a slot v1 server with a
|
||||
`cert_path` is treated the same way; the path itself is ignored);
|
||||
- the certificate's SAN URI equals the server's `server_uri` in the slot — it is sent
|
||||
as the client `application_uri` (KEPServer rejects a mismatch with
|
||||
`BadCertificateUriInvalid`);
|
||||
- the server exposes a `Basic256Sha256` endpoint and trusts the certificate (e.g.
|
||||
KEPServer: *OPC UA Configuration Manager → Trusted Clients → Trust*, then restart
|
||||
its runtime).
|
||||
- **Security only with a certificate.** With a `certificate`, the connection uses the
|
||||
slot's `security.mode` (`Sign` / `SignAndEncrypt`) and `security.policy`
|
||||
(`Basic256Sha256`, `Aes128Sha256RsaOaep`, `Aes256Sha256RsaPss`, `Basic256`,
|
||||
`Basic128Rsa15`); an unknown one fails the connection. Without a certificate
|
||||
(`null`), or with `mode: None`, it connects without security, and a server that
|
||||
requires a certificate refuses the connection (the error shows in the log and the
|
||||
connection metrics). There is no default certificate.
|
||||
- The certificate's `application_uri` (its SAN URI) is sent as the client application
|
||||
URI (KEPServer rejects a mismatch with `BadCertificateUriInvalid`).
|
||||
- The server must trust the certificate (e.g. KEPServer: *OPC UA Configuration Manager
|
||||
→ Trusted Clients → Trust*, then restart its runtime).
|
||||
|
||||
MinIO is only contacted when a certificate is first needed, so a MinIO outage only
|
||||
affects secured servers, which are retried on the next main-loop iteration.
|
||||
|
||||
### Plan B: legacy read engine (`READ_ENGINE=legacy`)
|
||||
|
||||
Until the end of SIENTIAPDE-1974 the legacy reader is kept for validation and as a
|
||||
fallback. `READ_ENGINE` selects the read path (default `v2`; an invalid value falls
|
||||
back to `v2` with a warning):
|
||||
|
||||
| `READ_ENGINE` | Reader | Slot | Certificate |
|
||||
|---|---|---|---|
|
||||
| `v2` (default) | `ServerReader` + `SubscriptionEngine` over the shared pool | v1 or v2 | MinIO + Mongo (`CertProvider`), `Basic256Sha256` |
|
||||
| `legacy` | the rc3 `OpcManager`, unchanged, with its own connection | **v1 only** (a v2 slot is rejected and notified) | files from the repository the pod clones (`cert_path` / `private_key_path` in the slot), `Basic256` |
|
||||
|
||||
Writes and gRPC always use the pool (MinIO certificate), whatever the read engine.
|
||||
The legacy engine needs the certificate files in the Gitea branch the pod clones;
|
||||
`scripts/generate-and-push-opc-cert.sh` generates them and pushes them to
|
||||
`certs/` (`GITEA_PASSWORD='<senha>' BRANCH=<branch> ./scripts/generate-and-push-opc-cert.sh`).
|
||||
Each version deploy creates a new Gitea branch without `certs/`, so they have to be
|
||||
copied again for the legacy engine. Remove the legacy engine (OpcManager, its tests
|
||||
and the script) once it is unused and the test suite passes the same without it.
|
||||
MinIO is only contacted when a certificate is first needed (one client per bucket), so
|
||||
a MinIO outage only affects secured servers, which are retried on the next main-loop
|
||||
iteration.
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
@@ -349,7 +343,7 @@ The OPC Ingestor exposes comprehensive Prometheus metrics for monitoring:
|
||||
| `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
|
||||
|
||||
See the `# OPC Gateway` block in `.env.example` (`NATS_URL`, `GRPC_ENABLED`,
|
||||
`CERT_BUCKET`, `WRITE_TIMESTAMP`, ...) and `docs/deploy-notes.md` for the
|
||||
`CERT_CACHE_DIR`, `WRITE_TIMESTAMP`, ...) and `docs/deploy-notes.md` for the
|
||||
gateway-specific environment variables.
|
||||
|
||||
### OPC Server Configuration
|
||||
@@ -357,8 +351,8 @@ gateway-specific environment variables.
|
||||
OPC servers are configured by the Orchestrator through Redis, one key per slot
|
||||
(`slot:opc_tags:<n>`). The gateway reads the **slot v2** structure (contract C1):
|
||||
one entry per address, with one `names` item per collection / tag name / model, so
|
||||
the same address can feed several models under different names. For secured
|
||||
connections set `use_security: true` (the certificate comes from MinIO, see
|
||||
the same address can feed several models under different names. Each server carries
|
||||
its `security` and its `certificate` (`null` = no security, see
|
||||
[OPC Client Certificate](#-opc-client-certificate)):
|
||||
|
||||
```json
|
||||
@@ -366,11 +360,16 @@ connections set `use_security: true` (the certificate comes from MinIO, see
|
||||
"slot:opc_tags:1": {
|
||||
"server1": {
|
||||
"name": "server1",
|
||||
"server_id": "1",
|
||||
"server_id": "kepserver",
|
||||
"url": "opc.tcp://localhost:4841",
|
||||
"server_uri": "urn:sientia:opc-ingestor",
|
||||
"use_security": true,
|
||||
"subscription_period_ms": 7500,
|
||||
"security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256"},
|
||||
"certificate": {
|
||||
"certificate_id": 1,
|
||||
"application_uri": "urn:sientia:opc-ingestor",
|
||||
"cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-cert.pem"},
|
||||
"private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-key.pem"}
|
||||
},
|
||||
"tags": {
|
||||
"ns=2;i=2": {
|
||||
"frequency": "15000",
|
||||
@@ -386,11 +385,10 @@ connections set `use_security: true` (the certificate comes from MinIO, see
|
||||
}
|
||||
```
|
||||
|
||||
The legacy **slot v1** (each tag with `tag_name` + `topics`, the server with
|
||||
`cert_path`) is still accepted and converted on read (`ingestor/io/slots.py`): one
|
||||
`names` item per topic, and `use_security` is true when `cert_path` is set (the path
|
||||
itself is ignored). This lets the gateway and the Orchestrator be deployed in any
|
||||
order. Documents written from a v1 slot have no `model_id`, exactly as before.
|
||||
Only the slot v2 is accepted (`ingestor/io/slots.py`): a slot without `security`, a
|
||||
tag without `names`, or a name without `collection`, `tag_name` or `model_id` is
|
||||
rejected and notified (`OPC_SLOT_CONFIG_ERROR_<slot>`), and the last good config keeps
|
||||
being read. The gateway and the Orchestrator of SIENTIAPDE-1974 are deployed together.
|
||||
|
||||
## 🔧 Development
|
||||
|
||||
@@ -401,14 +399,13 @@ sientia-dataops-opc-ingestor/
|
||||
│ ├── managers/ # Component managers
|
||||
│ │ ├── data_manager.py # Data persistence and export
|
||||
│ │ ├── ingestor_manager.py # Main coordination
|
||||
│ │ ├── opc_manager.py # Legacy reader, plan B (READ_ENGINE=legacy)
|
||||
│ │ ├── resource_manager.py # Resource coordination
|
||||
│ │ └── server_reader.py # OPC UA server reads (SubscriptionEngine + pool)
|
||||
│ ├── app.py # Main application entry point
|
||||
│ ├── ingestor.py # Core ingestor logic
|
||||
│ └── metrics.py # Prometheus metrics definitions
|
||||
├── simulator/ # OPC simulation and testing tools
|
||||
├── scripts/ # Ops scripts (e2e seed; legacy OPC cert generation)
|
||||
├── scripts/ # Ops scripts (e2e seed)
|
||||
├── tests/ # Test suite
|
||||
└── requirements.txt # Python dependencies
|
||||
```
|
||||
|
||||
354
docs/planning/SIENTIAPDE-1972-banco-de-tags-e-certificados.md
Normal file
354
docs/planning/SIENTIAPDE-1972-banco-de-tags-e-certificados.md
Normal file
@@ -0,0 +1,354 @@
|
||||
# SIENTIAPDE-1972 — Estado atual, pendências e plano: banco de tags + servidores OPC ↔ certificados (MinIO)
|
||||
|
||||
> Levantamento em 28/10/2026, a partir do Jira (1972 e subtasks, 1568/1569/159x), do código do
|
||||
> `sientia-dataops-opc-ingestor` (`release/SIENTIAPDE-1973` @ `d68e842`, 2.0.0-rc4), do
|
||||
> `sientia-dataops-orchestrator_temporal` (`release/SIENTIAPDE-1974`), do Streamlit
|
||||
> (`sientia-projects-templates` `rc/SIENTIAPDE-1588`), do `sientia-dataops-database-migrations`
|
||||
> e do design `aig-docs/07 - Projects & R&D/sientia/opc-gateway/design.md`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Resumo executivo
|
||||
|
||||
- **O gateway funciona no dev** (leitura v2, escrita via NATS, `write_status`) e as subtasks 1973, 1974,
|
||||
1975 e 1985 estão **Concluídas**. Mas **nada disso está na `main`/produção**: o PR #48
|
||||
(`release/SIENTIAPDE-1973` → `main`) está aberto, e a `release/SIENTIAPDE-1974` do orchestrator
|
||||
nem PR para a `main` tem (5 commits à frente, 30 atrás).
|
||||
- **Risco imediato**: a `main` do Laborious (1.6.0) já escreve OPC **só via NATS**. Se ela for para
|
||||
um ambiente com o ingestor 1.4.x, toda escrita OPC fica em `pending`.
|
||||
- **O banco de tags não existe em nenhum ambiente.** `connectors`/`connector_tags` só existiram no dev
|
||||
durante o teste A2 da 1974 e foram apagadas. O orchestrator e o gateway já sabem ler o banco de tags
|
||||
(por `tag_id`), mas sem dado, tudo roda pelo caminho legado (`opc-servers` + `server_id/tag_address`).
|
||||
- **Certificado hoje é um só para a plataforma** (`platform_certificates`, `find_one({})`). Não existe
|
||||
correlação servidor OPC ↔ certificado no Mongo. O `server_cert_path` (cert do servidor) é ignorado
|
||||
com aviso. Bucket em uso: `streamlit-connectors` (o design pede `sientia-dataops`).
|
||||
- **O CRUD (Streamlit 1568/1569) está On Hold** desde 04/2026 para migrar ao Angular; não há card
|
||||
Angular aberto para isso (só a idea `SIENTIAPDI-84`, em parking lot).
|
||||
|
||||
O que falta para fechar o 1972 é pouco (merges + 2202 + 2270). O que vocês tinham pensado — **banco de
|
||||
tags + servidores OPC estruturados com certificados no MinIO** — é praticamente uma **fase 2 inteira**
|
||||
que não tem card hoje. Este documento propõe o modelo e quebra em cards.
|
||||
|
||||
---
|
||||
|
||||
## 2. Estado das subtasks do 1972
|
||||
|
||||
| Card | Status | O que entregou / o que falta |
|
||||
|---|---|---|
|
||||
| 1973 Gateway | ✅ Concluído (02/10) | Escrita NATS, gRPC ReadTag, pool, CertProvider. PRs empilhados #31–#33 fechados, código na `release/SIENTIAPDE-1973` |
|
||||
| 1974 Orchestrator + leitura v2 | ✅ Concluído (09/10) | Gateway #41 (leitura v2, cert do MinIO), #42 (KeyError) mergeados na release. Orchestrator #45 (slot v2, `names[]`, banco de tags opcional por `tag_id`) mergeado na **`release/SIENTIAPDE-1974`**, não na `main`. G6/G7 (A6/A7) não serão feitos: não existe leitura ativa programada; leitura ativa é só on-demand via gRPC, sem persistir (G-AR) |
|
||||
| 1975 Laborious NATS + `write_status` | ✅ Concluído (23/10) | migrations #37 (V16), gateway #43/#47, laborious #5 (main 1.6.0). Validado até o ProSYS (M0–M4) |
|
||||
| 1985 NATS no cluster | ✅ Concluído (30/09) | NATS 2.14.6 via helm. `#108` (release/1.0.0) e `#126` (development) — conferir se ambos já foram mergeados |
|
||||
| **2202** timestamp/qualidade na escrita | 🔄 Em andamento (Eduardo) | Decidir onde fica a config (sugestão: por servidor). **Encaixa direto no modelo de `connectors` deste plano** |
|
||||
| **2270** mlops-bff `write_status` no Model Sustain | 🔄 Em andamento (Carla) | Depende de decisão do PO (falha de escrita deixa o modelo CRITICAL ou não) e da V16 aplicada antes do deploy do BFF |
|
||||
|
||||
---
|
||||
|
||||
## 3. Pendências (consolidado dos comentários dos cards)
|
||||
|
||||
### 3.1 Promoção para produção (bloqueia tudo o mais)
|
||||
|
||||
| # | Pendência | Onde | Situação |
|
||||
|---|---|---|---|
|
||||
| P1 | PR #48 `release/SIENTIAPDE-1973` → `main` (gateway 2.x) | opc-ingestor | Aberto. Combinar número de versão (bump pode não gerar 2.0.0) |
|
||||
| P2 | PR `release/SIENTIAPDE-1974` → `main` | orchestrator | **Não existe.** Release está 30 commits atrás da `main` (0.8.2), com conflito em 4 arquivos (`orchestrator_functions.py`, teste dele, `README.md`, `pyproject.toml`). **Desde 19/10 o dev roda o orchestrator 0.8.1 (`main`, sem slot v2)**: o deploy do PR #49 fez o bump de `GITHUB_BRANCH` 0.7.0-rc1 → 0.8.1 no chart (`development`, commit `07ffc99`) |
|
||||
| P3 | Card do helm (H1–H4 do 1975): migrations com V16, Grafana sem 12/13 e com `write_status`, `docs/nats-broker.md` §11. NATS + envs já estão em todo profile com `opc-ingestor` ligado (desligado só em galvani, quartz-homologacao, gcp-hom, que não têm `opc-ingestor` nem `laborious-worker`) | sientia-helm-chart | **Card não criado** |
|
||||
| P4 | Ordem de deploy por ambiente: migrations V16 → gateway 2.x → laborious 1.6.0 | todos | Documentar no runbook de release |
|
||||
| P5 | Correção da `main` do gateway (SQLAlchemy 2.1 + senha do Mongo no log), branch `fix/sqlalchemy-21-mongo-uri-log` | opc-ingestor | Fica obsoleta se o #48 entrar; senão, abrir PR de hotfix |
|
||||
| P6 | Servidores OPC de produção aceitarem `Basic256Sha256` e confiarem no certificado da plataforma (SAN URI = `server_uri`) | infra/cliente | Pré-requisito do deploy da leitura v2 em cada cliente |
|
||||
| P7 | Certificado da plataforma cadastrado (`platform_certificates` + MinIO) em cada ambiente | por ambiente | Hoje só no dev, cadastrado à mão |
|
||||
|
||||
### 3.2 Gateway (código)
|
||||
|
||||
| # | Pendência | Referência |
|
||||
|---|---|---|
|
||||
| G-AR | **Leitura ativa = unitária e atômica: lê e devolve o valor, nada mais** (decisão 28/10). Sem leitura programada e sem persistência no gRPC. Hoje já se comporta assim (`slot_lookup=None` → `persisted=false`), falta remover o código morto: `scheduled_tick`/`_tick_tag`/`_should_read`/`_ticks` e o parâmetro `names` + `sink.publish_reading` de `read_now`/`_build_result` (`engines/active_read.py`); `slot_lookup` de `ReadTagServicer`/`GrpcServer` (`adapters/grpc_server.py`); `ponytail` de `gateway_runtime.py:10,93`; campo `persisted` do `ReadTagResponse` (marcar `reserved 5` no proto); `active_read` do slot v2 no orchestrator e no `parse_slot`; testes correspondentes | decisão 28/10 |
|
||||
| G-DOC | Atualizar design §6 C4 e §14 ("Reads, persists to the pipeline, and returns the value") e os requisitos G3/G4/A6/A7 do 1974 para refletir a decisão | aig-docs `opc-gateway/design.md` |
|
||||
| G-PB | Remover plano B `READ_ENGINE=legacy` (`OpcManager`, `scripts/generate-and-push-opc-cert.sh`) | critério do 1974; pode sair no #48 ou logo depois |
|
||||
| G-RC | Reconectar ao receber `status_change_notification` (KEPServer derruba sessão a cada 2 h → 2,5–3 min sem dado) | achado do 1974 |
|
||||
| G-SC | Suporte a `server_cert_path` (validar cert do servidor) — hoje só aviso | `ingestor/managers/ingestor_manager.py:708` — resolvido pela Fase 2 abaixo |
|
||||
|
||||
### 3.3 Fora do gateway
|
||||
|
||||
| # | Pendência | Situação |
|
||||
|---|---|---|
|
||||
| C1 | Consumidores dos códigos 12/13/14: `sientia-dataops-report`, `aignosi-scripts` (`sientia_status_report.sh`) | **Sem card** (só o BFF tem, 2270) |
|
||||
| C2 | Laborious: remover `write_opc_data` antigo quando não houver execução anterior ao patch | sem card |
|
||||
| C3 | Limpeza no dev: tabela `sientia_data.e2e_predictions`, pipeline `laborious-test-1975-write`, branches `test-SIENTIAPDE-197x-*` no Gitea | pendente de ok |
|
||||
| C4 | Melhoria de baixa prioridade do scouter (timeout Redis, retries, catch-up) | sugerido, sem card |
|
||||
| C5 | Usuário `aignosi` da VM ProSYS + senha em texto no README do `sientia-iac` | sem card |
|
||||
|
||||
### 3.4 Banco de tags e certificados (bloqueios B1–B6 do 1974 — **nenhum resolvido**)
|
||||
|
||||
| # | Bloqueio | Situação hoje |
|
||||
|---|---|---|
|
||||
| B1 | `connectors`/`connector_tags` não existem em nenhum ambiente | **escopo do orchestrator** (decisão 28/10) |
|
||||
| B2 | Pipelines não gravam `tag_id` em `read_tags`/`write_tags` | **escopo do orchestrator** |
|
||||
| B3 | ~~Onde fica `active_read`~~ | não se aplica: leitura ativa é só on-demand, sem persistir |
|
||||
| B4 | Certificado por ambiente + bucket (`sientia-dataops` do design × `streamlit-connectors` do chart/D7) e correlação servidor ↔ certificado | **escopo do orchestrator**; bucket indefinido |
|
||||
| B5 | Tela de certificados: 1599 entregou no **Streamlit** (rc, não em produção) | não existe no Angular |
|
||||
| B6 | ~~gRPC persist / timestamp nulo~~ | não se aplica: gRPC não persiste; `source_timestamp` vazio só volta vazio na resposta |
|
||||
| — | Renomear tag em `connector_tags` não reimplanta o scouter (só `pipelines.updated_at`) | limitação conhecida |
|
||||
| — | Restos do 1569: 1598 (`tag_id` sequencial sem corrida), 1600 (`metadata`), 1602 (bucket por servidor), 1603 (Server URI), 1604, 1605 (CSV) | On Hold, "passíveis de cancelamento" |
|
||||
|
||||
---
|
||||
|
||||
## 4. Diagnóstico do modelo atual (Mongo + MinIO)
|
||||
|
||||
Hoje coexistem **dois registros de servidor OPC** e **um certificado global**:
|
||||
|
||||
```
|
||||
opc-servers { id, name, url, uri, cert_path, private_key_path, server_cert_path, ... } ← legado, usado em todos os ambientes
|
||||
connectors (opc_ua) { connector_id, type, name, url, server_uri, use_security, subscription_period_ms, status, description }
|
||||
connector_tags { tag_id, connector_id, address, tag_name, category, data_type, unit, status, description, <custom flat keys> }
|
||||
platform_certificates { client_cert: {minio_object_name, sha256_hash, expiration_date, ...}, private_key: {...} } ← 1 documento só
|
||||
MinIO <bucket>/platform/certificates/client_cert_<arquivo>, private_key_<arquivo>
|
||||
```
|
||||
|
||||
Problemas concretos:
|
||||
|
||||
1. **Sem correlação servidor ↔ certificado.** `CertProvider._download_all` faz `find_one('platform_certificates', {})`
|
||||
(`ingestor/connection/certs.py:47`). Um cliente com dois servidores que exigem certificados diferentes
|
||||
(ou um servidor que exija validação do cert do servidor) não é atendível.
|
||||
2. **`server_uri` é dado do certificado, não do servidor.** O `application_uri` do cliente precisa ser
|
||||
igual à SAN URI do certificado da plataforma (comentário 06/10 do 1974). Hoje ele é digitado por
|
||||
conector (`connectors.server_uri` / `opc-servers.uri`) — e o 1603 queria "chumbar" no backend.
|
||||
Erro de digitação = conexão segura recusada sem pista clara.
|
||||
3. **`server_cert_path` ignorado.** O cert do servidor não tem lugar no MinIO/Mongo.
|
||||
4. **Dois espaços de id.** `ServerConfigResolver` procura `connectors.connector_id` e depois `opc-servers.id`
|
||||
com o mesmo `server_id` (`ingestor/connection/servers.py:105-115`); o orchestrator trata "mesmos campos"
|
||||
como mesmo servidor. Se os ids divergirem, um `server_id` aponta para servidores diferentes conforme a fonte.
|
||||
5. **Ids por `$max + 1`** (Streamlit `tag_monitor.py:141`) — corrida em criação simultânea/import em lote.
|
||||
O repo de migrations já tem o padrão certo (`model_id_counters` com `$inc` atômico).
|
||||
6. **Sem schema/índices.** Nada impede `(connector_id, tag_name)` duplicado nem conector sem `url`.
|
||||
7. **Sem visibilidade de expiração** do certificado (o Streamlit calcula `expiration_date`, ninguém alerta).
|
||||
|
||||
---
|
||||
|
||||
## 5. Modelo proposto
|
||||
|
||||
Princípio: **`connectors` vira o registro canônico de servidores OPC**, `opc-servers` vira legado
|
||||
(somente leitura, depois removido); **certificados viram entidade própria** referenciada pelo conector.
|
||||
|
||||
### 5.1 `opc_certificates` (nova; substitui `platform_certificates`)
|
||||
|
||||
```js
|
||||
{
|
||||
certificate_id: 1, // int, counter atômico
|
||||
kind: "client", // client | server | ca
|
||||
name: "sientia-platform-2026",
|
||||
is_default: true, // no máximo 1 client default (índice parcial único)
|
||||
status: "active", // active | retired
|
||||
application_uri: "urn:sientia:opc-ingestor", // SAN URI extraída do cert no upload (kind=client)
|
||||
subject: "CN=sientia-opc-ingestor,...",
|
||||
thumbprint_sha1: "…", // o que aparece no console do KEPServer/servidor
|
||||
not_before: ISODate(), not_after: ISODate(),
|
||||
cert: { bucket, object_name, sha256, size, content_type, original_filename },
|
||||
private_key: { bucket, object_name, sha256, size }, // só kind=client
|
||||
created_at, created_by, updated_at
|
||||
}
|
||||
```
|
||||
|
||||
- Índices: `certificate_id` único; `{is_default:1}` único parcial (`kind:"client", is_default:true`);
|
||||
`not_after` (para alerta).
|
||||
- `application_uri`, `thumbprint`, `not_after` **extraídos do arquivo no upload**, nunca digitados.
|
||||
|
||||
### 5.2 `connectors` (opc_ua) — campos novos
|
||||
|
||||
```js
|
||||
{
|
||||
connector_id: 1, // = opc-servers.id no backfill (mantém server_id estável)
|
||||
type: "opc_ua", name: "kepserver", url: "opc.tcp://…", status: "Active",
|
||||
subscription_period_ms: 1000,
|
||||
security: {
|
||||
mode: "SignAndEncrypt", // None | Sign | SignAndEncrypt
|
||||
policy: "Basic256Sha256",
|
||||
client_certificate_id: null, // null → certificado default da plataforma
|
||||
server_certificate_id: 7 // opcional: valida o cert do servidor (resolve G-SC)
|
||||
},
|
||||
write_options: { write_timestamp: true, write_quality: true }, // SIENTIAPDE-2202 (Kepware: false/false)
|
||||
legacy_opc_server_id: "1", // rastreio do backfill
|
||||
created_at, updated_at
|
||||
}
|
||||
```
|
||||
|
||||
- `use_security` vira derivado (`security.mode != "None"`); manter o campo booleano durante a transição
|
||||
para não quebrar orchestrator/gateway atuais.
|
||||
- `server_uri` **sai do conector**: o `application_uri` vem do certificado cliente resolvido.
|
||||
(Manter leitura do campo antigo como fallback na transição.)
|
||||
- Índices: `connector_id` único; `{type, name}` único.
|
||||
|
||||
### 5.3 `connector_tags` — ajustes
|
||||
|
||||
```js
|
||||
{
|
||||
tag_id: 9001, connector_id: 1, address: "ns=2;s=…", tag_name: "TargetSin",
|
||||
category: "Read", // Read | Write | Prediction Confidence
|
||||
data_type: "Double", unit: "", status: "Active", description: "",
|
||||
metadata: {}, // objeto aninhado (decisão do design §14; fecha a dúvida do 1600)
|
||||
created_at, updated_at
|
||||
}
|
||||
```
|
||||
|
||||
- Índices: `tag_id` único; `{connector_id, tag_name}` único; `{connector_id, address}` (não único —
|
||||
multi-nome é permitido); `updated_at` (orchestrator detectar mudança).
|
||||
|
||||
### 5.4 `connector_id_counters` (padrão `model_id_counters`)
|
||||
|
||||
Documentos `{_id: "connectors"|"connector_tags"|"opc_certificates", seq}`, semeados com o `max` atual,
|
||||
incremento por `find_one_and_update($inc)`. Resolve o 1598 sem `_get_next_tag_id_batch` com corrida
|
||||
(batch = `$inc: N` e usa a faixa).
|
||||
|
||||
### 5.5 MinIO
|
||||
|
||||
```
|
||||
<CERT_BUCKET>/opc-ua/certificates/<certificate_id>/cert.der
|
||||
<CERT_BUCKET>/opc-ua/certificates/<certificate_id>/private_key.pem
|
||||
```
|
||||
|
||||
- Chave por `certificate_id` (não por nome de conector: renomear conector não move objeto; um cert serve vários conectores).
|
||||
- Bucket: **decidir** (ver D1). Policies: escrita só para o usuário do front/BFF; leitura para o gateway.
|
||||
- O `sha256` no Mongo é conferido pelo gateway no download (detecta objeto trocado/corrompido).
|
||||
|
||||
### 5.6 Como o gateway resolve uma conexão (alvo)
|
||||
|
||||
```
|
||||
server_id ──► connectors (connector_id) ──► security.client_certificate_id ?? default
|
||||
│ │
|
||||
│ └─► opc_certificates ─► MinIO (cert+key) ─► cache por certificate_id
|
||||
├─► security.server_certificate_id ─► opc_certificates (kind=server) ─► MinIO
|
||||
└─► application_uri := opc_certificates.application_uri
|
||||
fallback (transição): opc-servers.id + certificado default
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Plano em fases
|
||||
|
||||
### Fase 0 — Orchestrator primeiro, depois levar a 2.x para produção (curto prazo)
|
||||
|
||||
Ordem: **orchestrator antes do gateway**. Sem o slot v2 na `main` do orchestrator, a 2.x em produção só
|
||||
roda o caminho legado (slot v1 convertido): sem multi-nome, sem banco de tags. O orchestrator também é dono
|
||||
de popular o banco de tags, gravar `tag_id` nas pipelines e correlacionar servidores ↔ certificados (Fase 1).
|
||||
|
||||
1. Atualizar `release/SIENTIAPDE-1974` com a `main` (0.8.2), resolver os 4 conflitos, rodar testes + e2e,
|
||||
abrir PR → `main`, deploy no dev e **refazer A1, A2 e A8** (o dev perdeu o slot v2 em 19/10; os testes
|
||||
da 1975 de 19–23/10 rodaram com slot v1).
|
||||
2. Só então revisar/mergear **#48** (gateway → `main`); definir versão `2.0.0`. Decidir se o plano B sai aqui (G-PB).
|
||||
3. **Criar card do helm (P3)** com H1–H4.
|
||||
4. Runbook de deploy por ambiente (P4, P6, P7): checklist V16 → gateway → laborious; cert cadastrado; servidor
|
||||
confia no cert; Kepware com `write_timestamp/quality=false`.
|
||||
5. Concluir **2202** já usando `connectors.write_options` (fallback env `WRITE_TIMESTAMP/WRITE_QUALITY`) — é
|
||||
o primeiro campo do modelo novo e não depende do resto.
|
||||
6. **2270**: obter decisão do PO; criar cards de C1 (report, aignosi-scripts).
|
||||
7. Limpeza do dev (C3).
|
||||
|
||||
**Saída:** 1972 pode ser fechado. Tudo abaixo vira **nova história/épico** (sugestão: "Banco de tags e
|
||||
certificados OPC estruturados", ligada à `SIENTIAPDI-84`).
|
||||
|
||||
### Fase 1 — Modelo de dados + backfill — escopo do orchestrator (collections/índices via migrate-mongo no `sientia-dataops-database-migrations`)
|
||||
|
||||
1. Migration `…-opc-tag-database.js`: cria `connectors`, `connector_tags`, `opc_certificates`,
|
||||
`connector_id_counters` com `$jsonSchema` (validationLevel `moderate` na transição) e os índices de §5.
|
||||
2. Migration `…-backfill-tag-database.js` (idempotente, só insere o que não existe):
|
||||
- `opc-servers` → `connectors` com **`connector_id = int(opc-servers.id)`**, `security.mode` derivado de
|
||||
`cert_path`, `legacy_opc_server_id`.
|
||||
- `platform_certificates` → `opc_certificates` (id 1, `is_default: true`), copiando os metadados MinIO
|
||||
existentes (sem mover objeto ainda).
|
||||
- `pipelines.read_tags[]`/`write_tags[]` (`server_id`, `tag_address`, `tag_name`) → `connector_tags`
|
||||
(dedupe por `(connector_id, tag_name)`), e **grava `tag_id` de volta na pipeline** (resolve B1 + B2 de uma vez).
|
||||
- semeia os counters com os máximos.
|
||||
3. Dry-run com dump do dev e de um cliente (relatório de conflitos: mesmo `tag_name` com endereços diferentes,
|
||||
`server_id` sem servidor).
|
||||
|
||||
### Fase 2 — Gateway (`sientia-dataops-opc-ingestor`)
|
||||
|
||||
1. `CertProvider` → `CertificateStore`: `fetch(certificate_id | None)`, cache e lock **por `certificate_id`**,
|
||||
verifica `sha256`; `None` = default (`is_default`), fallback `platform_certificates` durante a transição.
|
||||
2. `ServerConfigResolver` + slot: conexão passa a carregar `client_certificate_id`, `server_certificate_id`,
|
||||
`application_uri`, `write_options`. Pool usa esses campos em `apply_security` (passa `server_certificate`).
|
||||
3. Validar no carregamento: SAN do cert == `application_uri`; cert não expirado → erro claro + notificação
|
||||
(`OPC_CERTIFICATE_*`), em vez de falha opaca do asyncua.
|
||||
4. Métrica `opc_certificate_expiry_seconds{certificate_id}` + alerta (ex.: < 30 dias).
|
||||
5. `WriteEngine`: `write_timestamp/quality` = comando → `connectors.write_options` → env (fecha 2202 de vez).
|
||||
6. G-AR: limpar leitura ativa para leitura unitária sem persistência (pode ir junto com o #48 ou logo depois) + G-DOC.
|
||||
|
||||
### Fase 3 — Orchestrator (`sientia-dataops-orchestrator_temporal`)
|
||||
|
||||
1. Fonte de servidor: `connectors` primeiro, `opc-servers` fallback (hoje é o contrário: só usa `connectors`
|
||||
quando a pipeline tem `tag_id`).
|
||||
2. Slot v2 por servidor: `connector_id`, `client_certificate_id`, `server_certificate_id`, `application_uri`
|
||||
(no lugar de `use_security`/`server_uri` soltos — manter os antigos até o gateway novo estar em todo lugar).
|
||||
3. Remover `active_read` do slot v2 (sem leitura ativa programada).
|
||||
4. Reagir a mudança em `connector_tags.updated_at`/`connectors.updated_at` (reimplantar scouter/slot) — fecha a
|
||||
limitação de renomear tag.
|
||||
|
||||
### Fase 4 — CRUD no Angular (front `sientia` + BFF)
|
||||
|
||||
1. Novo épico/história substituindo 1568/1569: telas de **Conectores**, **Tags** (com import CSV — escopo do 1605)
|
||||
e **Certificados** (upload extrai SAN/thumbprint/validade; marca default; lista conectores que usam).
|
||||
2. BFF é quem escreve no Mongo/MinIO (front não fala com MinIO direto); usa os counters.
|
||||
3. Tela de pipeline passa a escolher tag por `tag_id` (pipelines novas já nascem com `tag_id`).
|
||||
4. Cancelar no Jira os restos Streamlit não iniciados (1600, 1604, 1605, 1603) com link para o novo card.
|
||||
|
||||
### Fase 5 — Desligar o legado
|
||||
|
||||
1. Orchestrator e gateway sem fallback para `opc-servers`/`platform_certificates`; `server_id/tag_address`
|
||||
nas pipelines viram só leitura histórica.
|
||||
2. Remover `opc-servers`, `platform_certificates` (migration com backup) e o caminho Streamlit.
|
||||
3. `$jsonSchema` em `validationLevel: strict`.
|
||||
|
||||
---
|
||||
|
||||
## 7. Decisões necessárias
|
||||
|
||||
| # | Decisão | Recomendação | Dono |
|
||||
|---|---|---|---|
|
||||
| D1 | Bucket dos certificados: `sientia-dataops` (design §7/§14, Streamlit rc) × `streamlit-connectors` (chart, D7 da 1974) | `sientia-dataops` (bucket por serviço, estratégia MinIO); migrar objetos na Fase 1 e trocar `CERT_BUCKET` no chart | Vitor / platform |
|
||||
| D2 | Cert por servidor ou único da plataforma | Modelo suporta os dois; **default único** + override por conector só quando necessário | Vitor |
|
||||
| D3 | `connector_id` = `opc-servers.id` no backfill | Sim — mantém `server_id` de slots, `write_tags` e métricas estáveis | Eduardo |
|
||||
| D6 | `metadata` aninhado × chaves soltas (1600) | Aninhado (`metadata: {}`), como no design §14 | PO |
|
||||
| D7 | `write_options` por servidor (2202) | Por servidor, padrão `true`; Kepware configurado `false` antes do deploy | Eduardo |
|
||||
| D8 | Quem escreve banco de tags em produção até o Angular existir | Backfill por migration + edição via migration/script revisado (sem Streamlit em produção) | Bruno / PO |
|
||||
| D9 | Plano B `READ_ENGINE=legacy` sai no #48? | Sim, se A1–A8 seguem ok no dev com a 2.0.0-rc4 | Eduardo |
|
||||
|
||||
---
|
||||
|
||||
## 8. Cards a criar / ajustar
|
||||
|
||||
| Card | Tipo | Repo | Fase |
|
||||
|---|---|---|---|
|
||||
| Helm: V16, NATS+gateway 2.x nos profiles, `OPC_WRITE_MODE`, Grafana `write_status`, docs (H1–H4) | Subtask do 1972 | sientia-helm-chart | 0 |
|
||||
| PR `release/SIENTIAPDE-1974` → `main` (orchestrator) | Subtask do 1972 | orchestrator | 0 |
|
||||
| Consumidores 12/13/14: report + aignosi-scripts | Subtask | report, aignosi-scripts | 0 |
|
||||
| **[Nova história] Banco de tags e certificados OPC estruturados** | História (épico 1274 ou novo) | — | 1–5 |
|
||||
| ↳ Orchestrator: popular banco de tags, gravar `tag_id` nas pipelines, correlação servidor ↔ certificado (collections/índices via migrate-mongo) | Subtask | orchestrator, database-migrations | 1 |
|
||||
| ↳ Gateway: `CertificateStore` por `certificate_id`, validação SAN/expiração, métrica de expiração | Subtask | opc-ingestor | 2 |
|
||||
| ↳ Gateway: leitura ativa unitária sem persistência (G-AR) + design (G-DOC) | Subtask | opc-ingestor, orchestrator, aig-docs | 2 |
|
||||
| ↳ Gateway: reconexão por `status_change_notification` | Subtask | opc-ingestor | 2 |
|
||||
| ↳ Orchestrator: `connectors` como fonte, slot com certificados, sem `active_read`, reação a `updated_at` | Subtask | orchestrator | 3 |
|
||||
| ↳ Angular + BFF: CRUD conectores/tags/certificados (substitui 1568/1569) | História própria | sientia, mlops-bff | 4 |
|
||||
| ↳ Remoção do legado (`opc-servers`, `platform_certificates`, `READ_ENGINE=legacy`) | Subtask | vários | 5 |
|
||||
|
||||
---
|
||||
|
||||
## 9. Riscos
|
||||
|
||||
- **Laborious 1.6.0 antes do gateway 2.x** em qualquer ambiente → escritas presas em `pending` (P4).
|
||||
- **Backfill com ids divergentes** entre `opc-servers` e um `connectors` já existente em algum cliente (o
|
||||
Streamlit rc pode ter rodado em algum lugar) → checar antes, migration aborta em conflito.
|
||||
- **Troca de bucket/objeto do certificado** derruba conexões seguras se o gateway não invalidar o cache —
|
||||
já existe `invalidate()` em falha; manter e testar por `certificate_id`.
|
||||
- **Servidores de cliente com Basic256 apenas** (asyncua marca deprecated) → leitura v2 não conecta; levantar por cliente antes.
|
||||
|
||||
---
|
||||
|
||||
## 10. Referências
|
||||
|
||||
- Jira: SIENTIAPDE-1972, 1973, 1974, 1975, 1985, 2202, 2270, 1568, 1569, 1598, 1599, 1603, 1605; idea SIENTIAPDI-84.
|
||||
- Código: `ingestor/connection/certs.py`, `ingestor/connection/servers.py`, `ingestor/gateway_runtime.py`,
|
||||
`ingestor/managers/ingestor_manager.py:708`, `ingestor/io/slots.py`.
|
||||
- Orchestrator: `release/SIENTIAPDE-1974` — `orchestrator/activities/formatters.py`, `orchestrator/utils/orchestrator_functions.py`.
|
||||
- Streamlit: `sientia-projects-templates` `rc/SIENTIAPDE-1588` — `app/pages/connectors.py` (`_save_platform_certificate`), `app/pages/tag_monitor.py`.
|
||||
- Migrations: `mongodb/migrations/20260908170557-model_id_counters_collection.js` (padrão de counter).
|
||||
- Design: `aig-docs/07 - Projects & R&D/sientia/opc-gateway/design.md` §3, §7, §14, §15.
|
||||
241
docs/planning/SIENTIAPDE-1974-certificado-por-conector.md
Normal file
241
docs/planning/SIENTIAPDE-1974-certificado-por-conector.md
Normal file
@@ -0,0 +1,241 @@
|
||||
# SIENTIAPDE-1974 — Certificado OPC por conector (`opc_certificates`)
|
||||
|
||||
> Plano de implementação. Atualizado em 2026-10-29.
|
||||
> Repositórios: `sientia-dataops-database-migrations`, `sientia-helm-chart`,
|
||||
> `sientia-dataops-orchestrator_temporal`, `sientia-dataops-opc-ingestor`.
|
||||
|
||||
## Objetivo
|
||||
|
||||
Ligar cada servidor OPC (`connectors`) ao certificado que ele usa (`opc_certificates`) por **um único caminho**,
|
||||
sem transição, sem fallback e sem legado:
|
||||
|
||||
```
|
||||
connectors.security.client_certificate_id ──► opc_certificates.certificate_id
|
||||
│
|
||||
orchestrator: faz o join e grava no slot v2, por servidor, security + certificado (paths no MinIO + application_uri)
|
||||
│
|
||||
ingestor: lê o slot, baixa cert + chave do MinIO e conecta. Sem certificado no slot → conecta sem segurança
|
||||
```
|
||||
|
||||
O ingestor não usa certificado "default", `platform_certificates`, `opc-servers`, nem o caminho legado
|
||||
(`READ_ENGINE=legacy`, `OpcManager`, slot v1).
|
||||
|
||||
**Exceção consciente (D1):** a escrita mantém um fallback no Mongo. Quando o pod que recebe o comando não tem a
|
||||
conexão no pool, ele resolve o servidor em `connectors` → `opc_certificates`, com a mesma regra do orchestrator. São
|
||||
duas fontes da mesma regra, aceitas por enquanto. A otimização fica num card próprio (ver "Fora do escopo").
|
||||
|
||||
**Fora do escopo:** migração e correção de dados. As migrations deste plano tratam só de estrutura.
|
||||
|
||||
---
|
||||
|
||||
## Estado atual (verificado)
|
||||
|
||||
| Item | Estado | Evidência |
|
||||
|---|---|---|
|
||||
| `platform_certificates` | Criada pelo front Streamlit, sem migration | `sientia-projects-templates` `origin/rc/SIENTIAPDE-1588` `app/pages/connectors.py:27` |
|
||||
| `opc_certificates` | Criada pela migration `20261028170000-opc_tag_database_collections.js`, com schema parcial. Tem dados no dev | Migration, consulta no dev |
|
||||
| Vínculo connector → certificado | O campo existe no schema (`connectors.security.client_certificate_id`). No dev está `null` | Migration, consulta no dev |
|
||||
| `security.mode` / `security.policy` | Estão no schema e nos dados do dev, mas o ingestor ignora os dois: usa `use_security` e fixa `Basic256Sha256` no código | `ingestor/connection/security.py`, `pool.py:130` |
|
||||
| `application_uri` | Vem de `connectors.server_uri`, mas é dado do certificado (tem que bater com a SAN URI do `.pem`) | `pool.py:126-132`, `security.py:9` |
|
||||
| Leitura do certificado | `CertProvider` faz `find_one('platform_certificates', {})` e baixa do `CERT_BUCKET` | `ingestor/connection/certs.py:10,47` |
|
||||
| Bucket `opc-certificates` | Citado em `opc_certificates.cert.bucket`, mas não provisionado. O chart aponta `CERT_BUCKET=streamlit-connectors` | `sientia-helm-chart/values/minio.yaml:516-523`, `opc-ingestor/values.yaml:116` |
|
||||
| Mudança de config de um servidor no slot | O ingestor compara a config inteira do servidor (sem `tags`) e reconecta se algo mudou | `ingestor/managers/ingestor_manager.py:293-307` |
|
||||
| Escrita (NATS) | Core NATS, sem JetStream, entrega *at-most-once*. Queue group `gateway-writers`: cada comando vai para um só pod, que não tem como devolvê-lo. O Laborious publica e esquece; o resultado vai para `write_status` no Postgres | `sientia-helm-chart/docs/nats-broker.md:29-39`, `ingestor/adapters/nats_consumer.py`, `laborious/activities/nats_write.py` |
|
||||
|
||||
---
|
||||
|
||||
## Decisões
|
||||
|
||||
| # | Decisão | Status |
|
||||
|---|---|---|
|
||||
| D0 | `opc_certificates` é a oficial; o ingestor deixa de ler a `platform_certificates` (a collection fica) | **Decidido** |
|
||||
| D0.1 | Migration da `opc_certificates` completa, documentada e auditável | **Decidido** (Etapa 1) |
|
||||
| D0.2 | Bucket `opc-certificates` provisionado no `sientia-helm-chart`, branch `feature/SIENTIAPDE-1974-connectors-pi-web-api` | **Decidido** (Etapa 2) |
|
||||
| D0.3 | Sem transição. Caminho único na leitura; a escrita tem a exceção de D1 | **Decidido** |
|
||||
| D1 | Escrita: pool primeiro; sem conexão no pool, resolve no Mongo (`connectors` → `opc_certificates`) com a mesma regra do orchestrator. Roteamento ao pod dono vira card novo | **Decidido** (ver abaixo) |
|
||||
| D2 | Zero legado: `READ_ENGINE=legacy`, `OpcManager`, slot v1, `opc-servers` | **Decidido** |
|
||||
| D3 | Dados fora do plano; migrations só de estrutura; sem runbook | **Decidido** |
|
||||
| D4 | `security.mode` + `security.policy` são a fonte. `use_security` e `server_uri` saem | **Decidido** |
|
||||
| D5 | Sem certificado registrado, o servidor é tratado como um que não exige certificado: conecta sem segurança (e falha se o servidor exigir) | **Decidido** |
|
||||
| D6 | Branch do ingestor: `release/SIENTIAPDE-1973` | **Decidido** |
|
||||
| D7 | `created_at`, `updated_at`, `created_by`, `updated_by` obrigatórios. Rastro do backfill não é validado. Sem `additionalProperties: false` | **Decidido** |
|
||||
|
||||
### D1 — Escrita: pool primeiro, Mongo como fallback (decidido)
|
||||
|
||||
**Como funciona hoje:** o Laborious publica o comando (C2) em `opc.write` só com o `server_id`. No ingestor, o
|
||||
`ServerConfigResolver` (`ingestor/connection/servers.py`) usa a conexão do pool, se existir. Senão, consulta o Mongo
|
||||
(`connectors` e depois `opc-servers`), com cache de 60 s (10 s quando não encontra), e o pool abre a conexão.
|
||||
|
||||
**O que muda:**
|
||||
- Sai `opc-servers` (D2) e sai o override por dados de conexão no comando (o Laborious nunca envia).
|
||||
- O lookup no Mongo passa a fazer o join `connectors` → `opc_certificates` e monta a config **no mesmo formato do
|
||||
servidor no slot** (`server_id`, `url`, `security`, `certificate`). O pool e o `CertProvider` só conhecem esse formato.
|
||||
- A regra do certificado (D5: certificado `active` e do tipo `client`, senão `certificate: None`) fica isolada numa
|
||||
função do ingestor, com testes que espelham os do orchestrator.
|
||||
|
||||
**Por que não só o pool:** o Core NATS entrega cada comando a um único pod, sem devolver nem reentregar. Rotear o
|
||||
comando ao pod dono da conexão (subject por servidor, republicação ou JetStream) não cobre um servidor que nenhum pod
|
||||
tem no pool (servidor só de escrita, ou pod recém-reiniciado) e ainda exige uma regra de expiração do `pending`.
|
||||
O fallback no Mongo cobre todos esses casos hoje.
|
||||
|
||||
**Custo aceito:**
|
||||
- Duas fontes da mesma regra (orchestrator e ingestor).
|
||||
- Sessões duplicadas: o pod que recebe uma escrita sem ter a conexão abre a sua própria sessão (já acontece hoje).
|
||||
|
||||
### D5 — Sem certificado registrado (decidido)
|
||||
|
||||
- O orchestrator grava `certificate: None` no slot quando `client_certificate_id` é `null`, aponta para um id que não
|
||||
existe, ou aponta para um certificado que não é `active` ou não é `client`. Nos dois últimos casos (id preenchido,
|
||||
mas sem certificado válido), o orchestrator registra um aviso em `issues`, porque provavelmente é erro de cadastro.
|
||||
Mesmo assim o servidor entra no slot.
|
||||
- O ingestor aplica segurança **só quando o slot tem `certificate`**. Nesse caso usa `security.mode` e
|
||||
`security.policy`. Sem `certificate`, conecta sem segurança, e um servidor que exige certificado recusa a conexão.
|
||||
O erro aparece no log e nas métricas de conexão do ingestor.
|
||||
- Não existe certificado "default": `is_default` e o índice parcial `ux_opc_certificates_default_client` saem do schema.
|
||||
|
||||
---
|
||||
|
||||
## Etapa 1 — Migrations de estrutura (`sientia-dataops-database-migrations`)
|
||||
|
||||
**Nova migration** `mongodb/migrations/<timestamp>-opc_certificates_schema.js` (SIENTIAPDE-1974):
|
||||
|
||||
1. **Schema da `opc_certificates`**, com o cabeçalho do arquivo descrevendo cada campo:
|
||||
|
||||
| Campo | Tipo | Obrigatório | Descrição |
|
||||
|---|---|---|---|
|
||||
| `certificate_id` | int/long | sim | Id estável, referenciado por `connectors.security.client_certificate_id` |
|
||||
| `kind` | `client` \| `server` \| `ca` | sim | `client`: cert + chave do ingestor. `server`/`ca`: só cert |
|
||||
| `name` | string | sim | Nome legível, único por `kind` |
|
||||
| `status` | `active` \| `retired` | sim | Só `active` entra no slot |
|
||||
| `application_uri` | string | sim se `client` | SAN URI do certificado, usada como `client.application_uri` |
|
||||
| `cert` | `{bucket, object_name, sha256}` | sim (`bucket`, `object_name`) | Objeto no MinIO. `sha256`: string (64 hex) ou null |
|
||||
| `private_key` | `{bucket, object_name, sha256}` | sim se `client`, proibido nos outros | Chave privada no MinIO |
|
||||
| `thumbprint_sha1` | string (40 hex) \| null | não | O que aparece no console do servidor OPC |
|
||||
| `not_before` / `not_after` | date \| null | não | Validade (`not_after` já tem índice) |
|
||||
| `created_at` / `updated_at` | date | sim | Auditoria |
|
||||
| `created_by` / `updated_by` | string | sim | Auditoria: quem cadastrou ou trocou o certificado |
|
||||
|
||||
- "Obrigatório se `client`" fica num `anyOf` do `$jsonSchema`.
|
||||
- Campos fora da tabela (como o rastro do backfill) não são descritos nem validados: sem `additionalProperties: false`.
|
||||
- Índices: novo `ux_opc_certificates_kind_name` em `{kind, name}`, único. Sai `ux_opc_certificates_default_client` (D5). Ficam `ux_opc_certificates_certificate_id` e `ix_opc_certificates_not_after`.
|
||||
- `validationLevel: strict`.
|
||||
- `down`: volta ao validator e aos índices da `20261028170000`.
|
||||
|
||||
2. **Schema dos `connectors`** (parte do schema da `20261028190000-connectors_pi_web_api_types.js`):
|
||||
- Saem `use_security` e `server_uri` (D4).
|
||||
- `security` passa a ser obrigatório para `type: 'opc_ua'`, com `mode` e `policy` obrigatórios.
|
||||
- `security.client_certificate_id` continua opcional/nullable (D5).
|
||||
- `down` restaura o schema anterior.
|
||||
|
||||
3. **Testes** em `tests/`, no padrão de `test_connectors_types.py` e `test_head.py`:
|
||||
- Documentos válidos e inválidos por `kind` (com e sem `private_key` e `application_uri`).
|
||||
- Campos de auditoria ausentes.
|
||||
- Unicidade de `{kind, name}`.
|
||||
- Connector `opc_ua` com e sem `security`, e com `client_certificate_id` nulo.
|
||||
- Roundtrip `up` → `down`.
|
||||
- Head esperado.
|
||||
|
||||
## Etapa 2 — Provisionamento (`sientia-helm-chart`, branch `feature/SIENTIAPDE-1974-connectors-pi-web-api`)
|
||||
|
||||
1. `values/minio.yaml`: bucket `opc-certificates` em `provisioning.buckets` (linhas 516-523).
|
||||
2. Acesso: o ingestor usa as credenciais root do MinIO (`opc-ingestor/values.yaml:106-115`). Avaliar com quem cuida
|
||||
da plataforma um usuário e uma policy de **somente leitura** em `opc-certificates`, como `model-training-user`.
|
||||
3. `src/charts-internal/opc-ingestor/values.yaml:98-117`: remover a env `CERT_BUCKET` e o comentário sobre
|
||||
`platform_certificates`. Remover `READ_ENGINE`, se estiver definida em algum profile (D2).
|
||||
4. `docs/nats-broker.md:373`: trocar a menção à `platform_certificates` pelo fluxo novo. O NATS não muda (D1).
|
||||
|
||||
## Etapa 3 — Orchestrator (`sientia-dataops-orchestrator_temporal`)
|
||||
|
||||
1. **`orchestrator/workflows/orchestrator.py` (`DEFAULT_CONNECTORS_QUERY`)**:
|
||||
- Projetar `security.mode`, `security.policy` e `security.client_certificate_id`. Saem `server_uri` e `use_security`.
|
||||
- `$lookup` de `security.client_certificate_id` para `opc_certificates.certificate_id`, com `status: 'active'` e `kind: 'client'`.
|
||||
- Projetar `certificate_id`, `application_uri`, `cert.{bucket,object_name}` e `private_key.{bucket,object_name}`. As datas não entram (BSON que o payload do Temporal não carrega).
|
||||
2. **`orchestrator/utils/opc_slots.py`**: servidor do slot v2 (contrato C1):
|
||||
```python
|
||||
{
|
||||
'server_id': ..., 'name': ..., 'url': ...,
|
||||
'subscription_period_ms': ...,
|
||||
'security': {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'},
|
||||
'certificate': { # None: sem certificado registrado (D5)
|
||||
'certificate_id': 1,
|
||||
'application_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/sientia-platform/opc-ingestor-cert.pem'},
|
||||
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/sientia-platform/opc-ingestor-key.pem'},
|
||||
},
|
||||
'tags': {...},
|
||||
}
|
||||
```
|
||||
3. **`orchestrator/utils/tag_database.py`**:
|
||||
- `CONNECTOR_FIELDS` sem `server_uri`.
|
||||
- `client_certificate_id` preenchido, mas sem certificado ativo do tipo `client` → `certificate: None` mais aviso em `issues` e em `formatters.py` (D5).
|
||||
4. **Testes**:
|
||||
- `test_opc_slots.py`, `test_tag_database.py`, `tag_database_fixtures.py`, `test_formatters.py`.
|
||||
- `test_orchestrator.py` e a history de replay, se o input do workflow mudar.
|
||||
- Cenários e2e.
|
||||
5. **Docs**: README (seção "OPC slots", contrato C1) e CLAUDE.md.
|
||||
|
||||
## Etapa 4 — Ingestor (`sientia-dataops-opc-ingestor`, branch `release/SIENTIAPDE-1973`)
|
||||
|
||||
**Certificado e segurança**
|
||||
|
||||
| Arquivo | Mudança |
|
||||
|---|---|
|
||||
| `ingestor/connection/certs.py` | Sai o Mongo (`COLLECTION_NAME`, `find_one`, o `mongo` do construtor). `fetch(certificate)` baixa o `bucket/object_name` vindo do slot. Cache, lock e pasta (`<cache_dir>/<certificate_id>/`) separados por `certificate_id`. `invalidate(certificate_id)` |
|
||||
| `ingestor/config.py` | Sai o `cert_bucket`; fica o `cert_cache_dir` |
|
||||
| `ingestor/connection/security.py` | `apply_security(client, security, certificate, cert_path, key_path)`, com o `application_uri` do certificado e mode e policy do slot (mapa de string para classe do asyncua; desconhecida → erro) |
|
||||
| `ingestor/connection/pool.py` | `_connect`: aplica segurança só quando `certificate` não é `None` (D5). Sai o `server_uri`. `_open` invalida o certificado da conexão. `_check_cfg` passa a comparar `security` e `certificate` (só aviso; a reconexão vem do `ingestor_manager`) |
|
||||
| `ingestor/io/slots.py` | `SERVER_FIELDS`: `server_id`, `url`, `subscription_period_ms`, `security`, `certificate`. Sai a conversão do slot v1 |
|
||||
|
||||
**Escrita (D1)**
|
||||
|
||||
| Arquivo | Mudança |
|
||||
|---|---|
|
||||
| `ingestor/connection/servers.py` | `ServerConfigResolver`: pool primeiro (`config_for`); senão, Mongo, com o cache de TTL atual. `_lookup` faz uma aggregation em `connectors` (`type: 'opc_ua'`, `status: 'Active'`) com `$lookup` em `opc_certificates`, e devolve a config no formato do servidor no slot (`server_id`, `url`, `security`, `certificate`). Saem `opc-servers` (`OPC_SERVERS`, `_opc_server_cfg`, `_id_forms`, já que `connector_id` é slug string) e o override pelo comando |
|
||||
| `ingestor/connection/certificates.py` (novo) ou dentro de `servers.py` | Função que, a partir do documento do connector e do certificado do `$lookup`, devolve `certificate` ou `None` com a regra de D5 (certificado `active` e `client`; id preenchido sem certificado válido → aviso no log). Testes que espelham os do orchestrator |
|
||||
| `ingestor/engines/write.py` | `_server_cfg` sem o ramo de conexão vinda do comando. Sem mudança de comportamento: servidor não encontrado continua indo para erro no `write_status` |
|
||||
|
||||
**Remoção do legado (D2)**
|
||||
|
||||
| Arquivo | Mudança |
|
||||
|---|---|
|
||||
| `ingestor/managers/opc_manager.py` | Removido |
|
||||
| `ingestor/managers/ingestor_manager.py` | Saem `READ_ENGINES`/`READ_ENGINE`, os ramos `legacy`, a checagem de slot v1 (`:698`) e `_warn_server_cert_path` (`:708`) |
|
||||
| `ingestor/io/data_sink.py` | Sai o tratamento de slot v1 (`:56`) |
|
||||
| `ingestor/gateway_runtime.py` | `CertProvider` sem Mongo. `ServerConfigResolver` continua com Mongo (D1) |
|
||||
| `tests/` | Remover os testes do legado; adaptar `certs`, `pool`, `slots`, `servers`, `ingestor_manager` e `write` |
|
||||
| `README.md`, `CLAUDE.md` | Sem plano B e sem `platform_certificates` (README:67 e :236) |
|
||||
|
||||
## Etapa 5 — Deploy (por ambiente)
|
||||
|
||||
1. Migrations (Etapa 1). Os dados precisam estar migrados antes (fora do escopo): o `strict` rejeita documento fora do schema.
|
||||
2. Helm (Etapa 2).
|
||||
3. Ingestor e orchestrator **juntos**. Sem transição, um slot antigo não tem `security`/`certificate`.
|
||||
4. Validação no dev:
|
||||
- O slot no Redis tem `security` e `certificate`.
|
||||
- O ingestor conecta ao `kepserver` com o mode e a policy do banco.
|
||||
- A troca de `client_certificate_id` reconecta com o certificado novo.
|
||||
- Um servidor sem certificado conecta sem segurança.
|
||||
- Um id de certificado inválido gera aviso.
|
||||
- Uma escrita para um servidor que está no pool usa a conexão do pool.
|
||||
- Uma escrita para um servidor fora do pool resolve pelo Mongo, com o mesmo certificado que o slot teria.
|
||||
- Uma escrita para um `server_id` inexistente termina em erro no `write_status`.
|
||||
|
||||
## Fora do escopo
|
||||
|
||||
- Migração e correção de dados (`opc_certificates`, `connectors`, objetos no MinIO).
|
||||
- Remoção da collection `platform_certificates` (o ingestor só deixa de lê-la).
|
||||
- Validação do certificado do servidor (`security.server_certificate_id`, `kind: server`).
|
||||
- Métrica e alerta de expiração (`not_after`).
|
||||
- Conferência de `sha256` no download.
|
||||
- Upload e CRUD de certificados no front Angular/BFF (quem preenche `created_by`/`updated_by`).
|
||||
- **Card novo — Escrita OPC roteada ao pod dono da conexão:** subject por servidor (`opc.write.<server_id>`), slot
|
||||
cobrindo servidores só de escrita e tratamento de comando sem destino (expiração do `pending` ou `request` com
|
||||
*no responders*). Quando estiver pronto, remove o fallback no Mongo do `ServerConfigResolver` e a segunda fonte da
|
||||
regra de certificado (D1).
|
||||
|
||||
## Referências
|
||||
|
||||
- Plano geral: `docs/planning/SIENTIAPDE-1972-banco-de-tags-e-certificados.md` (§5.1, §5.2, Fases 1–3).
|
||||
- Migrations: `mongodb/migrations/20261028170000-opc_tag_database_collections.js`, `20261028190000-connectors_pi_web_api_types.js`.
|
||||
- Escrita: `sientia-dataops-laborious` `laborious/activities/nats_write.py`; ingestor `ingestor/connection/servers.py`, `ingestor/engines/write.py:148-160`; `sientia-helm-chart/docs/nats-broker.md`.
|
||||
- Streamlit (origem da `platform_certificates`): `sientia-projects-templates` `origin/rc/SIENTIAPDE-1588` `app/pages/connectors.py`.
|
||||
@@ -22,9 +22,8 @@ class GatewayConfig:
|
||||
nats_queue: str = 'gateway-writers'
|
||||
grpc_enabled: bool = False
|
||||
grpc_port: int = 50051
|
||||
# Legacy frontend bucket the platform already provisions (SIENTIAPDE-1974 D7);
|
||||
# the design's `sientia-dataops` bucket does not exist in the platform MinIO.
|
||||
cert_bucket: str = 'streamlit-connectors'
|
||||
# The certificate bucket and objects come from the slot / the write lookup
|
||||
# (opc_certificates, SIENTIAPDE-1974); only the local cache dir is configured here.
|
||||
cert_cache_dir: str = '/tmp/opc-certs' # noqa: S108 # nosec B108 - default is spec'd, overridable via env
|
||||
write_timestamp: bool = True
|
||||
write_quality: bool = True
|
||||
@@ -36,10 +35,7 @@ class GatewayConfig:
|
||||
"""Only the certificate settings — what the read path needs. Plain strings,
|
||||
no number parsing, so a malformed write/gRPC env var (GRPC_PORT,
|
||||
WRITE_MIN_PAUSE_S, ...) can only disable the runtime, never the reads."""
|
||||
return cls(
|
||||
cert_bucket=getenv('CERT_BUCKET', cls.cert_bucket),
|
||||
cert_cache_dir=getenv('CERT_CACHE_DIR', cls.cert_cache_dir),
|
||||
)
|
||||
return cls(cert_cache_dir=getenv('CERT_CACHE_DIR', cls.cert_cache_dir))
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> 'GatewayConfig':
|
||||
@@ -49,7 +45,6 @@ class GatewayConfig:
|
||||
nats_queue=getenv('NATS_QUEUE', cls.nats_queue),
|
||||
grpc_enabled=_env_bool('GRPC_ENABLED', cls.grpc_enabled),
|
||||
grpc_port=int(getenv('GRPC_PORT', cls.grpc_port)),
|
||||
cert_bucket=getenv('CERT_BUCKET', cls.cert_bucket),
|
||||
cert_cache_dir=getenv('CERT_CACHE_DIR', cls.cert_cache_dir),
|
||||
write_timestamp=_env_bool('WRITE_TIMESTAMP', cls.write_timestamp),
|
||||
write_quality=_env_bool('WRITE_QUALITY', cls.write_quality),
|
||||
|
||||
92
ingestor/connection/certificates.py
Normal file
92
ingestor/connection/certificates.py
Normal file
@@ -0,0 +1,92 @@
|
||||
"""Client certificate of a tag database connector (SIENTIAPDE-1974 D5).
|
||||
|
||||
Mirrors the Orchestrator's `tag_database.resolve_certificate`, which puts the
|
||||
certificate in the read slot: the write lookup (ServerConfigResolver, D1) resolves a
|
||||
server that is not in the pool with the same rule, so a write uses the certificate the
|
||||
slot would have. Keep both in sync.
|
||||
"""
|
||||
|
||||
from typing import Any
|
||||
|
||||
# security fields of an OPC UA connector: the source of its OPC security (D4)
|
||||
SECURITY_FIELDS = ('mode', 'policy')
|
||||
CERTIFICATE_FIELDS = ('certificate_id', 'application_uri')
|
||||
CERTIFICATE_OBJECTS = ('cert', 'private_key')
|
||||
STORED_OBJECT_FIELDS = ('bucket', 'object_name')
|
||||
|
||||
# opc_certificates lookup of a connectors aggregation: the certificate the connector's
|
||||
# security.client_certificate_id points to, only when it is active and of kind client.
|
||||
CERTIFICATE_LOOKUP = {
|
||||
'$lookup': {
|
||||
'from': 'opc_certificates',
|
||||
'let': {'certificate_id': '$security.client_certificate_id'},
|
||||
'pipeline': [
|
||||
{
|
||||
'$match': {
|
||||
'$expr': {'$eq': ['$certificate_id', '$$certificate_id']},
|
||||
'status': 'active',
|
||||
'kind': 'client',
|
||||
}
|
||||
},
|
||||
{
|
||||
'$project': {
|
||||
'_id': 0,
|
||||
'certificate_id': 1,
|
||||
'application_uri': 1,
|
||||
'cert.bucket': 1,
|
||||
'cert.object_name': 1,
|
||||
'private_key.bucket': 1,
|
||||
'private_key.object_name': 1,
|
||||
}
|
||||
},
|
||||
],
|
||||
'as': 'certificates',
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def _is_complete(document: Any, fields: tuple[str, ...]) -> bool:
|
||||
return isinstance(document, dict) and all(
|
||||
document.get(field) not in (None, '') for field in fields
|
||||
)
|
||||
|
||||
|
||||
def _is_usable_certificate(certificate: Any) -> bool:
|
||||
return _is_complete(certificate, CERTIFICATE_FIELDS) and all(
|
||||
_is_complete(certificate.get(name), STORED_OBJECT_FIELDS) for name in CERTIFICATE_OBJECTS
|
||||
)
|
||||
|
||||
|
||||
def client_certificate_id(connector: dict[str, Any]) -> Any:
|
||||
"""The connector's security.client_certificate_id (None when it has none)."""
|
||||
security = connector.get('security')
|
||||
return security.get('client_certificate_id') if isinstance(security, dict) else None
|
||||
|
||||
|
||||
def has_security(connector: dict[str, Any]) -> bool:
|
||||
"""Whether the connector has security.mode and security.policy (D4)."""
|
||||
return _is_complete(connector.get('security'), SECURITY_FIELDS)
|
||||
|
||||
|
||||
def resolve_certificate(connector: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""
|
||||
The client certificate of a connector from CERTIFICATE_LOOKUP, or None (D5).
|
||||
|
||||
None means no certificate: the connection is opened without security. A connector
|
||||
whose client_certificate_id is set but has no usable certificate is most likely a
|
||||
registration error; the caller reports it (see client_certificate_id).
|
||||
|
||||
Returns:
|
||||
dict[str, Any] | None: certificate_id, application_uri, and cert and private_key
|
||||
as {bucket, object_name}, as in the slot
|
||||
"""
|
||||
for certificate in connector.get('certificates') or []:
|
||||
if _is_usable_certificate(certificate):
|
||||
return {
|
||||
**{field: certificate[field] for field in CERTIFICATE_FIELDS},
|
||||
**{
|
||||
name: {field: certificate[name][field] for field in STORED_OBJECT_FIELDS}
|
||||
for name in CERTIFICATE_OBJECTS
|
||||
},
|
||||
}
|
||||
return None
|
||||
@@ -7,63 +7,88 @@ from sientia_do.repository.minio_repository import MinioRepository
|
||||
|
||||
from ingestor.config import _env_bool
|
||||
|
||||
COLLECTION_NAME = 'platform_certificates'
|
||||
# The two objects of a client certificate, and the local file each one is saved as
|
||||
CERTIFICATE_OBJECTS = (('cert', 'cert'), ('private_key', 'key'))
|
||||
|
||||
|
||||
class CertificateNotConfiguredError(RuntimeError):
|
||||
"""No usable platform OPC client certificate is registered (Mongo + MinIO)."""
|
||||
"""The certificate of a server does not name its MinIO objects (opc_certificates)."""
|
||||
|
||||
|
||||
def _objects(certificate: dict) -> tuple[tuple[str, str], ...]:
|
||||
"""(bucket, object_name) of the cert and the private key, in CERTIFICATE_OBJECTS order."""
|
||||
try:
|
||||
return tuple(
|
||||
(str(certificate[field]['bucket']), str(certificate[field]['object_name']))
|
||||
for field, _ in CERTIFICATE_OBJECTS
|
||||
)
|
||||
except (KeyError, TypeError) as e:
|
||||
raise CertificateNotConfiguredError(
|
||||
f'OPC client certificate {certificate!r} does not name the MinIO bucket and '
|
||||
f'object of its cert and private_key (opc_certificates)'
|
||||
) from e
|
||||
|
||||
|
||||
def _certificate_id(certificate: dict) -> str:
|
||||
"""certificate_id as the name of its cache folder: an integer, never a path."""
|
||||
try:
|
||||
return str(int(certificate['certificate_id']))
|
||||
except (KeyError, TypeError, ValueError) as e:
|
||||
raise CertificateNotConfiguredError(
|
||||
f'OPC client certificate {certificate!r} has no integer certificate_id'
|
||||
) from e
|
||||
|
||||
|
||||
class CertProvider:
|
||||
"""Downloads the platform's client cert + private key (Mongo metadata,
|
||||
MinIO bytes) into a writable cache dir — pod root FS is read-only (§12),
|
||||
so `cache_dir` must be an emptyDir/tmpfs mount.
|
||||
"""Downloads the client cert + private key of a connection (the MinIO objects its
|
||||
slot server or write lookup names, SIENTIAPDE-1974) into a writable cache dir — pod
|
||||
root FS is read-only (§12), so `cache_dir` must be an emptyDir/tmpfs mount.
|
||||
|
||||
The certificate is single and shared by every connection (reads, writes,
|
||||
gRPC), so it is downloaded once and cached; `invalidate()` forces a fresh
|
||||
download (the pool calls it when a secured connection fails, e.g. after the
|
||||
certificate was replaced in MinIO). The lock serializes downloads so two
|
||||
connections never rewrite the cache files at the same time."""
|
||||
Each certificate (`certificate_id`) has its own folder (`<cache_dir>/<certificate_id>/`),
|
||||
cache entry and lock, so two connections never rewrite the same files at the same
|
||||
time. A cached certificate is reused while it names the same objects; when the slot
|
||||
names other objects for the same id (the certificate was replaced), it is downloaded
|
||||
again. `invalidate(certificate_id)` forces a fresh download (the pool calls it when a
|
||||
secured connection fails, e.g. after the objects were replaced in MinIO)."""
|
||||
|
||||
def __init__(self, mongo, minio, cache_dir: str, bucket: str, metadata: dict | None = None):
|
||||
self.mongo = mongo
|
||||
def __init__(self, minio, cache_dir: str, metadata: dict | None = None):
|
||||
self.minio = minio
|
||||
self.cache_dir = cache_dir
|
||||
self.bucket = bucket
|
||||
self.metadata = metadata or {}
|
||||
self._lock = asyncio.Lock()
|
||||
self._paths: tuple[str, str] | None = None
|
||||
# ponytail: one Lock per certificate, never cleaned up — the set is small.
|
||||
self._locks: dict[str, asyncio.Lock] = {}
|
||||
self._paths: dict[str, tuple[tuple[tuple[str, str], ...], tuple[str, str]]] = {}
|
||||
|
||||
async def fetch(self) -> tuple[str, str]:
|
||||
async with self._lock:
|
||||
if self._paths is None:
|
||||
self._paths = await self._download_all()
|
||||
return self._paths
|
||||
async def fetch(self, certificate: dict) -> tuple[str, str]:
|
||||
"""Local paths of the cert and private key of `certificate` (slot format)."""
|
||||
certificate_id = _certificate_id(certificate)
|
||||
objects = _objects(certificate)
|
||||
async with self._locks.setdefault(certificate_id, asyncio.Lock()):
|
||||
cached = self._paths.get(certificate_id)
|
||||
if cached is None or cached[0] != objects:
|
||||
self._paths[certificate_id] = (
|
||||
objects,
|
||||
await self._download_all(certificate_id, objects),
|
||||
)
|
||||
return self._paths[certificate_id][1]
|
||||
|
||||
def invalidate(self) -> None:
|
||||
self._paths = None
|
||||
def invalidate(self, certificate_id) -> None:
|
||||
self._paths.pop(str(certificate_id), None)
|
||||
|
||||
async def _download_all(self) -> tuple[str, str]:
|
||||
doc = await self.mongo.find_one(COLLECTION_NAME, {}, self.metadata)
|
||||
try:
|
||||
cert_object = doc['client_cert']['minio_object_name']
|
||||
key_object = doc['private_key']['minio_object_name']
|
||||
except (KeyError, TypeError) as e:
|
||||
raise CertificateNotConfiguredError(
|
||||
f'No OPC client certificate configured: the Mongo collection '
|
||||
f"'{COLLECTION_NAME}' needs a document with client_cert.minio_object_name "
|
||||
f'and private_key.minio_object_name (objects in the MinIO bucket '
|
||||
f"'{self.bucket}'). Register the platform certificate before using "
|
||||
f'secured OPC servers.'
|
||||
) from e
|
||||
Path(self.cache_dir).mkdir(parents=True, exist_ok=True)
|
||||
cert_path = await self._download(cert_object)
|
||||
key_path = await self._download(key_object)
|
||||
async def _download_all(
|
||||
self, certificate_id: str, objects: tuple[tuple[str, str], ...]
|
||||
) -> tuple[str, str]:
|
||||
folder = Path(self.cache_dir) / certificate_id
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
(cert_bucket, cert_object), (key_bucket, key_object) = objects
|
||||
cert_path = await self._download(cert_bucket, cert_object, folder / 'cert')
|
||||
key_path = await self._download(key_bucket, key_object, folder / 'key')
|
||||
return cert_path, key_path
|
||||
|
||||
async def _download(self, object_name: str) -> str:
|
||||
data = await self.minio.download_file(object_name, self.bucket, self.metadata)
|
||||
local_path = Path(self.cache_dir) / Path(object_name).name
|
||||
async def _download(self, bucket: str, object_name: str, stem: Path) -> str:
|
||||
data = await self.minio.download_file(object_name, bucket, self.metadata)
|
||||
# The extension tells asyncua the format (.pem / .der)
|
||||
local_path = stem.with_suffix(Path(object_name).suffix)
|
||||
# Atomic replace: a connection loading the previous file never sees it half-written.
|
||||
tmp_path = local_path.with_name(local_path.name + '.tmp')
|
||||
tmp_path.write_bytes(data)
|
||||
@@ -72,43 +97,43 @@ class CertProvider:
|
||||
|
||||
|
||||
class LazyMinio:
|
||||
"""Builds the MinIO client on first use. MinioRepository's constructor hits the
|
||||
network (bucket check), so building it eagerly would let a MinIO outage at pod
|
||||
start take the whole read path down. Built lazily, only the servers that need a
|
||||
certificate fail — and they are retried on the next loop."""
|
||||
"""Builds one MinIO client per bucket on first use. MinioRepository's constructor
|
||||
hits the network (bucket check), so building it eagerly would let a MinIO outage at
|
||||
pod start take the whole read path down. Built lazily, only the servers that need a
|
||||
certificate fail — and they are retried on the next loop. One client per bucket,
|
||||
because the constructor creates its bucket when missing: it is only ever given a
|
||||
bucket a certificate names."""
|
||||
|
||||
def __init__(self, factory):
|
||||
self._factory = factory
|
||||
self._client = None
|
||||
self._clients: dict[str, MinioRepository] = {}
|
||||
|
||||
async def download_file(self, object_name: str, bucket: str, metadata: dict):
|
||||
if self._client is None:
|
||||
self._client = self._factory()
|
||||
return await self._client.download_file(object_name, bucket, metadata)
|
||||
if bucket not in self._clients:
|
||||
self._clients[bucket] = self._factory(bucket)
|
||||
return await self._clients[bucket].download_file(object_name, bucket, metadata)
|
||||
|
||||
|
||||
def build_cert_provider(
|
||||
mongo, config, logger, notification_handler, metrics_controller, metadata=None
|
||||
config, logger, notification_handler, metrics_controller, metadata=None
|
||||
) -> CertProvider:
|
||||
"""CertProvider shared by the read path (IngestorManager) and the write path
|
||||
(GatewayRuntime): one certificate source — MinIO + Mongo — for every connection."""
|
||||
(GatewayRuntime): one certificate cache for every connection."""
|
||||
|
||||
def minio_factory() -> MinioRepository:
|
||||
def minio_factory(bucket: str) -> MinioRepository:
|
||||
return MinioRepository(
|
||||
endpoint=getenv('MINIO_ENDPOINT', 'localhost:9000'),
|
||||
access_key=getenv('MINIO_ACCESS_KEY', 'minioadmin'),
|
||||
secret_key=getenv('MINIO_SECRET_KEY', 'minioadmin'),
|
||||
secure=_env_bool('MINIO_SECURE', True),
|
||||
bucket=config.cert_bucket,
|
||||
bucket=bucket,
|
||||
logger=logger,
|
||||
notification_handler=notification_handler,
|
||||
metrics_controller=metrics_controller,
|
||||
)
|
||||
|
||||
return CertProvider(
|
||||
mongo=mongo,
|
||||
minio=LazyMinio(minio_factory),
|
||||
cache_dir=config.cert_cache_dir,
|
||||
bucket=config.cert_bucket,
|
||||
metadata=metadata,
|
||||
)
|
||||
|
||||
@@ -9,7 +9,7 @@ from sientia_do.observability.sientia_monitoring import SientiaMonitoring
|
||||
import ingestor.metrics as metrics
|
||||
from ingestor.connection.certs import CertProvider
|
||||
from ingestor.connection.clock import MonotonicClock
|
||||
from ingestor.connection.security import apply_security
|
||||
from ingestor.connection.security import apply_security, is_secured
|
||||
|
||||
|
||||
class OpcConnectionPool(SientiaMonitoring):
|
||||
@@ -65,11 +65,10 @@ class OpcConnectionPool(SientiaMonitoring):
|
||||
"""Warn when a caller asks for a server with other connection settings than
|
||||
the pooled client was opened with. The client is reused as is: reconnecting
|
||||
on every mismatch would let one malformed write command drop the reads.
|
||||
Config changes from the slot evict the client first (ServerReader.shutdown)."""
|
||||
Config changes from the slot (security, certificate) evict the client first:
|
||||
the IngestorManager re-initializes the reader (ServerReader.shutdown)."""
|
||||
opened = self._configs.get(server_id, {})
|
||||
diff = [
|
||||
k for k in ('url', 'server_uri', 'use_security') if opened.get(k) != server_cfg.get(k)
|
||||
]
|
||||
diff = [k for k in ('url', 'security', 'certificate') if opened.get(k) != server_cfg.get(k)]
|
||||
if diff:
|
||||
self.logger.warning(
|
||||
f'OPC server {server_id}: reusing the pooled connection, but {diff} differ '
|
||||
@@ -98,10 +97,10 @@ class OpcConnectionPool(SientiaMonitoring):
|
||||
try:
|
||||
client = await self._connect(server_cfg)
|
||||
except Exception:
|
||||
if server_cfg.get('use_security'):
|
||||
if is_secured(server_cfg):
|
||||
# The cached certificate may be the cause (replaced in MinIO,
|
||||
# rejected by the server): download it again on the next attempt.
|
||||
self.cert_provider.invalidate()
|
||||
self.cert_provider.invalidate(server_cfg['certificate']['certificate_id'])
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_CONNECTIONS_FAILED,
|
||||
tags={'pod_id': self.pod_id, 'server_name': server_id},
|
||||
@@ -123,13 +122,12 @@ class OpcConnectionPool(SientiaMonitoring):
|
||||
pod_uri = self.pod_id.replace('-', ':')
|
||||
client.product_uri = pod_uri
|
||||
client.application_uri = pod_uri
|
||||
server_uri = server_cfg.get('server_uri')
|
||||
if server_uri:
|
||||
# Must match the certificate's SAN URI (KEPServer: BadCertificateUriInvalid).
|
||||
client.application_uri = server_uri
|
||||
if server_cfg.get('use_security'):
|
||||
cert, key = await self.cert_provider.fetch()
|
||||
await apply_security(client, server_cfg.get('server_uri'), cert, key, None)
|
||||
# Security only with a client certificate (SIENTIAPDE-1974 D5); apply_security
|
||||
# sets the certificate's application URI, which must match its SAN URI.
|
||||
if is_secured(server_cfg):
|
||||
certificate = server_cfg['certificate']
|
||||
cert, key = await self.cert_provider.fetch(certificate)
|
||||
await apply_security(client, server_cfg['security'], certificate, cert, key)
|
||||
await client.connect()
|
||||
return client
|
||||
|
||||
|
||||
@@ -1,17 +1,65 @@
|
||||
from asyncua.crypto.security_policies import SecurityPolicyBasic256Sha256
|
||||
from asyncua import ua
|
||||
from asyncua.crypto.security_policies import (
|
||||
SecurityPolicyAes128Sha256RsaOaep,
|
||||
SecurityPolicyAes256Sha256RsaPss,
|
||||
SecurityPolicyBasic128Rsa15,
|
||||
SecurityPolicyBasic256,
|
||||
SecurityPolicyBasic256Sha256,
|
||||
)
|
||||
|
||||
SESSION_TIMEOUT_MS = 600_000 # 10 min; tune per server (Samarco: long sessions degrade)
|
||||
|
||||
# connectors.security.policy / .mode (SIENTIAPDE-1974 D4) to the asyncua classes
|
||||
POLICIES = {
|
||||
'Basic256Sha256': SecurityPolicyBasic256Sha256,
|
||||
'Aes128Sha256RsaOaep': SecurityPolicyAes128Sha256RsaOaep,
|
||||
'Aes256Sha256RsaPss': SecurityPolicyAes256Sha256RsaPss,
|
||||
'Basic256': SecurityPolicyBasic256,
|
||||
'Basic128Rsa15': SecurityPolicyBasic128Rsa15,
|
||||
}
|
||||
MODES = {
|
||||
'Sign': ua.MessageSecurityMode.Sign,
|
||||
'SignAndEncrypt': ua.MessageSecurityMode.SignAndEncrypt,
|
||||
}
|
||||
|
||||
async def apply_security(client, server_uri, cert_path, private_key_path, server_cert_path):
|
||||
"""Apply Basic256Sha256 + certificate auth (1879 §1) and set channel > session
|
||||
lifetime (1879 §3: secure_channel_timeout = session_timeout * 1.25)."""
|
||||
client.application_uri = server_uri
|
||||
|
||||
class UnsupportedSecurityError(ValueError):
|
||||
"""The server's security mode or policy has no asyncua counterpart."""
|
||||
|
||||
|
||||
def is_secured(server_cfg: dict) -> bool:
|
||||
"""Whether a connection applies security: only with a client certificate (D5), and
|
||||
not when its mode is 'None'. Without one, it connects without security, and a server
|
||||
that requires a certificate refuses the connection."""
|
||||
security = server_cfg.get('security') or {}
|
||||
return server_cfg.get('certificate') is not None and security.get('mode') != 'None'
|
||||
|
||||
|
||||
async def apply_security(client, security, certificate, cert_path, private_key_path):
|
||||
"""Apply the server's mode + policy with the connector's client certificate, and set
|
||||
channel > session lifetime (1879 §3: secure_channel_timeout = session_timeout * 1.25).
|
||||
|
||||
The application URI is the certificate's (its SAN URI): the server rejects a
|
||||
different one (KEPServer: BadCertificateUriInvalid).
|
||||
|
||||
Raises:
|
||||
UnsupportedSecurityError: When the mode or policy is unknown.
|
||||
"""
|
||||
policy = POLICIES.get(security.get('policy'))
|
||||
mode = MODES.get(security.get('mode'))
|
||||
if policy is None or mode is None:
|
||||
raise UnsupportedSecurityError(
|
||||
f'Unsupported OPC security mode={security.get("mode")!r} '
|
||||
f'policy={security.get("policy")!r} (modes: {", ".join(MODES)}; '
|
||||
f'policies: {", ".join(POLICIES)})'
|
||||
)
|
||||
client.application_uri = certificate['application_uri']
|
||||
await client.set_security(
|
||||
SecurityPolicyBasic256Sha256,
|
||||
policy,
|
||||
certificate=cert_path,
|
||||
private_key=private_key_path,
|
||||
server_certificate=server_cert_path,
|
||||
server_certificate=None,
|
||||
mode=mode,
|
||||
)
|
||||
client.session_timeout = SESSION_TIMEOUT_MS
|
||||
client.secure_channel_timeout = int(SESSION_TIMEOUT_MS * 1.25)
|
||||
|
||||
@@ -1,60 +1,61 @@
|
||||
"""Connection settings of the OPC server a write command targets (SIENTIAPDE-1975 GW1).
|
||||
|
||||
The Laborious only knows the `server_id` of a write tag (pipeline `write_tags`); the
|
||||
connection (url, server URI, security) belongs to the gateway, which owns every OPC
|
||||
session. It is resolved with the same rule the orchestrator uses to build the read slots
|
||||
(SIENTIAPDE-1974): the tag database `connectors` first, the legacy `opc-servers` otherwise.
|
||||
connection (url, security, certificate) belongs to the gateway, which owns every OPC
|
||||
session. A server already in the pool is written through that connection. Otherwise it
|
||||
is resolved from the tag database (`connectors` → `opc_certificates`) with the rule the
|
||||
Orchestrator uses to build the read slots (SIENTIAPDE-1974 D1), in the slot format.
|
||||
"""
|
||||
|
||||
from typing import Any
|
||||
|
||||
from sientia_do.observability.logger import Logger
|
||||
|
||||
from ingestor.connection.certificates import (
|
||||
CERTIFICATE_LOOKUP,
|
||||
client_certificate_id,
|
||||
has_security,
|
||||
resolve_certificate,
|
||||
)
|
||||
from ingestor.connection.clock import MonotonicClock
|
||||
from ingestor.connection.pool import OpcConnectionPool
|
||||
|
||||
CONNECTORS = 'connectors'
|
||||
OPC_SERVERS = 'opc-servers'
|
||||
|
||||
|
||||
def _id_forms(server_id: str) -> list[Any]:
|
||||
"""`server_id` as stored: the tag database keeps ints, opc-servers strings or ints."""
|
||||
forms: list[Any] = [server_id]
|
||||
if server_id.isdigit():
|
||||
forms.append(int(server_id))
|
||||
return forms
|
||||
|
||||
|
||||
def _connector_cfg(connector: dict[str, Any]) -> dict[str, Any] | None:
|
||||
if not connector.get('url'):
|
||||
return None
|
||||
return {
|
||||
'url': connector['url'],
|
||||
'server_uri': connector.get('server_uri'),
|
||||
'use_security': bool(connector.get('use_security', False)),
|
||||
}
|
||||
|
||||
|
||||
def _opc_server_cfg(server: dict[str, Any]) -> dict[str, Any] | None:
|
||||
if not server.get('url'):
|
||||
return None
|
||||
return {
|
||||
'url': server['url'],
|
||||
'server_uri': server.get('uri'),
|
||||
# The certificate comes from MinIO now: a cert_path only says the server uses
|
||||
# security, as in the orchestrator's slot (SIENTIAPDE-1974 RF4).
|
||||
'use_security': bool(server.get('cert_path')),
|
||||
}
|
||||
def _connector_pipeline(server_id: str) -> list[dict[str, Any]]:
|
||||
"""The active OPC UA connector `server_id` (connector_id is its slug), with its
|
||||
certificate. Only the fields used are projected."""
|
||||
return [
|
||||
{'$match': {'connector_id': server_id, 'type': 'opc_ua', 'status': 'Active'}},
|
||||
{'$limit': 1},
|
||||
CERTIFICATE_LOOKUP,
|
||||
{
|
||||
'$project': {
|
||||
'_id': 0,
|
||||
'connector_id': 1,
|
||||
'url': 1,
|
||||
'subscription_period_ms': 1,
|
||||
'security.mode': 1,
|
||||
'security.policy': 1,
|
||||
'security.client_certificate_id': 1,
|
||||
'certificates': 1,
|
||||
}
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
class ServerConfigResolver:
|
||||
"""Resolve and cache the connection settings of an OPC server by `server_id`.
|
||||
|
||||
Order: settings carried by the command itself (an explicit override; the C2
|
||||
command of the Laborious carries none), the live pooled connection (what the read
|
||||
path opened, so writes never fight the reads over settings), then MongoDB. Lookups
|
||||
are cached for `ttl_s`; a server that is not found, for `missing_ttl_s`, so a
|
||||
pipeline pointing at a removed server does not query MongoDB on every command."""
|
||||
Order: the live pooled connection (what the read path opened, so writes never fight
|
||||
the reads over settings), then MongoDB. Lookups are cached for `ttl_s`; a server
|
||||
that is not found, for `missing_ttl_s`, so a pipeline pointing at a removed server
|
||||
does not query MongoDB on every command.
|
||||
|
||||
# ponytail: the MongoDB fallback goes away once write commands are routed to the pod
|
||||
# that owns the connection (SIENTIAPDE-1974 D1, new card).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -74,19 +75,13 @@ class ServerConfigResolver:
|
||||
self.clock = MonotonicClock()
|
||||
self._cache: dict[str, tuple[dict[str, Any] | None, float]] = {}
|
||||
|
||||
async def resolve(self, server_id: str, cmd: dict) -> dict[str, Any] | None:
|
||||
async def resolve(self, server_id: str) -> dict[str, Any] | None:
|
||||
"""Connection settings for `server_id`, or None when no source knows it.
|
||||
|
||||
Raises:
|
||||
Exception: When MongoDB cannot be read (the caller reports the write as
|
||||
failed).
|
||||
"""
|
||||
if cmd.get('url'):
|
||||
return {
|
||||
'url': cmd['url'],
|
||||
'server_uri': cmd.get('server_uri'),
|
||||
'use_security': cmd.get('use_security', False),
|
||||
}
|
||||
server_id = str(server_id)
|
||||
pooled = self.pool.config_for(server_id)
|
||||
if pooled is not None:
|
||||
@@ -102,19 +97,30 @@ class ServerConfigResolver:
|
||||
return dict(cfg) if cfg is not None else None
|
||||
|
||||
async def _lookup(self, server_id: str) -> dict[str, Any] | None:
|
||||
ids = {'$in': _id_forms(server_id)}
|
||||
connector = await self.mongo.find_one(
|
||||
CONNECTORS,
|
||||
{'connector_id': ids, 'type': 'opc_ua', 'status': 'Active'},
|
||||
self.metadata,
|
||||
connectors = await self.mongo.aggregate(
|
||||
CONNECTORS, _connector_pipeline(server_id), self.metadata
|
||||
)
|
||||
cfg = _connector_cfg(connector) if connector else None
|
||||
if cfg is not None:
|
||||
return cfg
|
||||
server = await self.mongo.find_one(OPC_SERVERS, {'id': ids}, self.metadata)
|
||||
cfg = _opc_server_cfg(server) if server else None
|
||||
if cfg is None:
|
||||
connector = connectors[0] if connectors else None
|
||||
if connector is None or not connector.get('url') or not has_security(connector):
|
||||
self.logger.warning(
|
||||
f'OPC server {server_id} not found in {CONNECTORS} nor {OPC_SERVERS}'
|
||||
f'OPC server {server_id} not found, not active or incomplete (url, '
|
||||
f'security.mode, security.policy) in {CONNECTORS}'
|
||||
)
|
||||
return cfg
|
||||
return None
|
||||
certificate = resolve_certificate(connector)
|
||||
if certificate is None and client_certificate_id(connector) is not None:
|
||||
self.logger.warning(
|
||||
f'OPC server {server_id}: client_certificate_id '
|
||||
f'{client_certificate_id(connector)} has no active client certificate in '
|
||||
f'opc_certificates; connecting without security'
|
||||
)
|
||||
return {
|
||||
'server_id': server_id,
|
||||
'url': connector['url'],
|
||||
'subscription_period_ms': connector.get('subscription_period_ms'),
|
||||
'security': {
|
||||
'mode': connector['security']['mode'],
|
||||
'policy': connector['security']['policy'],
|
||||
},
|
||||
'certificate': certificate,
|
||||
}
|
||||
|
||||
@@ -83,7 +83,7 @@ class WriteEngine(SientiaMonitoring):
|
||||
self.pool = pool
|
||||
self.sink = sink
|
||||
# Resolves the server connection by server_id (SIENTIAPDE-1975 GW1). Without it,
|
||||
# only the connection carried by the command itself is used.
|
||||
# no server is known and every write fails as server_not_found.
|
||||
self.servers = servers
|
||||
self.write_timestamp = write_timestamp
|
||||
self.write_quality = write_quality
|
||||
@@ -146,18 +146,12 @@ class WriteEngine(SientiaMonitoring):
|
||||
self._last_write_at = now
|
||||
|
||||
async def _server_cfg(self, cmd: dict) -> dict | None:
|
||||
"""Connection settings of the target server: resolved by server_id (connection
|
||||
carried by the command, live pooled connection, connectors, opc-servers), or only
|
||||
the ones carried by the command when no resolver is wired."""
|
||||
if self.servers is not None:
|
||||
return await self.servers.resolve(str(cmd['server_id']), cmd)
|
||||
if not cmd.get('url'):
|
||||
"""Connection settings of the target server, resolved by server_id (live pooled
|
||||
connection, then connectors → opc_certificates). The command carries only the
|
||||
server_id (C2)."""
|
||||
if self.servers is None:
|
||||
return None
|
||||
return {
|
||||
'url': cmd.get('url'),
|
||||
'use_security': cmd.get('use_security', False),
|
||||
'server_uri': cmd.get('server_uri'),
|
||||
}
|
||||
return await self.servers.resolve(str(cmd['server_id']))
|
||||
|
||||
def _variant_type(self, data_type: str) -> ua.VariantType:
|
||||
# direct index: unknown dtypes already became ConversionError in to_typed_value
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
object graph and wires it into the app's lifecycle.
|
||||
|
||||
Reuses the existing ingestor's logger/notification_handler/metrics_controller and
|
||||
its already-connected DataManager. The OpcConnectionPool (and its CertProvider,
|
||||
MinIO + Mongo) belongs to the read path in IngestorManager (SIENTIAPDE-1974):
|
||||
its already-connected DataManager. The OpcConnectionPool (and its CertProvider, which
|
||||
downloads each server's client certificate from MinIO) belongs to the read path in IngestorManager (SIENTIAPDE-1974):
|
||||
reads start in prepare_ingestor, before this runtime exists, and must keep running
|
||||
if anything here fails. The runtime only borrows the pool for writes and gRPC.
|
||||
|
||||
@@ -63,8 +63,9 @@ class GatewayRuntime(SientiaMonitoring):
|
||||
self.sink = DataSink(data_manager, postgres)
|
||||
|
||||
# Write commands carry only the server_id (SIENTIAPDE-1975): the connection is
|
||||
# resolved here, from the live pooled connection or the connectors/opc-servers
|
||||
# documents, with the same rule the orchestrator uses for the read slots.
|
||||
# resolved here, from the live pooled connection or the connectors →
|
||||
# opc_certificates documents, with the rule the orchestrator uses for the read
|
||||
# slots (SIENTIAPDE-1974 D1).
|
||||
self.servers = ServerConfigResolver(
|
||||
self.pool, data_manager.mongo_repository, logger, metadata=self.metadata
|
||||
)
|
||||
|
||||
@@ -52,14 +52,16 @@ class DataSink:
|
||||
async def publish_reading(self, tag: str, value, timestamp, names: list[dict]) -> None:
|
||||
"""Publish one MongoDB document per name entry and record its tag value metric.
|
||||
|
||||
`model_id` is only written when the name carries one: names converted from a
|
||||
slot v1 have none, and their documents must stay identical to the legacy ones."""
|
||||
Every name of the slot v2 carries collection, tag_name and model_id (parse_slot
|
||||
checks it)."""
|
||||
for name in names:
|
||||
doc = {'tag': tag, 'name': name['tag_name']}
|
||||
if 'model_id' in name:
|
||||
doc['model_id'] = name['model_id']
|
||||
doc['timestamp'] = timestamp
|
||||
doc['value'] = value
|
||||
doc = {
|
||||
'tag': tag,
|
||||
'name': name['tag_name'],
|
||||
'model_id': name['model_id'],
|
||||
'timestamp': timestamp,
|
||||
'value': value,
|
||||
}
|
||||
await self.data_manager.publish(name['collection'], doc)
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_TAG_VALUE,
|
||||
@@ -68,7 +70,7 @@ class DataSink:
|
||||
tags={
|
||||
'pod_id': self.data_manager.pod_id,
|
||||
'tag_name': name['tag_name'],
|
||||
'model_id': name.get('model_id', ''),
|
||||
'model_id': name['model_id'],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@@ -1,28 +1,27 @@
|
||||
"""Parse the slot written by the Orchestrator to Redis into the slot v2 structure (contract C1).
|
||||
"""Parse the slot v2 the Orchestrator writes to Redis (contract C1, SIENTIAPDE-1974).
|
||||
|
||||
Both shapes are accepted so the gateway and the Orchestrator can be deployed in
|
||||
any order (SIENTIAPDE-1974):
|
||||
|
||||
- v2: each tag carries `names` (one entry per collection / tag name / model);
|
||||
- v1 (legacy): each tag carries `tag_name` + `topics`, and the server carries the
|
||||
file-based `cert_path`. v1 is converted to v2: one `names` entry per topic, and
|
||||
`use_security` is true when the server had a certificate path (the certificate
|
||||
itself now always comes from MinIO via CertProvider, never from that path).
|
||||
Each server carries its connection (`url`, `subscription_period_ms`), its OPC security
|
||||
(`security`: mode + policy) and its client certificate (`certificate`: certificate_id,
|
||||
application_uri and the MinIO objects of the cert and the private key; None = connect
|
||||
without security, D5). Each tag carries `names`: one entry per collection / tag name /
|
||||
model the reading is written to.
|
||||
"""
|
||||
|
||||
SERVER_FIELDS = ('server_id', 'url', 'server_uri', 'use_security', 'subscription_period_ms')
|
||||
SERVER_FIELDS = ('server_id', 'url', 'subscription_period_ms', 'security', 'certificate')
|
||||
# What a connection/subscription needs (the pool keys clients by server_id itself).
|
||||
CONNECTION_FIELDS = tuple(f for f in SERVER_FIELDS if f != 'server_id')
|
||||
NAME_FIELDS = ('collection', 'tag_name', 'model_id')
|
||||
|
||||
|
||||
def _parse_names(address: str, tag: dict) -> list[dict]:
|
||||
names = tag.get('names')
|
||||
if names:
|
||||
return names
|
||||
topics = tag.get('topics')
|
||||
if topics and tag.get('tag_name'):
|
||||
return [{'collection': topic, 'tag_name': tag['tag_name']} for topic in topics]
|
||||
raise ValueError(f'tag {address!r} has no names')
|
||||
if not names:
|
||||
raise ValueError(f'tag {address!r} has no names')
|
||||
for name in names:
|
||||
missing = [field for field in NAME_FIELDS if field not in name]
|
||||
if missing:
|
||||
raise ValueError(f'tag {address!r} has a name without {missing}')
|
||||
return names
|
||||
|
||||
|
||||
def _parse_tag(address: str, tag: dict) -> dict:
|
||||
@@ -34,14 +33,19 @@ def _parse_tag(address: str, tag: dict) -> dict:
|
||||
|
||||
|
||||
def _parse_server(name: str, server: dict) -> dict:
|
||||
parsed = {field: server[field] for field in SERVER_FIELDS if field in server}
|
||||
if not isinstance(server.get('security'), dict):
|
||||
raise ValueError(f'server {name!r} has no security')
|
||||
parsed = {field: server.get(field) for field in SERVER_FIELDS}
|
||||
parsed['name'] = server.get('name', name)
|
||||
if 'use_security' not in parsed:
|
||||
parsed['use_security'] = bool(server.get('cert_path'))
|
||||
parsed['tags'] = {addr: _parse_tag(addr, tag) for addr, tag in server['tags'].items()}
|
||||
return parsed
|
||||
|
||||
|
||||
def parse_slot(raw: dict) -> dict:
|
||||
"""Normalize a raw slot (v1 or v2) into `{server_name: {..., tags: {addr: tag}}}`."""
|
||||
"""Normalize a raw slot v2 into `{server_name: {..., tags: {addr: tag}}}`.
|
||||
|
||||
Raises:
|
||||
KeyError | TypeError | ValueError: When the slot is not a valid slot v2 (e.g. a
|
||||
slot v1 with `topics`).
|
||||
"""
|
||||
return {name: _parse_server(name, server) for name, server in raw.items()}
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import traceback
|
||||
from copy import deepcopy
|
||||
from os import getenv
|
||||
|
||||
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
|
||||
from sientia_do.notifications.models import NotificationLevel
|
||||
@@ -15,16 +14,9 @@ from ingestor.connection.pool import OpcConnectionPool
|
||||
from ingestor.io.data_sink import DataSink
|
||||
from ingestor.io.slots import parse_slot
|
||||
from ingestor.managers.data_manager import DataManager
|
||||
from ingestor.managers.opc_manager import OpcManager
|
||||
from ingestor.managers.resource_manager import ResourceManager
|
||||
from ingestor.managers.server_reader import ServerReader
|
||||
|
||||
# READ_ENGINE (SIENTIAPDE-1974): 'v2' = ServerReader/SubscriptionEngine over the shared
|
||||
# pool, certificate from MinIO (default). 'legacy' = plan B: the untouched rc3
|
||||
# OpcManager, slot v1 only, certificate files from the Gitea clone (cert_path). Kept
|
||||
# for validation until the end of 1974; remove once unused and the tests pass without it.
|
||||
READ_ENGINES = ('v2', 'legacy')
|
||||
|
||||
|
||||
class IngestorManager(SientiaMonitoring):
|
||||
"""
|
||||
@@ -85,7 +77,6 @@ class IngestorManager(SientiaMonitoring):
|
||||
notification_handler: NotificationHandler,
|
||||
metrics_controller: MetricsController,
|
||||
gateway_config: GatewayConfig | None = None,
|
||||
read_engine: str | None = None,
|
||||
):
|
||||
redis_host: str = redis_data['host']
|
||||
redis_port: int = int(redis_data['port'])
|
||||
@@ -108,12 +99,12 @@ class IngestorManager(SientiaMonitoring):
|
||||
metrics_controller=metrics_controller,
|
||||
)
|
||||
self.opc_managers: dict = {}
|
||||
# Read path (SIENTIAPDE-1974): pooled connections with the certificate from
|
||||
# MinIO. Nothing here touches the network at construction (MinIO is lazy), so
|
||||
# a MinIO/Postgres/NATS outage can't stop the reads before they start.
|
||||
# Read path (SIENTIAPDE-1974): pooled connections with each server's client
|
||||
# certificate (named in the slot) from MinIO. Nothing here touches the network at
|
||||
# construction (MinIO is lazy), so a MinIO/Postgres/NATS outage can't stop the
|
||||
# reads before they start.
|
||||
config = gateway_config or GatewayConfig.certs_from_env()
|
||||
self.cert_provider = build_cert_provider(
|
||||
mongo=self.data_manager.mongo_repository,
|
||||
config=config,
|
||||
logger=logger,
|
||||
notification_handler=notification_handler,
|
||||
@@ -124,10 +115,6 @@ class IngestorManager(SientiaMonitoring):
|
||||
self.cert_provider, logger, notification_handler, metrics_controller
|
||||
)
|
||||
self.sink = DataSink(self.data_manager, None)
|
||||
self._server_cert_warned: set[str] = set()
|
||||
self.read_engine = self._resolve_read_engine(
|
||||
read_engine or getenv('READ_ENGINE') or 'v2', logger
|
||||
)
|
||||
self.resource_manager = ResourceManager(
|
||||
host=redis_host,
|
||||
port=redis_port,
|
||||
@@ -147,32 +134,7 @@ class IngestorManager(SientiaMonitoring):
|
||||
|
||||
self.metadata = metadata
|
||||
|
||||
@staticmethod
|
||||
def _resolve_read_engine(value: str, logger: Logger) -> str:
|
||||
engine = value.strip().lower()
|
||||
if engine not in READ_ENGINES:
|
||||
logger.warning(f'Invalid READ_ENGINE {value!r}, using v2 ({"|".join(READ_ENGINES)})')
|
||||
engine = 'v2'
|
||||
logger.info(f'OPC read engine: {engine}')
|
||||
return engine
|
||||
|
||||
def _build_reader(self, server_config: dict) -> ServerReader | OpcManager:
|
||||
if self.read_engine == 'legacy':
|
||||
# Plan B: the rc3 OpcManager, unchanged (own connection, cert files).
|
||||
return OpcManager(
|
||||
name=server_config['name'],
|
||||
url=server_config['url'],
|
||||
subscription_period_ms=server_config['subscription_period_ms'],
|
||||
data_manager=self.data_manager,
|
||||
logger=self.logger,
|
||||
server_uri=server_config['server_uri'],
|
||||
notification_handler=self.notification_handler,
|
||||
metadata=self.metadata,
|
||||
cert_path=server_config.get('cert_path'),
|
||||
private_key_path=server_config.get('private_key_path'),
|
||||
server_cert_path=server_config.get('server_cert_path'),
|
||||
metrics_controller=self.metrics_controller,
|
||||
)
|
||||
def _build_reader(self, server_config: dict) -> ServerReader:
|
||||
return ServerReader(
|
||||
name=server_config['name'],
|
||||
server_config=server_config,
|
||||
@@ -184,24 +146,21 @@ class IngestorManager(SientiaMonitoring):
|
||||
metadata=self.metadata,
|
||||
)
|
||||
|
||||
async def initialize_opc_from_config(
|
||||
self, server_config: dict
|
||||
) -> ServerReader | OpcManager | None:
|
||||
async def initialize_opc_from_config(self, server_config: dict) -> ServerReader | None:
|
||||
"""
|
||||
Initializes the reader of one OPC UA server from its slot configuration.
|
||||
|
||||
With READ_ENGINE=v2 (default) creates a ServerReader and opens its pooled
|
||||
connection; with `use_security`, the certificate comes from MinIO. With
|
||||
READ_ENGINE=legacy (plan B) creates the rc3 OpcManager, which connects on its
|
||||
own with the certificate files of the slot v1 (`cert_path`).
|
||||
Creates a ServerReader and opens its pooled connection. With a `certificate`, the
|
||||
connection uses the slot's security mode and policy, with the certificate
|
||||
downloaded from MinIO; without one, it connects without security (D5).
|
||||
|
||||
Args:
|
||||
server_config (dict): The server configuration of the slot, without the
|
||||
'tags' key. v2: 'name', 'server_id', 'url', 'server_uri',
|
||||
'use_security', 'subscription_period_ms'. legacy: the slot v1 fields.
|
||||
server_config (dict): The server configuration of the slot v2, without the
|
||||
'tags' key: 'name', 'server_id', 'url', 'subscription_period_ms',
|
||||
'security' and 'certificate'.
|
||||
|
||||
Returns:
|
||||
ServerReader | OpcManager | None: The connected reader, or None if the
|
||||
ServerReader | None: The connected reader, or None if the
|
||||
connection fails (logged and notified, so the other servers keep working).
|
||||
"""
|
||||
|
||||
@@ -254,11 +213,7 @@ class IngestorManager(SientiaMonitoring):
|
||||
the shared connection stays for writes/gRPC until the idle sweep.
|
||||
"""
|
||||
if server in self.opc_managers:
|
||||
if self.read_engine == 'legacy':
|
||||
# OpcManager owns its connection: shutdown always disconnects it.
|
||||
await self.opc_managers[server].shutdown()
|
||||
else:
|
||||
await self.opc_managers[server].shutdown(evict=evict)
|
||||
await self.opc_managers[server].shutdown(evict=evict)
|
||||
del self.opc_managers[server]
|
||||
for slot, _config in self.managed_tags.items():
|
||||
self.managed_tags[slot].pop(server, None)
|
||||
@@ -567,14 +522,11 @@ class IngestorManager(SientiaMonitoring):
|
||||
if slot in reader.subscriptions:
|
||||
await self._unsubscribe_reader(server, reader, slot)
|
||||
|
||||
async def _unsubscribe_reader(
|
||||
self, server: str, reader: ServerReader | OpcManager, slot: str
|
||||
) -> None:
|
||||
async def _unsubscribe_reader(self, server: str, reader: ServerReader, slot: str) -> None:
|
||||
"""
|
||||
Unsubscribe one reader from a slot without letting a failure stop the main loop.
|
||||
|
||||
A failure (e.g. the legacy OpcManager, whose delete() raises when the server
|
||||
session is gone) increments OPC_SUBSCRIPTION_ERRORS, sends
|
||||
A failure (e.g. delete() raising when the server session is gone) increments OPC_SUBSCRIPTION_ERRORS, sends
|
||||
OPC_UNSUBSCRIBE_ERROR_<slot>:<server> and is logged. The slot is dropped from the
|
||||
reader's books, as the v2 engine already does, so it can be subscribed again.
|
||||
|
||||
@@ -584,7 +536,7 @@ class IngestorManager(SientiaMonitoring):
|
||||
|
||||
Args:
|
||||
server (str): The reader's key in opc_managers
|
||||
reader (ServerReader | OpcManager): The OPC reader
|
||||
reader (ServerReader): The OPC reader
|
||||
slot (str): The slot to unsubscribe
|
||||
"""
|
||||
try:
|
||||
@@ -664,22 +616,17 @@ class IngestorManager(SientiaMonitoring):
|
||||
|
||||
def parse_slot_config(self, slot: str, raw: dict) -> dict | None:
|
||||
"""
|
||||
Normalizes the raw slot read from Redis to the slot v2 structure.
|
||||
Normalizes the raw slot read from Redis to the slot v2 structure (contract C1).
|
||||
|
||||
Accepts slot v1 (`topics`) and v2 (`names`), so the gateway and the
|
||||
Orchestrator can be deployed in any order (SIENTIAPDE-1974). With
|
||||
READ_ENGINE=legacy the raw slot v1 is kept as is (what OpcManager reads), and
|
||||
a slot v2 is rejected.
|
||||
Only the slot v2 is accepted (SIENTIAPDE-1974): the gateway and the Orchestrator
|
||||
are deployed together, and a slot v1 (`topics`, no `security`) is rejected.
|
||||
|
||||
Returns:
|
||||
dict | None: The parsed slot, or None when the config is malformed
|
||||
(logged and notified).
|
||||
"""
|
||||
try:
|
||||
if self.read_engine == 'legacy':
|
||||
self._validate_slot_v1(raw)
|
||||
return raw
|
||||
parsed = parse_slot(raw)
|
||||
return parse_slot(raw)
|
||||
except (AttributeError, KeyError, TypeError, ValueError) as e:
|
||||
self.send_notification(
|
||||
metadata=self.metadata,
|
||||
@@ -690,32 +637,6 @@ class IngestorManager(SientiaMonitoring):
|
||||
)
|
||||
self.logger.error(f'Slot {slot} has an invalid configuration: {e!r}')
|
||||
return None
|
||||
self._warn_server_cert_path(raw)
|
||||
return parsed
|
||||
|
||||
@staticmethod
|
||||
def _validate_slot_v1(raw: dict) -> None:
|
||||
"""READ_ENGINE=legacy: OpcManager only understands slot v1 (`tag_name` +
|
||||
`topics` per tag); a v2 slot (`names`) would fail in its datachange callback."""
|
||||
for name, server in raw.items():
|
||||
for address, tag in server['tags'].items():
|
||||
if 'topics' not in tag or 'tag_name' not in tag or 'frequency' not in tag:
|
||||
raise ValueError(
|
||||
f'READ_ENGINE=legacy needs a slot v1 (tag_name + topics); '
|
||||
f'server {name!r}, tag {address!r} is not'
|
||||
)
|
||||
|
||||
def _warn_server_cert_path(self, raw: dict) -> None:
|
||||
"""Logs once per server when a slot still pins a server certificate
|
||||
(`server_cert_path`): connections now come from the pool with the platform
|
||||
certificate from MinIO, and that file-based pin is no longer applied."""
|
||||
for name, server in raw.items():
|
||||
if server.get('server_cert_path') and name not in self._server_cert_warned:
|
||||
self._server_cert_warned.add(name)
|
||||
self.logger.warning(
|
||||
f'OPC server {name}: server_cert_path is set in the slot but is no '
|
||||
f'longer applied (server certificate validation is disabled)'
|
||||
)
|
||||
|
||||
async def drop_slot_leases(self, ids: list[str]) -> None:
|
||||
"""
|
||||
|
||||
@@ -1,588 +0,0 @@
|
||||
import asyncio
|
||||
import json
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
|
||||
from asyncua import Client
|
||||
from asyncua.crypto.security_policies import SecurityPolicyBasic256
|
||||
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
|
||||
from sientia_do.notifications.models import NotificationLevel
|
||||
from sientia_do.observability.logger import Logger
|
||||
from sientia_do.observability.metrics_controller import MetricsController
|
||||
from sientia_do.observability.sientia_monitoring import SientiaMonitoring
|
||||
from sientia_do.temporal.constants import DATETIME_FORMAT_WITH_TZ, OPC_TIMEZONE
|
||||
|
||||
import ingestor.metrics as metrics
|
||||
from ingestor.managers.data_manager import DataManager
|
||||
|
||||
|
||||
class OpcManager(SientiaMonitoring):
|
||||
"""
|
||||
Manages OPC UA server connections and tag subscriptions.
|
||||
|
||||
The OpcManager is responsible for:
|
||||
- Establishing and maintaining secure connections to OPC UA servers
|
||||
- Managing tag subscriptions and data collection
|
||||
- Handling server reconnection and error recovery
|
||||
- Processing OPC data and forwarding it to the data manager
|
||||
- Monitoring connection health and performance metrics
|
||||
|
||||
The manager supports both secure and unsecured connections, with optional
|
||||
certificate-based authentication for enhanced security.
|
||||
|
||||
Args:
|
||||
name (str): Unique identifier for the OPC server
|
||||
url (str): OPC UA server endpoint URL
|
||||
data_manager (DataManager): Manager for data persistence and export
|
||||
logger (Logger): Logger instance for application logging
|
||||
server_uri (str): OPC UA server application URI
|
||||
notification_handler (NotificationHandler): Handler for sending notifications
|
||||
metadata (dict): Application metadata for notifications and tracking
|
||||
cert_path (str, optional): Path to client certificate file for secure connections
|
||||
private_key_path (str, optional): Path to client private key file
|
||||
server_cert_path (str, optional): Path to server certificate file for validation
|
||||
|
||||
Attributes:
|
||||
url (str): OPC UA server endpoint URL
|
||||
name (str): Unique identifier for the OPC server
|
||||
server_uri (str): OPC UA server application URI
|
||||
data_queue (dict): Queue for buffering OPC data before processing
|
||||
non_receive_count (int): Counter for cycles without data reception
|
||||
client (Client): OPC UA client instance
|
||||
cert_path (str): Path to client certificate file
|
||||
private_key_path (str): Path to client private key file
|
||||
server_cert_path (str): Path to server certificate file
|
||||
nodes (dict): Dictionary of OPC node references
|
||||
subscriptions (dict): Active OPC subscriptions
|
||||
data_manager (DataManager): Manager for data persistence and export
|
||||
metadata (dict): Application metadata
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
name: str,
|
||||
url: str,
|
||||
subscription_period_ms: int,
|
||||
data_manager: DataManager,
|
||||
logger: Logger,
|
||||
server_uri: str,
|
||||
notification_handler: NotificationHandler,
|
||||
metrics_controller: MetricsController,
|
||||
metadata: dict,
|
||||
cert_path: str | None = None,
|
||||
private_key_path: str | None = None,
|
||||
server_cert_path: str | None = None,
|
||||
):
|
||||
self.url = url
|
||||
self.name = name
|
||||
self.server_uri = server_uri
|
||||
self.data_queue: dict = {}
|
||||
self.non_receive_count = 0
|
||||
self.client: Client | None = None
|
||||
self.subscription_period_ms = subscription_period_ms
|
||||
self.cert_path = cert_path
|
||||
self.private_key_path = private_key_path
|
||||
self.server_cert_path = server_cert_path
|
||||
self.nodes: dict = {}
|
||||
self.subscriptions: dict = {}
|
||||
self.data_manager = data_manager
|
||||
self.metadata = metadata
|
||||
|
||||
SientiaMonitoring.__init__(
|
||||
self,
|
||||
logger=logger,
|
||||
notification_handler=notification_handler,
|
||||
metrics_controller=metrics_controller,
|
||||
)
|
||||
|
||||
metrics.OPC_CONNECTION_STATUS.labels(
|
||||
pod_id=self.pod_id, server_name=self.name, server_url=self.url
|
||||
).set(0)
|
||||
metrics.OPC_TAGS_SUBSCRIBED.labels(pod_id=self.pod_id, server_name=self.name).set(0)
|
||||
|
||||
def __str__(self):
|
||||
"""
|
||||
String representation of the OPC Manager.
|
||||
|
||||
Returns:
|
||||
str: Human-readable representation showing server details and current state.
|
||||
"""
|
||||
return (
|
||||
f'OpcManager(name={self.name}, url={self.url}, server_uri={self.server_uri})\n'
|
||||
f'nodes={self.nodes}, subscriptions={self.subscriptions}'
|
||||
)
|
||||
|
||||
async def shutdown(self):
|
||||
"""
|
||||
Comprehensive cleanup method for graceful shutdown.
|
||||
|
||||
This method ensures proper cleanup of all OPC UA resources:
|
||||
- Closes active subscriptions
|
||||
- Disconnects from the OPC server
|
||||
- Releases allocated resources
|
||||
|
||||
Should be called before the application terminates to prevent resource leaks
|
||||
and ensure clean disconnection from OPC servers.
|
||||
"""
|
||||
try:
|
||||
await self.disconnect()
|
||||
except Exception as e:
|
||||
self.logger.error(f'Error during cleanup: {e}')
|
||||
|
||||
async def set_security(self):
|
||||
"""
|
||||
Configures the security settings for the OPC UA client.
|
||||
|
||||
This method sets up the security policy, certificates, and timeouts
|
||||
required for establishing a secure connection with the OPC UA server.
|
||||
It implements Basic256 security policy with certificate-based authentication.
|
||||
|
||||
Raises:
|
||||
ValueError: If either the certificate path or private key path is not provided.
|
||||
|
||||
Security Settings:
|
||||
- Security Policy: Basic256
|
||||
- Secure Channel Timeout: 10,000,000 ms
|
||||
- Session Timeout: 10,000,000 ms
|
||||
|
||||
The method configures:
|
||||
- Client application URI
|
||||
- Certificate-based authentication
|
||||
- Server certificate validation (if provided)
|
||||
- Connection timeouts for stability
|
||||
"""
|
||||
|
||||
if not all([self.cert_path, self.private_key_path]):
|
||||
raise ValueError(
|
||||
'Certificate and private key paths must be provided for secure connection.'
|
||||
)
|
||||
cert = str(Path(self.cert_path)) if self.cert_path else None
|
||||
private_key = str(Path(self.private_key_path)) if self.private_key_path else None
|
||||
server_cert = str(Path(self.server_cert_path)) if self.server_cert_path else None
|
||||
assert cert is not None and private_key is not None
|
||||
|
||||
if self.client:
|
||||
self.client.application_uri = self.server_uri
|
||||
self.logger.info('Setting security...')
|
||||
await self.client.set_security(
|
||||
SecurityPolicyBasic256,
|
||||
certificate=cert,
|
||||
private_key=private_key,
|
||||
server_certificate=server_cert,
|
||||
)
|
||||
self.client.secure_channel_timeout = 10000000
|
||||
self.client.session_timeout = 10000000
|
||||
|
||||
async def connect(self):
|
||||
"""
|
||||
Establishes a connection to the OPC server.
|
||||
|
||||
This method initializes the OPC client using the provided URL and
|
||||
sets up security if a certificate path is specified. It then
|
||||
attempts to connect to the server and logs the connection status.
|
||||
|
||||
The connection process includes:
|
||||
1. Client initialization with server URL
|
||||
2. Security configuration (if certificates are provided)
|
||||
3. Connection establishment
|
||||
4. Metrics recording for monitoring
|
||||
|
||||
Raises:
|
||||
Exception: If the connection to the OPC server fails.
|
||||
|
||||
Metrics:
|
||||
- OPC_CONNECTIONS_TOTAL: Incremented on connection attempt
|
||||
- OPC_CONNECTION_STATUS: Set to 1 on successful connection
|
||||
"""
|
||||
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_CONNECTIONS_TOTAL,
|
||||
tags={
|
||||
'pod_id': self.pod_id,
|
||||
'server_name': self.name,
|
||||
},
|
||||
)
|
||||
try:
|
||||
self.client = Client(self.url, timeout=10, watchdog_intervall=3600000)
|
||||
assert self.client is not None # Informa ao mypy que client não é None
|
||||
|
||||
self.client.name = self.pod_id
|
||||
self.client.description = self.pod_id
|
||||
pod_uri = self.pod_id.replace('-', ':')
|
||||
self.client.application_uri = pod_uri
|
||||
self.client.product_uri = pod_uri
|
||||
|
||||
if self.cert_path:
|
||||
await self.set_security()
|
||||
self.logger.info(f'Starting connection to {self.name}...')
|
||||
await self.client.connect()
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_CONNECTION_STATUS,
|
||||
method='set',
|
||||
value=1,
|
||||
tags={
|
||||
'pod_id': self.pod_id,
|
||||
'server_name': self.name,
|
||||
'server_url': self.url,
|
||||
},
|
||||
)
|
||||
self.logger.info(f'Connection to {self.name} successful.')
|
||||
except Exception:
|
||||
await self.disconnect()
|
||||
|
||||
raise
|
||||
|
||||
async def create_subscription(self, name: str):
|
||||
"""
|
||||
Creates a subscription with the specified monitoring period.
|
||||
|
||||
This method establishes a subscription to monitor data changes or events
|
||||
from the OPC UA server. If the client is not connected, an exception is raised.
|
||||
|
||||
Args:
|
||||
name (str): The name identifier for the subscription
|
||||
period (int, optional): The monitoring period in milliseconds. Defaults to 500 ms.
|
||||
|
||||
Raises:
|
||||
ValueError: If the client is not connected.
|
||||
|
||||
Side Effects:
|
||||
- Sets the `self.period` attribute to the specified or default period.
|
||||
- Creates a subscription and assigns it to `self.subscriptions[name]`.
|
||||
- Logs the creation of the subscription.
|
||||
- Increments subscription creation metrics.
|
||||
"""
|
||||
|
||||
if not self.client:
|
||||
raise ValueError('Client not connected. Call connect first.')
|
||||
try:
|
||||
self.subscriptions[name] = await self.client.create_subscription(
|
||||
self.subscription_period_ms, self
|
||||
)
|
||||
self.logger.info(f'Subscription {name} created on {self.name}.')
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_SUBSCRIPTIONS_CREATED,
|
||||
method='inc',
|
||||
value=1,
|
||||
tags={
|
||||
'pod_id': self.pod_id,
|
||||
'server_name': self.name,
|
||||
'slot_name': name,
|
||||
},
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.error(f'Failed to create subscription {name} on {self.name}: {e}')
|
||||
raise
|
||||
|
||||
async def subscribe(self, subscription: str, nodes: dict, collect_period: int):
|
||||
"""
|
||||
Subscribes to a set of OPC UA nodes for data change notifications.
|
||||
|
||||
This method adds the specified nodes to the subscription and configures
|
||||
their data collection rules based on the provided collection period and
|
||||
node-specific frequency.
|
||||
|
||||
Args:
|
||||
subscription (str): The name of the subscription to use
|
||||
nodes (dict): A dictionary where keys are node identifiers and values are
|
||||
configurations for each node. Each configuration must include a 'frequency'
|
||||
key indicating the frequency of data collection in Hz.
|
||||
collect_period (int): The data collection period in seconds.
|
||||
|
||||
Raises:
|
||||
ValueError: If the subscription has not been created by calling
|
||||
`create_subscription` prior to this method.
|
||||
|
||||
Side Effects:
|
||||
- Updates internal node tracking and cycle rules
|
||||
- Establishes data change monitoring for specified nodes
|
||||
- Updates metrics for subscribed tags count
|
||||
"""
|
||||
|
||||
if not self.subscriptions.get(subscription):
|
||||
raise ValueError('Subscription not created. Call create_subscription first.')
|
||||
|
||||
self.logger.info(f'Subscribing to {subscription} on {self.name}...')
|
||||
self.logger.info(f'Subscribing to nodes: {nodes}')
|
||||
assert self.client is not None # Informa ao mypy que client não é None
|
||||
addr_nodes = [self.client.get_node(n) for n in nodes]
|
||||
self.logger.debug(f'Addr nodes: {addr_nodes}')
|
||||
self.nodes.update(nodes)
|
||||
self.logger.debug(f'Nodes: {self.nodes}')
|
||||
|
||||
self.collect_period = collect_period
|
||||
|
||||
for node, config in self.nodes.items():
|
||||
self.nodes[node]['cycle_rule'] = {
|
||||
'cycle_increment': collect_period * 1000 / float(config['frequency']),
|
||||
'cycle_count': 0,
|
||||
}
|
||||
|
||||
await self.subscriptions[subscription].subscribe_data_change(addr_nodes)
|
||||
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
|
||||
method='set',
|
||||
value=len(self.nodes),
|
||||
tags={
|
||||
'pod_id': self.pod_id,
|
||||
'server_name': self.name,
|
||||
},
|
||||
)
|
||||
|
||||
async def unsubscribe(self, subscription: str):
|
||||
"""
|
||||
Unsubscribes from a given subscription.
|
||||
|
||||
This method removes the specified subscription and cleans up associated
|
||||
resources. It handles cases where the subscription doesn't exist gracefully.
|
||||
|
||||
Args:
|
||||
subscription (str): The name of the subscription to unsubscribe from.
|
||||
|
||||
Logs:
|
||||
- A warning if the specified subscription does not exist.
|
||||
- An info message upon successful unsubscription.
|
||||
|
||||
Behavior:
|
||||
- If the subscription exists, it is deleted and removed from the
|
||||
subscriptions dictionary.
|
||||
- If the subscription does not exist, no action is taken.
|
||||
"""
|
||||
|
||||
if not self.subscriptions.get(subscription):
|
||||
self.logger.warning(f"Subscription '{subscription}' not found. Cannot unsubscribe.")
|
||||
return
|
||||
await self.subscriptions[subscription].delete()
|
||||
del self.subscriptions[subscription]
|
||||
self.logger.info(f'Unsubscribed from {subscription}.')
|
||||
|
||||
async def disconnection_fallback(self) -> list:
|
||||
"""
|
||||
Tries 5 times to disconnect from the OPC UA server, with a delay of 100ms x try.
|
||||
"""
|
||||
|
||||
assert self.client is not None
|
||||
error_stack = []
|
||||
for i in range(5):
|
||||
try:
|
||||
self.logger.info(f'Disconnecting from OPC UA server, attempt {i + 1} of 5')
|
||||
await self.client.disconnect()
|
||||
return []
|
||||
except Exception as e:
|
||||
self.logger.error(
|
||||
f'Failed to disconnect from OPC UA serve in attempt {i + 1} of 5: {e}'
|
||||
)
|
||||
error_stack.append(
|
||||
{
|
||||
'attempt': i + 1,
|
||||
'error': str(e),
|
||||
'traceback': traceback.format_exc(),
|
||||
}
|
||||
)
|
||||
await asyncio.sleep(0.1 * i)
|
||||
return error_stack
|
||||
|
||||
async def disconnect(self):
|
||||
"""
|
||||
Disconnects from the OPC UA server.
|
||||
|
||||
This method handles the disconnection process by deleting all subscriptions
|
||||
and disconnecting the client from the OPC UA server. It logs the disconnection
|
||||
process and handles any exceptions that may occur during cleanup.
|
||||
|
||||
Raises:
|
||||
Exception: If an error occurs while deleting the subscription or disconnecting
|
||||
from the OPC UA server, it logs the error details.
|
||||
|
||||
Side Effects:
|
||||
- Deletes all active subscriptions
|
||||
- Disconnects the OPC client
|
||||
- Updates connection status metrics
|
||||
- Clears internal client reference
|
||||
"""
|
||||
|
||||
self.logger.warning('Disconnecting from OPC server')
|
||||
if self.client is None:
|
||||
self.logger.warning('Client already disconnected.')
|
||||
return
|
||||
try:
|
||||
for sub in self.subscriptions:
|
||||
await self.subscriptions[sub].delete()
|
||||
self.logger.warning('Deleted all subscriptions.')
|
||||
except Exception as sub_error:
|
||||
self.logger.error(f'Failed to clean up subscription: {sub_error}')
|
||||
|
||||
errors = await self.disconnection_fallback()
|
||||
|
||||
if errors:
|
||||
await self.send_notification_async(
|
||||
metadata=self.metadata,
|
||||
notification_id=f'OPC_DISCONNECTION_ERROR_{self.name}',
|
||||
message=f'Failed to disconnect from OPC UA server {self.name} after 5 attempts',
|
||||
block='opc_manager',
|
||||
level=NotificationLevel.ERROR,
|
||||
attachment_content=json.dumps(errors, indent=4),
|
||||
)
|
||||
else:
|
||||
self.logger.warning('Disconnected from OPC UA server.')
|
||||
|
||||
del self.client
|
||||
self.client = None
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_CONNECTION_STATUS,
|
||||
method='set',
|
||||
value=0,
|
||||
tags={
|
||||
'pod_id': self.pod_id,
|
||||
'server_name': self.name,
|
||||
'server_url': self.url,
|
||||
},
|
||||
)
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_TAGS_SUBSCRIBED,
|
||||
method='set',
|
||||
value=0,
|
||||
tags={
|
||||
'pod_id': self.pod_id,
|
||||
'server_name': self.name,
|
||||
},
|
||||
)
|
||||
|
||||
async def datachange_notification(self, node, _val, data):
|
||||
"""
|
||||
Handles data change notifications for monitored OPC UA nodes.
|
||||
|
||||
This method is triggered when a monitored node's value changes. It processes
|
||||
the notification, updates internal state, and publishes the data to the
|
||||
appropriate topics.
|
||||
|
||||
Args:
|
||||
node (NodeId): The OPC UA node that triggered the data change notification.
|
||||
_val (Any): The new value of the node (unused in this implementation).
|
||||
data (DataChangeNotification): The data change notification object containing
|
||||
details about the change.
|
||||
|
||||
Behavior:
|
||||
- Extracts the value and source timestamp from the monitored item.
|
||||
- Resets the cycle count for the node's cycle rule.
|
||||
- Resets the non-receive count.
|
||||
- Constructs a data dictionary containing the tag, tag name, timestamp, and value.
|
||||
- Publishes the data to all topics associated with the node.
|
||||
"""
|
||||
|
||||
# get data value
|
||||
monitored_item = data.monitored_item
|
||||
value = monitored_item.Value.Value.Value
|
||||
# source_timestamp
|
||||
source_timestamp = monitored_item.Value.SourceTimestamp.replace(tzinfo=OPC_TIMEZONE)
|
||||
tag = str(node)
|
||||
|
||||
self.logger.debug(
|
||||
f'Data change notification received for tag:'
|
||||
f'{tag} after {self.nodes[tag]["cycle_rule"]["cycle_count"]} cycles'
|
||||
)
|
||||
|
||||
data = {
|
||||
'tag': tag,
|
||||
'name': self.nodes[str(node)]['tag_name'],
|
||||
'timestamp': source_timestamp.strftime(DATETIME_FORMAT_WITH_TZ),
|
||||
'value': value,
|
||||
}
|
||||
|
||||
for topic in self.nodes[tag]['topics']:
|
||||
await self.data_manager.publish(topic, data)
|
||||
|
||||
self.nodes[tag]['cycle_rule']['cycle_count'] = 0
|
||||
self.non_receive_count = 0
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
|
||||
method='set',
|
||||
value=0,
|
||||
tags={
|
||||
'pod_id': self.pod_id,
|
||||
'server_name': self.name,
|
||||
},
|
||||
)
|
||||
|
||||
async def check_cycles(self):
|
||||
"""
|
||||
Checks the cycle counts for all monitored nodes and sends
|
||||
notifications if thresholds are exceeded.
|
||||
|
||||
This method iterates through all monitored nodes and updates their cycle counts based on
|
||||
configured increments. If a node's cycle count exceeds a threshold (5 cycles), it triggers
|
||||
a warning notification.
|
||||
|
||||
Side Effects:
|
||||
- Updates cycle counts for all monitored nodes
|
||||
- Sends warning notifications for nodes exceeding cycle thresholds
|
||||
"""
|
||||
for node, config in self.nodes.items():
|
||||
self.nodes[node]['cycle_rule']['cycle_count'] += config['cycle_rule']['cycle_increment']
|
||||
if self.nodes[node]['cycle_rule']['cycle_count'] >= 5:
|
||||
name = config['tag_name']
|
||||
cycles = self.nodes[node]['cycle_rule']['cycle_count']
|
||||
await self.send_notification_async(
|
||||
metadata=self.metadata,
|
||||
notification_id=f'TAG_{node}:{name}_LISTENNING_STOPPED',
|
||||
message=f'{cycles} cycles without receive from {node}:{name}',
|
||||
block='opc_manager',
|
||||
level=NotificationLevel.WARNING,
|
||||
)
|
||||
|
||||
async def check_opc_listenning(self) -> bool:
|
||||
"""
|
||||
Checks the OPC connection and triggers notifications if the connection is lost.
|
||||
|
||||
This method monitors the data reception health by tracking cycles without
|
||||
data. It sends notifications at different thresholds and can trigger
|
||||
reconnection attempts.
|
||||
|
||||
Returns:
|
||||
bool: True if the connection is lost and reconnection should be attempted,
|
||||
False otherwise.
|
||||
|
||||
Side Effects:
|
||||
- Increments non-receive count
|
||||
- Updates metrics for cycles without data
|
||||
- Sends warning notifications at 5 cycles
|
||||
- Sends error notifications and triggers reconnection at 15 cycles
|
||||
"""
|
||||
|
||||
self.non_receive_count += 1
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
|
||||
method='set',
|
||||
value=self.non_receive_count,
|
||||
tags={
|
||||
'pod_id': self.pod_id,
|
||||
'server_name': self.name,
|
||||
},
|
||||
)
|
||||
if self.non_receive_count >= 5:
|
||||
await self.send_notification_async(
|
||||
metadata=self.metadata,
|
||||
notification_id=f'OPC_LISTENNING_STOPPED__{self.name}',
|
||||
message=f'{self.non_receive_count} cycles without '
|
||||
f'receive from OPC {self.name}. Tags: {json.dumps(self.nodes)}',
|
||||
block='opc_manager',
|
||||
level=NotificationLevel.ERROR,
|
||||
)
|
||||
if self.non_receive_count >= 15:
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_RECONNECTIONS_TOTAL,
|
||||
tags={
|
||||
'pod_id': self.pod_id,
|
||||
'server_name': self.name,
|
||||
},
|
||||
)
|
||||
await self.send_notification_async(
|
||||
metadata=self.metadata,
|
||||
notification_id=f'OPC_CONNECTION_RETRY__{self.name}',
|
||||
message=f'Retrying to connect to server {self.name}',
|
||||
block='opc_manager',
|
||||
level=NotificationLevel.ERROR,
|
||||
)
|
||||
return True
|
||||
return False
|
||||
@@ -1,13 +1,11 @@
|
||||
"""Per-server read path driven by the IngestorManager (SIENTIAPDE-1974).
|
||||
|
||||
Replaces the legacy OpcManager behind the same interface the IngestorManager
|
||||
already uses (connect / create_subscription / subscribe / unsubscribe /
|
||||
check_cycles / check_opc_listenning / shutdown), so the slot lease loop is
|
||||
unchanged. The difference is underneath:
|
||||
Exposes the interface the IngestorManager drives (connect / create_subscription /
|
||||
subscribe / unsubscribe / check_cycles / check_opc_listenning / shutdown):
|
||||
|
||||
- the connection comes from the shared OpcConnectionPool, so the certificate is
|
||||
fetched from MinIO by CertProvider (Basic256Sha256) — never from a file shipped
|
||||
in the Gitea repository;
|
||||
- the connection comes from the shared OpcConnectionPool, with the slot's security
|
||||
mode and policy and the client certificate the slot names, which CertProvider
|
||||
downloads from MinIO (no certificate: no security);
|
||||
- readings go through SubscriptionEngine, which fans each value out to every
|
||||
`names` entry of the slot v2 (one document per collection / tag name / model).
|
||||
"""
|
||||
@@ -58,7 +56,7 @@ class ServerReader:
|
||||
|
||||
async def connect(self):
|
||||
"""Open (or reuse) the pooled connection; raises if the server is unreachable
|
||||
or the certificate can't be fetched, like OpcManager.connect did."""
|
||||
or the certificate can't be fetched."""
|
||||
await self.pool.get(self.server_id, self.server_cfg)
|
||||
|
||||
async def create_subscription(self, slot: str):
|
||||
|
||||
@@ -1,95 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Gera o certificado X.509 de cliente OPC UA do ingestor e sobe pro Gitea
|
||||
# in-cluster (repo que o pod clona no boot — nada muda no values/deployment,
|
||||
# só rollout restart depois do push).
|
||||
#
|
||||
# Uso:
|
||||
# GITEA_PASSWORD='<senha>' ./scripts/generate-and-push-opc-cert.sh
|
||||
#
|
||||
# Vars (defaults pro cluster dev/gcp):
|
||||
# GITEA_URL default https://git.sientia.ai
|
||||
# GITEA_USER default gitea_admin
|
||||
# GITEA_PASSWORD obrigatória
|
||||
# GITEA_REPO default gitea_admin/sientia-dataops-opc-ingestor
|
||||
# BRANCH default main
|
||||
# APP_URI default urn:sientia:opc-ingestor (SAN URI do cert — TEM que
|
||||
# ser igual ao campo `uri` do documento na collection
|
||||
# OPC_servers: o ingestor usa esse valor como application_uri
|
||||
# e o KEPServer rejeita com BadCertificateUriInvalid se divergir)
|
||||
# DAYS default 3650
|
||||
# FORCE=1 sobrescreve cert já existente no repo
|
||||
set -euo pipefail
|
||||
|
||||
GITEA_URL=${GITEA_URL:-https://git.sientia.ai}
|
||||
GITEA_USER=${GITEA_USER:-gitea_admin}
|
||||
GITEA_PASSWORD=${GITEA_PASSWORD:?export GITEA_PASSWORD com a senha do Gitea}
|
||||
GITEA_REPO=${GITEA_REPO:-gitea_admin/sientia-dataops-opc-ingestor}
|
||||
BRANCH=${BRANCH:-main}
|
||||
APP_URI=${APP_URI:-urn:sientia:opc-ingestor}
|
||||
CN=${CN:-sientia-opc-ingestor}
|
||||
DAYS=${DAYS:-3650}
|
||||
|
||||
CERT_DIR_IN_REPO="certs"
|
||||
CERT_NAME="opc-ingestor-cert.pem"
|
||||
KEY_NAME="opc-ingestor-key.pem"
|
||||
DER_NAME="opc-ingestor-cert.der"
|
||||
|
||||
workdir=$(mktemp -d)
|
||||
trap 'rm -rf "$workdir"' EXIT
|
||||
|
||||
echo ">> Gerando certificado (CN=${CN}, URI=${APP_URI}, ${DAYS} dias)..."
|
||||
# Extensões exigidas por OPC UA (KEPServer valida keyUsage/EKU e o SAN URI).
|
||||
openssl req -x509 -newkey rsa:2048 -sha256 -nodes -days "$DAYS" \
|
||||
-keyout "$workdir/$KEY_NAME" -out "$workdir/$CERT_NAME" \
|
||||
-subj "/CN=${CN}/O=Aignosi/OU=Sientia" \
|
||||
-addext "subjectAltName=URI:${APP_URI},DNS:${CN}" \
|
||||
-addext "keyUsage=critical,digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment" \
|
||||
-addext "extendedKeyUsage=critical,clientAuth,serverAuth" \
|
||||
-addext "basicConstraints=critical,CA:FALSE"
|
||||
|
||||
# Cópia DER — só referência p/ import manual no KEPServer (o trust normal é
|
||||
# aceitar o cert rejeitado na 1ª conexão).
|
||||
openssl x509 -in "$workdir/$CERT_NAME" -outform der -out "$workdir/$DER_NAME"
|
||||
|
||||
echo ">> Clonando ${GITEA_REPO}@${BRANCH} do Gitea..."
|
||||
encoded_pass=$(python3 - <<EOF
|
||||
import urllib.parse; print(urllib.parse.quote('''${GITEA_PASSWORD}''', safe=''))
|
||||
EOF
|
||||
)
|
||||
clone_url="${GITEA_URL%%/}"
|
||||
clone_url="${clone_url/:\/\//:\/\/${GITEA_USER}:${encoded_pass}@}/${GITEA_REPO}.git"
|
||||
git clone --depth 1 --branch "$BRANCH" "$clone_url" "$workdir/repo" 2>&1 | sed "s#${encoded_pass}#***#g"
|
||||
|
||||
if [[ -f "$workdir/repo/$CERT_DIR_IN_REPO/$CERT_NAME" && "${FORCE:-0}" != "1" ]]; then
|
||||
echo "ERRO: $CERT_DIR_IN_REPO/$CERT_NAME já existe no repo. Re-rodar com FORCE=1 sobrescreve" >&2
|
||||
echo " (o cert antigo deixa de valer — retrust no KEPServer necessário)." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$workdir/repo/$CERT_DIR_IN_REPO"
|
||||
cp "$workdir/$CERT_NAME" "$workdir/$KEY_NAME" "$workdir/$DER_NAME" "$workdir/repo/$CERT_DIR_IN_REPO/"
|
||||
|
||||
cd "$workdir/repo"
|
||||
git add "$CERT_DIR_IN_REPO"
|
||||
git -c user.name="opc-cert-script" -c user.email="platform@aignosi.com.br" \
|
||||
commit -m "Add OPC UA client certificate for KEPServer connection (URI ${APP_URI})"
|
||||
git push origin "$BRANCH" 2>&1 | sed "s#${encoded_pass}#***#g"
|
||||
|
||||
fingerprint=$(openssl x509 -in "$CERT_DIR_IN_REPO/$CERT_NAME" -noout -fingerprint -sha1)
|
||||
|
||||
cat <<EOF
|
||||
|
||||
== PRONTO ==
|
||||
Cert no Gitea: ${GITEA_REPO}@${BRANCH} -> ${CERT_DIR_IN_REPO}/{${CERT_NAME},${KEY_NAME},${DER_NAME}}
|
||||
${fingerprint}
|
||||
|
||||
Paths dentro do pod (o entrypoint clona o repo em /app):
|
||||
cert_path: /app/${CERT_DIR_IN_REPO}/${CERT_NAME}
|
||||
private_key_path: /app/${CERT_DIR_IN_REPO}/${KEY_NAME}
|
||||
|
||||
Próximos passos:
|
||||
1. Documento na collection OPC_servers com "uri": "${APP_URI}" (ver README/PR).
|
||||
2. kubectl -n sientia rollout restart deployment/opc-ingestor # re-clona o repo
|
||||
3. Pod tenta conectar -> no KEPServer: OPC UA Configuration Manager ->
|
||||
Trusted Clients -> cert '${CN}' rejeitado -> Trust -> restart runtime.
|
||||
EOF
|
||||
@@ -3,7 +3,7 @@
|
||||
Populates:
|
||||
1. Mongo `connectors` + `connector_tags` (each tag with `metadata: {}` — required
|
||||
for documentation, per the feature owner).
|
||||
2. Redis slot v1 (`slot:opc_tags:1`) so the legacy subscription path collects them.
|
||||
2. Redis slot v2 (`slot:opc_tags:1`, contract C1) so the subscription path collects them.
|
||||
3. Postgres test predictions table (with the `write_status` column) + one row the
|
||||
NATS write test can UPDATE (correlated by timestamp + model_id).
|
||||
|
||||
@@ -37,41 +37,53 @@ except ImportError:
|
||||
|
||||
OPC_URL = os.getenv('OPC_SERVER_URL', 'opc.tcp://localhost:49320')
|
||||
|
||||
# No client certificate: the gateway connects without security (SIENTIAPDE-1974 D5).
|
||||
# To test security, point client_certificate_id at an active client certificate of
|
||||
# opc_certificates and set mode/policy (e.g. SignAndEncrypt / Basic256Sha256).
|
||||
SECURITY = {'mode': 'None', 'policy': 'None', 'client_certificate_id': None}
|
||||
|
||||
CONNECTOR = {
|
||||
'connector_id': 9001,
|
||||
'connector_id': 'e2e-kepserver',
|
||||
'type': 'opc_ua',
|
||||
'name': 'e2e-kepserver',
|
||||
'url': OPC_URL,
|
||||
'server_uri': 'urn:e2e:gateway:test',
|
||||
'subscription_period_ms': 1000,
|
||||
'use_security': False, # round 1; flip after certs are in sientia-dataops
|
||||
'security': SECURITY,
|
||||
'status': 'Active',
|
||||
'description': 'OPC Gateway e2e validation connector (SIENTIAPDE-1972)',
|
||||
}
|
||||
|
||||
# metadata: {} on every tag — feature owner asked test tags to carry the field.
|
||||
TAGS = [
|
||||
{'tag_id': 9101, 'connector_id': 9001, 'address': 'ns=2;s=Simulation Examples.Functions.Ramp1',
|
||||
{'tag_id': 9101, 'connector_id': 'e2e-kepserver', 'address': 'ns=2;s=Simulation Examples.Functions.Ramp1',
|
||||
'tag_name': 'e2e_ramp', 'category': 'Read', 'data_type': 'float', 'unit': '-',
|
||||
'status': 'Active', 'description': 'e2e read tag', 'metadata': {}},
|
||||
{'tag_id': 9102, 'connector_id': 9001, 'address': 'ns=2;s=Simulation Examples.Functions.User1',
|
||||
{'tag_id': 9102, 'connector_id': 'e2e-kepserver', 'address': 'ns=2;s=Simulation Examples.Functions.User1',
|
||||
'tag_name': 'e2e_write_target', 'category': 'Write', 'data_type': 'float', 'unit': '-',
|
||||
'status': 'Active', 'description': 'e2e write tag', 'metadata': {}},
|
||||
]
|
||||
|
||||
# Slot v1 (legacy path the gateway still runs until SIENTIAPDE-1974 delivers slot v2).
|
||||
SLOT_V1 = {
|
||||
# Slot v2 (contract C1), as the Orchestrator writes it for the connector above.
|
||||
SLOT = {
|
||||
'e2e-kepserver': {
|
||||
'name': 'e2e-kepserver',
|
||||
'server_id': '9001',
|
||||
'server_id': 'e2e-kepserver',
|
||||
'url': OPC_URL,
|
||||
'server_uri': 'urn:e2e:gateway:test',
|
||||
'subscription_period_ms': 1000,
|
||||
'security': {'mode': SECURITY['mode'], 'policy': SECURITY['policy']},
|
||||
'certificate': None,
|
||||
'tags': {
|
||||
TAGS[0]['address']: {
|
||||
'tag_name': TAGS[0]['tag_name'],
|
||||
'frequency': '5000',
|
||||
'topics': ['raw_e2e-gateway-test'],
|
||||
'frequency': 5000,
|
||||
'active_read': {'enabled': False},
|
||||
'names': [
|
||||
{
|
||||
'collection': 'raw_e2e-gateway-test',
|
||||
'tag_name': TAGS[0]['tag_name'],
|
||||
'model_id': '9001',
|
||||
'model_name': 'e2e-model',
|
||||
}
|
||||
],
|
||||
}
|
||||
},
|
||||
}
|
||||
@@ -94,12 +106,11 @@ ON CONFLICT DO NOTHING;
|
||||
|
||||
# The matching NATS write command for the row above (paste into `nats pub opc.write '...'`).
|
||||
NATS_CMD = {
|
||||
'msg_id': 'e2e-0001', 'server_id': '9001', 'addr': TAGS[1]['address'],
|
||||
'msg_id': 'e2e-0001', 'server_id': 'e2e-kepserver', 'addr': TAGS[1]['address'],
|
||||
'value': 42.5, 'data_type': 'float', 'tag_type': 'prediction',
|
||||
'model_id': '9001', 'model_name': 'e2e-model',
|
||||
'schema': 'sientia_data', 'table_name': 'e2e_predictions',
|
||||
'timestamp': '2026-07-15T12:00:00Z',
|
||||
'url': OPC_URL, 'use_security': False, 'server_uri': 'urn:e2e:gateway:test',
|
||||
}
|
||||
|
||||
|
||||
@@ -119,8 +130,8 @@ def seed_redis():
|
||||
import redis
|
||||
|
||||
r = redis.Redis(host=os.getenv('REDIS_HOST', 'localhost'), port=int(os.getenv('REDIS_PORT', '6379')))
|
||||
r.set('slot:opc_tags:1', json.dumps(SLOT_V1))
|
||||
print('redis: wrote slot:opc_tags:1 (v1)')
|
||||
r.set('slot:opc_tags:1', json.dumps(SLOT))
|
||||
print('redis: wrote slot:opc_tags:1 (v2)')
|
||||
|
||||
|
||||
def seed_postgres():
|
||||
@@ -144,7 +155,7 @@ def main() -> int:
|
||||
if args.print_only:
|
||||
print('connector:', json.dumps(CONNECTOR, indent=2))
|
||||
print('tags:', json.dumps(TAGS, indent=2))
|
||||
print('slot v1:', json.dumps(SLOT_V1, indent=2))
|
||||
print('slot v2:', json.dumps(SLOT, indent=2))
|
||||
print('sql:', PREDICTIONS_SQL)
|
||||
print('nats write cmd:', json.dumps(NATS_CMD, indent=2))
|
||||
return 0
|
||||
|
||||
67
tests/unit/connection/test_certificates.py
Normal file
67
tests/unit/connection/test_certificates.py
Normal file
@@ -0,0 +1,67 @@
|
||||
import pytest
|
||||
|
||||
from ingestor.connection.certificates import (
|
||||
client_certificate_id,
|
||||
has_security,
|
||||
resolve_certificate,
|
||||
)
|
||||
|
||||
LOOKED_UP = {
|
||||
'certificate_id': 1,
|
||||
'application_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/p/cert.pem'},
|
||||
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/p/key.pem'},
|
||||
}
|
||||
|
||||
|
||||
def _connector(certificates, certificate_id=1):
|
||||
return {
|
||||
'connector_id': 'kepserver',
|
||||
'url': 'opc.tcp://kep:49320',
|
||||
'security': {
|
||||
'mode': 'SignAndEncrypt',
|
||||
'policy': 'Basic256Sha256',
|
||||
'client_certificate_id': certificate_id,
|
||||
},
|
||||
'certificates': certificates,
|
||||
}
|
||||
|
||||
|
||||
def test_resolve_certificate_keeps_only_the_slot_fields():
|
||||
looked_up = {
|
||||
**LOOKED_UP,
|
||||
'thumbprint_sha1': 'ab',
|
||||
'cert': {**LOOKED_UP['cert'], 'sha256': None},
|
||||
}
|
||||
|
||||
assert resolve_certificate(_connector([looked_up])) == LOOKED_UP
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'certificates',
|
||||
[
|
||||
pytest.param(None, id='no-lookup'),
|
||||
pytest.param([], id='not-found'),
|
||||
pytest.param([{**LOOKED_UP, 'application_uri': ''}], id='no-application-uri'),
|
||||
pytest.param([{**LOOKED_UP, 'cert': None}], id='no-cert'),
|
||||
pytest.param([{**LOOKED_UP, 'private_key': {'object_name': 'k'}}], id='key-no-bucket'),
|
||||
pytest.param(['not a document'], id='not-a-document'),
|
||||
],
|
||||
)
|
||||
def test_resolve_certificate_without_a_usable_certificate(certificates):
|
||||
assert resolve_certificate(_connector(certificates)) is None
|
||||
|
||||
|
||||
def test_client_certificate_id():
|
||||
assert client_certificate_id(_connector([])) == 1
|
||||
assert client_certificate_id(_connector([], certificate_id=None)) is None
|
||||
assert client_certificate_id({'connector_id': 'x'}) is None
|
||||
assert client_certificate_id({'security': None}) is None
|
||||
|
||||
|
||||
def test_has_security():
|
||||
assert has_security(_connector([]))
|
||||
assert has_security({'security': {'mode': 'None', 'policy': 'None'}})
|
||||
assert not has_security({'security': {'mode': 'Sign'}})
|
||||
assert not has_security({'security': None})
|
||||
assert not has_security({})
|
||||
@@ -1,136 +1,107 @@
|
||||
from unittest.mock import AsyncMock
|
||||
import asyncio
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from ingestor.connection.certs import CertProvider
|
||||
from ingestor.connection.certs import CertificateNotConfiguredError, CertProvider, LazyMinio
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_returns_local_paths(tmp_path):
|
||||
mongo = AsyncMock()
|
||||
mongo.find_one.return_value = {
|
||||
'client_cert': {'minio_object_name': 'platform/certificates/client_cert_c.pem'},
|
||||
'private_key': {'minio_object_name': 'platform/certificates/private_key_k.pem'},
|
||||
def certificate(certificate_id=1, name='sientia-platform', ext='pem', bucket='opc-certificates'):
|
||||
"""A client certificate as the slot v2 carries it."""
|
||||
return {
|
||||
'certificate_id': certificate_id,
|
||||
'application_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert': {'bucket': bucket, 'object_name': f'client/{name}/cert.{ext}'},
|
||||
'private_key': {'bucket': bucket, 'object_name': f'client/{name}/key.{ext}'},
|
||||
}
|
||||
minio = AsyncMock()
|
||||
minio.download_file.side_effect = [b'cert-bytes', b'key-bytes']
|
||||
|
||||
provider = CertProvider(mongo, minio, cache_dir=str(tmp_path), bucket='sientia-dataops')
|
||||
cert, key = await provider.fetch()
|
||||
|
||||
assert cert.endswith('c.pem') and key.endswith('k.pem')
|
||||
assert minio.download_file.await_count == 2
|
||||
mongo.find_one.assert_awaited_once_with('platform_certificates', {}, {})
|
||||
minio.download_file.assert_any_await(
|
||||
'platform/certificates/client_cert_c.pem', 'sientia-dataops', {}
|
||||
)
|
||||
assert (tmp_path / 'client_cert_c.pem').read_bytes() == b'cert-bytes'
|
||||
assert (tmp_path / 'private_key_k.pem').read_bytes() == b'key-bytes'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lazy_minio_builds_client_once_on_first_download():
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from ingestor.connection.certs import LazyMinio
|
||||
|
||||
client = AsyncMock()
|
||||
client.download_file.return_value = b'bytes'
|
||||
factory = MagicMock(return_value=client)
|
||||
minio = LazyMinio(factory)
|
||||
|
||||
factory.assert_not_called() # no network at construction
|
||||
assert await minio.download_file('a.pem', 'bucket', {}) == b'bytes'
|
||||
await minio.download_file('b.pem', 'bucket', {})
|
||||
|
||||
factory.assert_called_once()
|
||||
assert client.download_file.await_count == 2
|
||||
|
||||
|
||||
def test_build_cert_provider_is_lazy_and_uses_config(monkeypatch):
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from ingestor.config import GatewayConfig
|
||||
from ingestor.connection import certs
|
||||
|
||||
minio_cls = MagicMock()
|
||||
monkeypatch.setattr(certs, 'MinioRepository', minio_cls)
|
||||
mongo = MagicMock()
|
||||
|
||||
provider = certs.build_cert_provider(
|
||||
mongo=mongo,
|
||||
config=GatewayConfig(cert_bucket='streamlit-connectors', cert_cache_dir='certs-cache'),
|
||||
logger=MagicMock(),
|
||||
notification_handler=MagicMock(),
|
||||
metrics_controller=MagicMock(),
|
||||
)
|
||||
|
||||
assert provider.mongo is mongo
|
||||
assert provider.bucket == 'streamlit-connectors'
|
||||
assert provider.cache_dir == 'certs-cache'
|
||||
assert isinstance(provider.minio, certs.LazyMinio)
|
||||
minio_cls.assert_not_called() # MinIO is only reached when a cert is needed
|
||||
|
||||
|
||||
def _provider(tmp_path, doc=None, data=b'pem'):
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
mongo = MagicMock()
|
||||
mongo.find_one = AsyncMock(
|
||||
return_value=doc
|
||||
if doc is not None
|
||||
else {
|
||||
'client_cert': {'minio_object_name': 'platform_certificates/c.pem'},
|
||||
'private_key': {'minio_object_name': 'platform_certificates/k.pem'},
|
||||
}
|
||||
)
|
||||
def _provider(tmp_path, data=b'pem'):
|
||||
minio = MagicMock()
|
||||
minio.download_file = AsyncMock(return_value=data)
|
||||
return CertProvider(mongo=mongo, minio=minio, cache_dir=str(tmp_path), bucket='b'), minio
|
||||
return CertProvider(minio=minio, cache_dir=str(tmp_path)), minio
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_downloads_the_objects_the_certificate_names(tmp_path):
|
||||
minio = AsyncMock()
|
||||
minio.download_file.side_effect = [b'cert-bytes', b'key-bytes']
|
||||
provider = CertProvider(minio, cache_dir=str(tmp_path))
|
||||
|
||||
cert, key = await provider.fetch(certificate(7))
|
||||
|
||||
assert (cert, key) == (str(tmp_path / '7' / 'cert.pem'), str(tmp_path / '7' / 'key.pem'))
|
||||
assert [c.args for c in minio.download_file.await_args_list] == [
|
||||
('client/sientia-platform/cert.pem', 'opc-certificates', {}),
|
||||
('client/sientia-platform/key.pem', 'opc-certificates', {}),
|
||||
]
|
||||
assert (tmp_path / '7' / 'cert.pem').read_bytes() == b'cert-bytes'
|
||||
assert (tmp_path / '7' / 'key.pem').read_bytes() == b'key-bytes'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_local_files_keep_the_object_extension(tmp_path):
|
||||
"""asyncua picks the format (.pem / .der) by the extension"""
|
||||
provider, _minio = _provider(tmp_path)
|
||||
|
||||
cert, key = await provider.fetch(certificate(ext='der'))
|
||||
|
||||
assert cert.endswith('cert.der') and key.endswith('key.der')
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
'doc',
|
||||
'broken',
|
||||
[
|
||||
None,
|
||||
{},
|
||||
{'client_cert': {'minio_object_name': 'c.pem'}},
|
||||
{'client_cert': None, 'private_key': {}},
|
||||
pytest.param({'cert': None}, id='no-cert'),
|
||||
pytest.param({'private_key': {'bucket': 'b'}}, id='key-without-object'),
|
||||
pytest.param({'certificate_id': None}, id='no-id'),
|
||||
pytest.param({'certificate_id': '../etc'}, id='id-not-an-integer'),
|
||||
],
|
||||
)
|
||||
async def test_fetch_without_platform_certificate_raises_clear_error(tmp_path, doc):
|
||||
from ingestor.connection.certs import CertificateNotConfiguredError
|
||||
|
||||
async def test_fetch_of_an_incomplete_certificate_raises_clear_error(tmp_path, broken):
|
||||
provider, minio = _provider(tmp_path)
|
||||
provider.mongo.find_one.return_value = doc
|
||||
|
||||
with pytest.raises(CertificateNotConfiguredError, match='platform_certificates'):
|
||||
await provider.fetch()
|
||||
with pytest.raises(CertificateNotConfiguredError, match='opc_certificates|certificate_id'):
|
||||
await provider.fetch({**certificate(), **broken})
|
||||
minio.download_file.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_is_cached_until_invalidated(tmp_path):
|
||||
async def test_fetch_is_cached_per_certificate_until_invalidated(tmp_path):
|
||||
provider, minio = _provider(tmp_path)
|
||||
|
||||
first = await provider.fetch()
|
||||
second = await provider.fetch()
|
||||
assert first == second
|
||||
first = await provider.fetch(certificate(1))
|
||||
assert await provider.fetch(certificate(1)) == first
|
||||
assert minio.download_file.await_count == 2 # cert + key, once
|
||||
|
||||
provider.invalidate()
|
||||
await provider.fetch()
|
||||
other = await provider.fetch(certificate(2, name='other'))
|
||||
assert other != first
|
||||
assert minio.download_file.await_count == 4
|
||||
|
||||
provider.invalidate(1)
|
||||
await provider.fetch(certificate(1))
|
||||
await provider.fetch(certificate(2, name='other')) # still cached
|
||||
assert minio.download_file.await_count == 6
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_replaced_certificate_is_downloaded_again(tmp_path):
|
||||
"""Same certificate_id, other objects in the slot: the cache does not serve the old files"""
|
||||
provider, minio = _provider(tmp_path)
|
||||
|
||||
await provider.fetch(certificate(1, name='old'))
|
||||
await provider.fetch(certificate(1, name='new'))
|
||||
|
||||
assert minio.download_file.await_count == 4
|
||||
assert minio.download_file.await_args_list[-1].args[0] == 'client/new/key.pem'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_concurrent_fetches_download_once(tmp_path):
|
||||
import asyncio
|
||||
|
||||
provider, minio = _provider(tmp_path)
|
||||
|
||||
results = await asyncio.gather(*(provider.fetch() for _ in range(5)))
|
||||
results = await asyncio.gather(*(provider.fetch(certificate()) for _ in range(5)))
|
||||
|
||||
assert len(set(results)) == 1
|
||||
assert minio.download_file.await_count == 2
|
||||
@@ -138,11 +109,51 @@ async def test_concurrent_fetches_download_once(tmp_path):
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_replaces_file_atomically(tmp_path):
|
||||
provider, minio = _provider(tmp_path, data=b'new')
|
||||
(tmp_path / 'c.pem').write_bytes(b'old')
|
||||
provider, _minio = _provider(tmp_path, data=b'new')
|
||||
(tmp_path / '1').mkdir()
|
||||
(tmp_path / '1' / 'cert.pem').write_bytes(b'old')
|
||||
|
||||
cert_path, _ = await provider.fetch()
|
||||
cert_path, _ = await provider.fetch(certificate(1))
|
||||
|
||||
assert (tmp_path / 'c.pem').read_bytes() == b'new'
|
||||
assert cert_path == str(tmp_path / 'c.pem')
|
||||
assert not list(tmp_path.glob('*.tmp'))
|
||||
assert (tmp_path / '1' / 'cert.pem').read_bytes() == b'new'
|
||||
assert cert_path == str(tmp_path / '1' / 'cert.pem')
|
||||
assert not list(tmp_path.glob('**/*.tmp'))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lazy_minio_builds_one_client_per_bucket_on_first_download():
|
||||
clients = {'a': AsyncMock(), 'b': AsyncMock()}
|
||||
for client in clients.values():
|
||||
client.download_file.return_value = b'bytes'
|
||||
factory = MagicMock(side_effect=lambda bucket: clients[bucket])
|
||||
minio = LazyMinio(factory)
|
||||
|
||||
factory.assert_not_called() # no network at construction
|
||||
assert await minio.download_file('x.pem', 'a', {}) == b'bytes'
|
||||
await minio.download_file('y.pem', 'a', {})
|
||||
await minio.download_file('z.pem', 'b', {})
|
||||
|
||||
assert [c.args for c in factory.call_args_list] == [('a',), ('b',)]
|
||||
assert clients['a'].download_file.await_count == 2
|
||||
clients['b'].download_file.assert_awaited_once_with('z.pem', 'b', {})
|
||||
|
||||
|
||||
def test_build_cert_provider_is_lazy_and_uses_config(monkeypatch):
|
||||
from ingestor.config import GatewayConfig
|
||||
from ingestor.connection import certs
|
||||
|
||||
minio_cls = MagicMock()
|
||||
monkeypatch.setattr(certs, 'MinioRepository', minio_cls)
|
||||
|
||||
provider = certs.build_cert_provider(
|
||||
config=GatewayConfig(cert_cache_dir='certs-cache'),
|
||||
logger=MagicMock(),
|
||||
notification_handler=MagicMock(),
|
||||
metrics_controller=MagicMock(),
|
||||
)
|
||||
|
||||
assert provider.cache_dir == 'certs-cache'
|
||||
assert isinstance(provider.minio, certs.LazyMinio)
|
||||
minio_cls.assert_not_called() # MinIO is only reached when a cert is needed
|
||||
provider.minio._factory('opc-certificates')
|
||||
assert minio_cls.call_args.kwargs['bucket'] == 'opc-certificates'
|
||||
|
||||
@@ -6,6 +6,15 @@ import pytest
|
||||
import ingestor.metrics as metrics
|
||||
from ingestor.connection.pool import OpcConnectionPool
|
||||
|
||||
SECURITY = {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}
|
||||
CERTIFICATE = {
|
||||
'certificate_id': 7,
|
||||
'application_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/p/cert.pem'},
|
||||
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/p/key.pem'},
|
||||
}
|
||||
SECURED = {'url': 'opc.tcp://h:4840', 'security': SECURITY, 'certificate': CERTIFICATE}
|
||||
|
||||
|
||||
def make_pool(cert_provider=None):
|
||||
pool = OpcConnectionPool(
|
||||
@@ -23,8 +32,8 @@ async def test_get_reuses_connection_per_server():
|
||||
pool = make_pool()
|
||||
fake_client = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
c1 = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
c2 = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
c1 = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
c2 = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
assert c1 is c2
|
||||
assert fake_client.connect.await_count == 1
|
||||
@@ -37,7 +46,7 @@ async def test_get_failure_records_metric_and_raises():
|
||||
fake_client.connect.side_effect = RuntimeError('boom')
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
with pytest.raises(RuntimeError):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
pool.emit_metric.assert_any_await(
|
||||
metric_object=metrics.OPC_CONNECTIONS_FAILED,
|
||||
@@ -48,13 +57,13 @@ async def test_get_failure_records_metric_and_raises():
|
||||
# a later successful get must reconnect (failed attempt was not cached)
|
||||
fake_client2 = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client2):
|
||||
client = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
client = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
assert client is fake_client2
|
||||
assert fake_client2.connect.await_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_applies_security_when_requested():
|
||||
async def test_get_applies_the_certificate_and_security_of_the_server():
|
||||
cert_provider = AsyncMock()
|
||||
cert_provider.fetch.return_value = ('cert.pem', 'key.pem')
|
||||
pool = make_pool(cert_provider=cert_provider)
|
||||
@@ -63,13 +72,32 @@ async def test_get_applies_security_when_requested():
|
||||
patch('ingestor.connection.pool.Client', return_value=fake_client),
|
||||
patch('ingestor.connection.pool.apply_security', new=AsyncMock()) as mock_apply,
|
||||
):
|
||||
await pool.get(
|
||||
'srv',
|
||||
{'url': 'opc.tcp://h:4840', 'server_uri': 'urn:test', 'use_security': True},
|
||||
)
|
||||
await pool.get('srv', SECURED)
|
||||
|
||||
cert_provider.fetch.assert_awaited_once()
|
||||
mock_apply.assert_awaited_once_with(fake_client, 'urn:test', 'cert.pem', 'key.pem', None)
|
||||
cert_provider.fetch.assert_awaited_once_with(CERTIFICATE)
|
||||
mock_apply.assert_awaited_once_with(fake_client, SECURITY, CERTIFICATE, 'cert.pem', 'key.pem')
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
'cfg',
|
||||
[
|
||||
pytest.param({**SECURED, 'certificate': None}, id='no-certificate'),
|
||||
pytest.param({**SECURED, 'security': {'mode': 'None', 'policy': 'None'}}, id='mode-none'),
|
||||
],
|
||||
)
|
||||
async def test_get_without_certificate_connects_without_security(cfg):
|
||||
"""No client certificate (or mode None): no security, no MinIO download (D5)"""
|
||||
cert_provider = AsyncMock()
|
||||
pool = make_pool(cert_provider=cert_provider)
|
||||
with (
|
||||
patch('ingestor.connection.pool.Client', return_value=AsyncMock()),
|
||||
patch('ingestor.connection.pool.apply_security', new=AsyncMock()) as mock_apply,
|
||||
):
|
||||
await pool.get('srv', cfg)
|
||||
|
||||
cert_provider.fetch.assert_not_awaited()
|
||||
mock_apply.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -77,7 +105,7 @@ async def test_release_idle_evicts_and_disconnects():
|
||||
pool = make_pool()
|
||||
fake_client = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
client, _ = pool._connections['srv']
|
||||
pool._connections['srv'] = (client, -1000.0) # force idle
|
||||
@@ -97,7 +125,7 @@ async def test_release_idle_keeps_recently_used():
|
||||
pool = make_pool()
|
||||
fake_client = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
await pool.release_idle(max_idle_s=3600)
|
||||
|
||||
@@ -110,8 +138,8 @@ async def test_close_all_disconnects_everything():
|
||||
pool = make_pool()
|
||||
fake_a, fake_b = AsyncMock(), AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', side_effect=[fake_a, fake_b]):
|
||||
await pool.get('a', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.get('b', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.get('a', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
await pool.get('b', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
await pool.close_all()
|
||||
|
||||
@@ -126,7 +154,7 @@ async def test_release_idle_disconnect_retries_then_gives_up():
|
||||
fake_client = AsyncMock()
|
||||
fake_client.disconnect.side_effect = RuntimeError('nope')
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
client, _ = pool._connections['srv']
|
||||
pool._connections['srv'] = (client, -1000.0)
|
||||
@@ -142,7 +170,7 @@ async def test_success_path_emits_connection_metrics():
|
||||
pool = make_pool()
|
||||
fake_client = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
pool.emit_metric.assert_any_await(
|
||||
metric_object=metrics.OPC_CONNECTIONS_TOTAL,
|
||||
@@ -170,7 +198,7 @@ async def test_concurrent_get_same_server_connects_once():
|
||||
|
||||
clients[0].connect.side_effect = slow_connect
|
||||
|
||||
cfg = {'url': 'opc.tcp://h:4840', 'use_security': False}
|
||||
cfg = {'url': 'opc.tcp://h:4840', 'certificate': None}
|
||||
with patch('ingestor.connection.pool.Client', side_effect=clients) as ctor:
|
||||
t1 = asyncio.create_task(pool.get('srv', cfg))
|
||||
await started.wait() # t1 is mid-connect, holding the per-server lock
|
||||
@@ -191,7 +219,7 @@ async def test_release_idle_keeps_pinned_connection():
|
||||
pool = make_pool()
|
||||
fake_client = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
pool.pin('srv')
|
||||
await pool.release_idle(max_idle_s=0)
|
||||
@@ -205,7 +233,7 @@ async def test_release_idle_evicts_after_last_unpin():
|
||||
pool = make_pool()
|
||||
fake_client = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
pool.pin('srv')
|
||||
pool.pin('srv') # two slots subscribed on the same server
|
||||
@@ -233,9 +261,9 @@ async def test_evict_disconnects_now_and_next_get_reconnects():
|
||||
pool = make_pool()
|
||||
first, second = AsyncMock(), AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', side_effect=[first, second]):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
await pool.evict('srv')
|
||||
client = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
client = await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
first.disconnect.assert_awaited_once()
|
||||
assert client is second
|
||||
@@ -255,8 +283,8 @@ async def test_get_normalizes_server_id_to_str():
|
||||
pool = make_pool()
|
||||
fake_client = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
c1 = await pool.get(1, {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
c2 = await pool.get('1', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
c1 = await pool.get(1, {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
c2 = await pool.get('1', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
assert c1 is c2
|
||||
assert fake_client.connect.await_count == 1
|
||||
@@ -267,8 +295,8 @@ async def test_get_with_different_cfg_reuses_client_and_warns():
|
||||
pool = make_pool()
|
||||
fake_client = AsyncMock()
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
client = await pool.get('srv', {'url': 'opc.tcp://10.0.0.5:4840', 'use_security': False})
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
client = await pool.get('srv', {'url': 'opc.tcp://10.0.0.5:4840', 'certificate': None})
|
||||
|
||||
assert client is fake_client
|
||||
assert fake_client.connect.await_count == 1
|
||||
@@ -276,6 +304,21 @@ async def test_get_with_different_cfg_reuses_client_and_warns():
|
||||
assert 'url' in pool.logger.warning.call_args.args[0]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_with_another_certificate_warns():
|
||||
cert_provider = AsyncMock()
|
||||
cert_provider.fetch.return_value = ('cert.pem', 'key.pem')
|
||||
pool = make_pool(cert_provider=cert_provider)
|
||||
with (
|
||||
patch('ingestor.connection.pool.Client', return_value=AsyncMock()),
|
||||
patch('ingestor.connection.pool.apply_security', new=AsyncMock()),
|
||||
):
|
||||
await pool.get('srv', SECURED)
|
||||
await pool.get('srv', {**SECURED, 'certificate': {**CERTIFICATE, 'certificate_id': 8}})
|
||||
|
||||
assert 'certificate' in pool.logger.warning.call_args.args[0]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_secured_connect_failure_invalidates_cert_cache():
|
||||
from unittest.mock import MagicMock
|
||||
@@ -290,9 +333,9 @@ async def test_secured_connect_failure_invalidates_cert_cache():
|
||||
patch('ingestor.connection.pool.apply_security', AsyncMock()),
|
||||
):
|
||||
with pytest.raises(RuntimeError):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': True})
|
||||
await pool.get('srv', SECURED)
|
||||
|
||||
cert_provider.invalidate.assert_called_once()
|
||||
cert_provider.invalidate.assert_called_once_with(7)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -305,7 +348,7 @@ async def test_insecure_connect_failure_keeps_cert_cache():
|
||||
fake_client.connect.side_effect = RuntimeError('unreachable')
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
with pytest.raises(RuntimeError):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
cert_provider.invalidate.assert_not_called()
|
||||
|
||||
@@ -317,7 +360,7 @@ async def test_connect_identifies_the_pod_on_the_server():
|
||||
pool = make_pool()
|
||||
fake_client = MagicMock(connect=AsyncMock())
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'use_security': False})
|
||||
await pool.get('srv', {'url': 'opc.tcp://h:4840', 'certificate': None})
|
||||
|
||||
assert fake_client.name == pool.pod_id
|
||||
assert fake_client.description == pool.pod_id
|
||||
@@ -326,20 +369,16 @@ async def test_connect_identifies_the_pod_on_the_server():
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_connect_server_uri_still_wins_for_application_uri():
|
||||
pool = make_pool()
|
||||
fake_client = MagicMock(connect=AsyncMock())
|
||||
async def test_connect_with_certificate_uses_its_application_uri():
|
||||
"""apply_security sets the certificate's URI: it must match its SAN URI
|
||||
(BadCertificateUriInvalid otherwise)"""
|
||||
cert_provider = AsyncMock()
|
||||
cert_provider.fetch.return_value = ('cert.pem', 'key.pem')
|
||||
pool = make_pool(cert_provider=cert_provider)
|
||||
fake_client = MagicMock(connect=AsyncMock(), set_security=AsyncMock())
|
||||
with patch('ingestor.connection.pool.Client', return_value=fake_client):
|
||||
await pool.get(
|
||||
'srv',
|
||||
{
|
||||
'url': 'opc.tcp://h:4840',
|
||||
'use_security': False,
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
},
|
||||
)
|
||||
await pool.get('srv', SECURED)
|
||||
|
||||
# must match the certificate SAN URI (BadCertificateUriInvalid otherwise)
|
||||
assert fake_client.application_uri == 'urn:sientia:opc-ingestor'
|
||||
assert fake_client.name == pool.pod_id
|
||||
|
||||
@@ -347,7 +386,7 @@ async def test_connect_server_uri_still_wins_for_application_uri():
|
||||
@pytest.mark.asyncio
|
||||
async def test_config_for_returns_a_copy_of_the_live_connection_settings():
|
||||
pool = make_pool()
|
||||
cfg = {'url': 'opc.tcp://h:4840', 'server_uri': 'urn:h', 'use_security': False}
|
||||
cfg = {'url': 'opc.tcp://h:4840', 'security': SECURITY, 'certificate': None}
|
||||
with patch('ingestor.connection.pool.Client', return_value=AsyncMock()):
|
||||
await pool.get(1, cfg)
|
||||
|
||||
|
||||
@@ -1,18 +1,92 @@
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from asyncua import ua
|
||||
from asyncua.crypto.security_policies import (
|
||||
SecurityPolicyAes128Sha256RsaOaep,
|
||||
SecurityPolicyBasic256Sha256,
|
||||
)
|
||||
|
||||
from ingestor.connection.security import SESSION_TIMEOUT_MS, apply_security
|
||||
from ingestor.connection.security import (
|
||||
SESSION_TIMEOUT_MS,
|
||||
UnsupportedSecurityError,
|
||||
apply_security,
|
||||
is_secured,
|
||||
)
|
||||
|
||||
CERTIFICATE = {'certificate_id': 1, 'application_uri': 'urn:sientia:opc-ingestor'}
|
||||
|
||||
|
||||
def _client():
|
||||
client = MagicMock()
|
||||
client.set_security = AsyncMock()
|
||||
return client
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_apply_security_sets_sha256_and_channel_lifetime():
|
||||
client = MagicMock()
|
||||
client.set_security = AsyncMock()
|
||||
await apply_security(client, 'urn:x', 'c.pem', 'k.pem', None)
|
||||
async def test_apply_security_uses_mode_and_policy_and_channel_lifetime():
|
||||
client = _client()
|
||||
|
||||
await apply_security(
|
||||
client, {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}, CERTIFICATE, 'c', 'k'
|
||||
)
|
||||
|
||||
assert client.application_uri == 'urn:sientia:opc-ingestor'
|
||||
assert client.session_timeout == SESSION_TIMEOUT_MS
|
||||
assert client.secure_channel_timeout == int(SESSION_TIMEOUT_MS * 1.25)
|
||||
args, kwargs = client.set_security.call_args
|
||||
from asyncua.crypto.security_policies import SecurityPolicyBasic256Sha256
|
||||
|
||||
assert args[0] is SecurityPolicyBasic256Sha256
|
||||
assert kwargs == {
|
||||
'certificate': 'c',
|
||||
'private_key': 'k',
|
||||
'server_certificate': None,
|
||||
'mode': ua.MessageSecurityMode.SignAndEncrypt,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_apply_security_sign_with_another_policy():
|
||||
client = _client()
|
||||
|
||||
await apply_security(
|
||||
client, {'mode': 'Sign', 'policy': 'Aes128Sha256RsaOaep'}, CERTIFICATE, 'c', 'k'
|
||||
)
|
||||
|
||||
args, kwargs = client.set_security.call_args
|
||||
assert args[0] is SecurityPolicyAes128Sha256RsaOaep
|
||||
assert kwargs['mode'] == ua.MessageSecurityMode.Sign
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
'security',
|
||||
[
|
||||
{'mode': 'Encrypt', 'policy': 'Basic256Sha256'},
|
||||
{'mode': 'Sign', 'policy': 'Basic999'},
|
||||
{'mode': 'Sign', 'policy': 'None'},
|
||||
{},
|
||||
],
|
||||
)
|
||||
async def test_apply_security_rejects_unknown_mode_or_policy(security):
|
||||
client = _client()
|
||||
|
||||
with pytest.raises(UnsupportedSecurityError):
|
||||
await apply_security(client, security, CERTIFICATE, 'c', 'k')
|
||||
client.set_security.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
('server_cfg', 'secured'),
|
||||
[
|
||||
({'security': {'mode': 'SignAndEncrypt'}, 'certificate': CERTIFICATE}, True),
|
||||
({'security': {'mode': 'Sign'}, 'certificate': CERTIFICATE}, True),
|
||||
# No certificate: no security (D5), whatever the mode
|
||||
({'security': {'mode': 'SignAndEncrypt'}, 'certificate': None}, False),
|
||||
({'security': {'mode': 'SignAndEncrypt'}}, False),
|
||||
# mode None: no security even with a certificate
|
||||
({'security': {'mode': 'None'}, 'certificate': CERTIFICATE}, False),
|
||||
({}, False),
|
||||
],
|
||||
)
|
||||
def test_is_secured(server_cfg, secured):
|
||||
assert is_secured(server_cfg) is secured
|
||||
|
||||
@@ -2,108 +2,113 @@ from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from ingestor.connection.servers import CONNECTORS, OPC_SERVERS, ServerConfigResolver
|
||||
from ingestor.connection.servers import CONNECTORS, ServerConfigResolver
|
||||
|
||||
LOOKED_UP = {
|
||||
'certificate_id': 1,
|
||||
'application_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/p/cert.pem'},
|
||||
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/p/key.pem'},
|
||||
}
|
||||
# A connector as the aggregation returns it ($lookup on opc_certificates)
|
||||
CONNECTOR = {
|
||||
'connector_id': 1,
|
||||
'type': 'opc_ua',
|
||||
'status': 'Active',
|
||||
'name': 'kepserver',
|
||||
'connector_id': 'kepserver',
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'server_uri': 'urn:kepserver',
|
||||
'use_security': True,
|
||||
'subscription_period_ms': 500,
|
||||
'security': {
|
||||
'mode': 'SignAndEncrypt',
|
||||
'policy': 'Basic256Sha256',
|
||||
'client_certificate_id': 1,
|
||||
},
|
||||
'certificates': [LOOKED_UP],
|
||||
}
|
||||
OPC_SERVER = {
|
||||
'id': '1',
|
||||
'server_name': 'kepserver',
|
||||
'url': 'opc.tcp://legacy:49320',
|
||||
'uri': 'urn:legacy',
|
||||
'cert_path': '/app/certs/cert.pem',
|
||||
# The same server in the slot v2 format
|
||||
CONFIG = {
|
||||
'server_id': 'kepserver',
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'subscription_period_ms': 500,
|
||||
'security': {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'},
|
||||
'certificate': LOOKED_UP,
|
||||
}
|
||||
|
||||
|
||||
def make_resolver(connector=None, opc_server=None, pooled=None, ttl_s=60.0):
|
||||
def make_resolver(connector=None, pooled=None, ttl_s=60.0):
|
||||
pool = MagicMock()
|
||||
pool.config_for = MagicMock(return_value=pooled)
|
||||
mongo = MagicMock()
|
||||
|
||||
async def find_one(collection, _filters, _metadata):
|
||||
return {CONNECTORS: connector, OPC_SERVERS: opc_server}[collection]
|
||||
|
||||
mongo.find_one = AsyncMock(side_effect=find_one)
|
||||
mongo.aggregate = AsyncMock(return_value=[connector] if connector is not None else [])
|
||||
return ServerConfigResolver(pool, mongo, MagicMock(), ttl_s=ttl_s)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_command_connection_overrides_every_other_source():
|
||||
resolver = make_resolver(connector=CONNECTOR)
|
||||
|
||||
cfg = await resolver.resolve('1', {'url': 'opc.tcp://h:1', 'server_uri': 'urn:h'})
|
||||
|
||||
assert cfg == {'url': 'opc.tcp://h:1', 'server_uri': 'urn:h', 'use_security': False}
|
||||
resolver.mongo.find_one.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_live_pooled_connection_wins_over_mongo():
|
||||
"""Writes reuse exactly the settings the read path opened the connection with."""
|
||||
pooled = {'url': 'opc.tcp://read:1', 'server_uri': 'urn:read', 'use_security': True}
|
||||
pooled = {'url': 'opc.tcp://read:1', 'security': CONFIG['security'], 'certificate': None}
|
||||
resolver = make_resolver(connector=CONNECTOR, pooled=pooled)
|
||||
|
||||
assert await resolver.resolve('1', {}) == pooled
|
||||
resolver.pool.config_for.assert_called_once_with('1')
|
||||
resolver.mongo.find_one.assert_not_awaited()
|
||||
assert await resolver.resolve('kepserver') == pooled
|
||||
resolver.pool.config_for.assert_called_once_with('kepserver')
|
||||
resolver.mongo.aggregate.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_tag_database_connector_first():
|
||||
resolver = make_resolver(connector=CONNECTOR, opc_server=OPC_SERVER)
|
||||
async def test_connector_with_its_certificate_in_the_slot_format():
|
||||
resolver = make_resolver(connector=CONNECTOR)
|
||||
|
||||
cfg = await resolver.resolve('1', {})
|
||||
|
||||
assert cfg == {
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'server_uri': 'urn:kepserver',
|
||||
'use_security': True,
|
||||
}
|
||||
collection, filters, _ = resolver.mongo.find_one.await_args_list[0].args
|
||||
assert await resolver.resolve('kepserver') == CONFIG
|
||||
collection, pipeline, _ = resolver.mongo.aggregate.await_args.args
|
||||
assert collection == CONNECTORS
|
||||
# connector_id is an int in the tag database; the command carries a string.
|
||||
assert filters == {'connector_id': {'$in': ['1', 1]}, 'type': 'opc_ua', 'status': 'Active'}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_opc_servers_fallback_without_tag_database():
|
||||
resolver = make_resolver(connector=None, opc_server=OPC_SERVER)
|
||||
|
||||
cfg = await resolver.resolve('1', {})
|
||||
|
||||
# Same rule as the orchestrator's read slot: a cert_path means security.
|
||||
assert cfg == {
|
||||
'url': 'opc.tcp://legacy:49320',
|
||||
'server_uri': 'urn:legacy',
|
||||
'use_security': True,
|
||||
assert pipeline[0] == {
|
||||
'$match': {'connector_id': 'kepserver', 'type': 'opc_ua', 'status': 'Active'}
|
||||
}
|
||||
collection, filters, _ = resolver.mongo.find_one.await_args_list[1].args
|
||||
assert collection == OPC_SERVERS
|
||||
assert filters == {'id': {'$in': ['1', 1]}}
|
||||
# The same rule as the orchestrator: only an active client certificate (D5)
|
||||
[lookup] = [stage['$lookup'] for stage in pipeline if '$lookup' in stage]
|
||||
assert lookup['from'] == 'opc_certificates'
|
||||
assert lookup['pipeline'][0]['$match']['status'] == 'active'
|
||||
assert lookup['pipeline'][0]['$match']['kind'] == 'client'
|
||||
resolver.logger.warning.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_opc_server_without_cert_path_is_insecure():
|
||||
resolver = make_resolver(opc_server={**OPC_SERVER, 'cert_path': ''})
|
||||
async def test_connector_without_certificate_id_connects_without_security():
|
||||
connector = {
|
||||
**CONNECTOR,
|
||||
'security': {'mode': 'None', 'policy': 'None', 'client_certificate_id': None},
|
||||
'certificates': [],
|
||||
}
|
||||
resolver = make_resolver(connector=connector)
|
||||
|
||||
assert (await resolver.resolve('1', {}))['use_security'] is False
|
||||
cfg = await resolver.resolve('kepserver')
|
||||
|
||||
assert cfg['certificate'] is None
|
||||
assert cfg['security'] == {'mode': 'None', 'policy': 'None'}
|
||||
resolver.logger.warning.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_non_numeric_server_id_is_looked_up_as_is():
|
||||
resolver = make_resolver(opc_server={**OPC_SERVER, 'id': 'kep-a'})
|
||||
async def test_certificate_id_without_an_active_client_certificate_warns():
|
||||
resolver = make_resolver(connector={**CONNECTOR, 'certificates': []})
|
||||
|
||||
await resolver.resolve('kep-a', {})
|
||||
cfg = await resolver.resolve('kepserver')
|
||||
|
||||
_, filters, _ = resolver.mongo.find_one.await_args_list[0].args
|
||||
assert filters['connector_id'] == {'$in': ['kep-a']}
|
||||
assert cfg == {**CONFIG, 'certificate': None}
|
||||
assert 'client_certificate_id 1' in resolver.logger.warning.call_args.args[0]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
'connector',
|
||||
[
|
||||
pytest.param({**CONNECTOR, 'url': ''}, id='no-url'),
|
||||
pytest.param({**CONNECTOR, 'security': {'mode': 'Sign'}}, id='no-policy'),
|
||||
pytest.param({k: v for k, v in CONNECTOR.items() if k != 'security'}, id='no-security'),
|
||||
],
|
||||
)
|
||||
async def test_incomplete_connectors_are_not_found(connector):
|
||||
resolver = make_resolver(connector=connector)
|
||||
|
||||
assert await resolver.resolve('kepserver') is None
|
||||
resolver.logger.warning.assert_called_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -113,39 +118,32 @@ async def test_unknown_server_is_cached_briefly():
|
||||
resolver.clock = MagicMock()
|
||||
resolver.clock.elapsed.side_effect = [0.0, 5.0, 11.0]
|
||||
|
||||
assert await resolver.resolve('9', {}) is None
|
||||
assert await resolver.resolve('9', {}) is None # within missing_ttl_s: no lookup
|
||||
assert await resolver.resolve('9', {}) is None # expired: looked up again
|
||||
assert resolver.mongo.find_one.await_count == 4 # 2 lookups (connectors + opc-servers)
|
||||
assert await resolver.resolve('9') is None
|
||||
assert await resolver.resolve('9') is None # within missing_ttl_s: no lookup
|
||||
assert await resolver.resolve('9') is None # expired: looked up again
|
||||
assert resolver.mongo.aggregate.await_count == 2
|
||||
assert resolver.logger.warning.call_count == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_documents_without_url_are_ignored():
|
||||
resolver = make_resolver(connector={**CONNECTOR, 'url': ''}, opc_server=OPC_SERVER)
|
||||
|
||||
assert (await resolver.resolve('1', {}))['url'] == 'opc.tcp://legacy:49320'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lookups_are_cached_for_the_ttl():
|
||||
resolver = make_resolver(connector=CONNECTOR, ttl_s=60.0)
|
||||
resolver.clock = MagicMock()
|
||||
resolver.clock.elapsed.side_effect = [0.0, 30.0, 61.0]
|
||||
|
||||
first = await resolver.resolve('1', {})
|
||||
first = await resolver.resolve('kepserver')
|
||||
first['url'] = 'mutated by the caller'
|
||||
second = await resolver.resolve('1', {}) # within the TTL: cached, and a copy
|
||||
await resolver.resolve('1', {}) # expired: looked up again
|
||||
second = await resolver.resolve('kepserver') # within the TTL: cached, and a copy
|
||||
await resolver.resolve('kepserver') # expired: looked up again
|
||||
|
||||
assert second['url'] == 'opc.tcp://kepserver:49320'
|
||||
assert resolver.mongo.find_one.await_count == 2
|
||||
assert resolver.mongo.aggregate.await_count == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mongo_errors_propagate_to_the_caller():
|
||||
resolver = make_resolver()
|
||||
resolver.mongo.find_one = AsyncMock(side_effect=RuntimeError('mongo down'))
|
||||
resolver.mongo.aggregate = AsyncMock(side_effect=RuntimeError('mongo down'))
|
||||
|
||||
with pytest.raises(RuntimeError, match='mongo down'):
|
||||
await resolver.resolve('1', {})
|
||||
await resolver.resolve('kepserver')
|
||||
|
||||
@@ -7,7 +7,12 @@ from asyncua import ua
|
||||
import ingestor.metrics as metrics
|
||||
from ingestor.engines.write import WriteEngine
|
||||
|
||||
SERVER_CFG = {'url': 'opc.tcp://kepserver:49320', 'server_uri': 'urn:kep', 'use_security': True}
|
||||
SERVER_CFG = {
|
||||
'server_id': 'srv1',
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'security': {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'},
|
||||
'certificate': None,
|
||||
}
|
||||
|
||||
|
||||
def make_resolver(cfg=SERVER_CFG):
|
||||
@@ -279,7 +284,7 @@ async def test_write_uses_the_resolved_server_connection():
|
||||
|
||||
await eng.write(base_cmd())
|
||||
|
||||
eng.servers.resolve.assert_awaited_once_with('srv1', base_cmd())
|
||||
eng.servers.resolve.assert_awaited_once_with('srv1')
|
||||
eng.pool.get.assert_awaited_once_with('srv1', SERVER_CFG)
|
||||
|
||||
|
||||
@@ -310,22 +315,22 @@ async def test_write_server_lookup_failure_records_error():
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_without_resolver_uses_the_command_connection():
|
||||
eng = make_engine(servers=None)
|
||||
async def test_write_ignores_a_connection_carried_by_the_command():
|
||||
"""The server is always resolved by server_id (SIENTIAPDE-1974): connection fields
|
||||
in the command are not an override"""
|
||||
eng = make_engine()
|
||||
eng.pool.get.return_value = make_client()
|
||||
|
||||
await eng.write(base_cmd(url='opc.tcp://h:4840', server_uri='urn:h', use_security=True))
|
||||
await eng.write(base_cmd(url='opc.tcp://h:4840'))
|
||||
|
||||
eng.pool.get.assert_awaited_once_with(
|
||||
'srv1', {'url': 'opc.tcp://h:4840', 'use_security': True, 'server_uri': 'urn:h'}
|
||||
)
|
||||
eng.pool.get.assert_awaited_once_with('srv1', SERVER_CFG)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_without_resolver_or_url_is_server_not_found():
|
||||
async def test_write_without_resolver_is_server_not_found():
|
||||
eng = make_engine(servers=None)
|
||||
|
||||
result = await eng.write(base_cmd())
|
||||
result = await eng.write(base_cmd(url='opc.tcp://h:4840'))
|
||||
|
||||
assert result.opc_status_code == 'server_not_found'
|
||||
eng.pool.get.assert_not_awaited()
|
||||
|
||||
@@ -67,26 +67,3 @@ async def test_publish_reading_emits_tag_value_metric_per_name():
|
||||
value=42,
|
||||
tags={'pod_id': dm.pod_id, 'tag_name': 'Counter', 'model_id': '1'},
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_publish_reading_without_model_id_keeps_legacy_doc():
|
||||
"""Names converted from a slot v1 carry no model_id: the document must be the
|
||||
same as the legacy OpcManager one (SIENTIAPDE-1974 NF2)."""
|
||||
sink, dm = make_sink()
|
||||
names = [{'collection': 'raw_x', 'tag_name': 'Counter'}]
|
||||
|
||||
await sink.publish_reading(
|
||||
tag='ns=2;i=2', value=42, timestamp='2026-07-13 00:00:00+0000', names=names
|
||||
)
|
||||
|
||||
collection, doc = dm.publish.await_args_list[0].args
|
||||
assert collection == 'raw_x'
|
||||
assert doc == {
|
||||
'tag': 'ns=2;i=2',
|
||||
'name': 'Counter',
|
||||
'timestamp': '2026-07-13 00:00:00+0000',
|
||||
'value': 42,
|
||||
}
|
||||
_, kwargs = sink.emit_metric.await_args
|
||||
assert kwargs['tags']['model_id'] == ''
|
||||
|
||||
@@ -1,11 +1,24 @@
|
||||
from ingestor.io.slots import parse_slot
|
||||
import pytest
|
||||
|
||||
from ingestor.io.slots import CONNECTION_FIELDS, parse_slot
|
||||
|
||||
SECURITY = {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}
|
||||
CERTIFICATE = {
|
||||
'certificate_id': 1,
|
||||
'application_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/p/cert.pem'},
|
||||
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/p/key.pem'},
|
||||
}
|
||||
NAME = {'collection': 'c', 'tag_name': 't', 'model_name': 'm', 'model_id': '1'}
|
||||
|
||||
RAW = {
|
||||
'server1': {
|
||||
'server_id': '1',
|
||||
'name': 'server1',
|
||||
'url': 'opc.tcp://h:4840',
|
||||
'use_security': True,
|
||||
'subscription_period_ms': 1000,
|
||||
'security': SECURITY,
|
||||
'certificate': CERTIFICATE,
|
||||
'tags': {
|
||||
'ns=2;i=2': {
|
||||
'frequency': '15000',
|
||||
@@ -30,6 +43,17 @@ RAW = {
|
||||
}
|
||||
|
||||
|
||||
def _server(**fields):
|
||||
server = {
|
||||
'server_id': '1',
|
||||
'url': 'u',
|
||||
'security': {'mode': 'None', 'policy': 'None'},
|
||||
'certificate': None,
|
||||
'tags': {'ns=2;i=1': {'frequency': '1000', 'names': [NAME]}},
|
||||
}
|
||||
return {'s': {**server, **fields}}
|
||||
|
||||
|
||||
def test_parse_slot_multi_name():
|
||||
slot = parse_slot(RAW)
|
||||
tag = slot['server1']['tags']['ns=2;i=2']
|
||||
@@ -37,127 +61,71 @@ def test_parse_slot_multi_name():
|
||||
assert tag['active_read']['cycle_multiple'] == 3
|
||||
|
||||
|
||||
def test_parse_slot_defaults_active_read_disabled():
|
||||
raw = {
|
||||
's': {
|
||||
'server_id': '1',
|
||||
'url': 'u',
|
||||
'tags': {
|
||||
'ns=2;i=1': {
|
||||
'frequency': '1000',
|
||||
'names': [
|
||||
{'collection': 'c', 'tag_name': 't', 'model_name': 'm', 'model_id': '1'}
|
||||
],
|
||||
}
|
||||
},
|
||||
}
|
||||
def test_parse_slot_keeps_security_and_certificate():
|
||||
server = parse_slot(RAW)['server1']
|
||||
|
||||
assert server['security'] == SECURITY
|
||||
assert server['certificate'] == CERTIFICATE
|
||||
assert {field: server[field] for field in CONNECTION_FIELDS} == {
|
||||
'url': 'opc.tcp://h:4840',
|
||||
'subscription_period_ms': 1000,
|
||||
'security': SECURITY,
|
||||
'certificate': CERTIFICATE,
|
||||
}
|
||||
tag = parse_slot(raw)['s']['tags']['ns=2;i=1']
|
||||
assert tag['active_read'] == {'enabled': False}
|
||||
|
||||
|
||||
def test_parse_slot_rejects_tag_without_names():
|
||||
raw = {'s': {'server_id': '1', 'url': 'u', 'tags': {'ns=2;i=1': {'frequency': '1000'}}}}
|
||||
import pytest
|
||||
|
||||
with pytest.raises((KeyError, ValueError)):
|
||||
parse_slot(raw)
|
||||
|
||||
|
||||
RAW_V1 = {
|
||||
'kepserver': {
|
||||
'name': 'kepserver',
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert_path': '/app/certs/opc-ingestor-cert.pem',
|
||||
'private_key_path': '/app/certs/opc-ingestor-key.pem',
|
||||
'server_cert_path': None,
|
||||
'subscription_period_ms': 15000.0,
|
||||
'tags': {
|
||||
'ns=2;s=A.TargetSin': {
|
||||
'aggr_func': 'lts',
|
||||
'frequency': 30000,
|
||||
'server_id': '1',
|
||||
'tag_address': 'ns=2;s=A.TargetSin',
|
||||
'tag_name': 'TargetSin',
|
||||
'topics': ['raw_scouter-sin-approx', 'raw_other-model'],
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def test_parse_slot_converts_v1_topics_to_names():
|
||||
"""Slot v1 (current Orchestrator) is read as v2: one name per topic, no model_id,
|
||||
so the documents stay identical to the legacy ones (SIENTIAPDE-1974 G1)."""
|
||||
server = parse_slot(RAW_V1)['kepserver']
|
||||
tag = server['tags']['ns=2;s=A.TargetSin']
|
||||
|
||||
assert tag['names'] == [
|
||||
{'collection': 'raw_scouter-sin-approx', 'tag_name': 'TargetSin'},
|
||||
{'collection': 'raw_other-model', 'tag_name': 'TargetSin'},
|
||||
]
|
||||
assert tag['frequency'] == 30000
|
||||
assert tag['active_read'] == {'enabled': False}
|
||||
|
||||
|
||||
def test_parse_slot_v1_cert_path_means_security_but_path_is_dropped():
|
||||
server = parse_slot(RAW_V1)['kepserver']
|
||||
|
||||
assert server['use_security'] is True
|
||||
assert server['name'] == 'kepserver'
|
||||
assert server['server_id'] == '1'
|
||||
assert server['subscription_period_ms'] == 15000.0
|
||||
# the certificate always comes from MinIO now: no file path survives the parse
|
||||
assert 'cert_path' not in server
|
||||
assert 'private_key_path' not in server
|
||||
|
||||
|
||||
def test_parse_slot_v1_without_cert_path_is_insecure():
|
||||
raw = {
|
||||
's': {
|
||||
'server_id': '2',
|
||||
'url': 'u',
|
||||
'cert_path': None,
|
||||
'tags': {'ns=2;i=1': {'tag_name': 't', 'frequency': '1000', 'topics': ['raw_c']}},
|
||||
}
|
||||
}
|
||||
def test_parse_slot_without_certificate_key_means_no_certificate():
|
||||
raw = _server()
|
||||
del raw['s']['certificate']
|
||||
|
||||
server = parse_slot(raw)['s']
|
||||
|
||||
assert server['use_security'] is False
|
||||
assert server['certificate'] is None
|
||||
assert server['name'] == 's' # falls back to the slot key
|
||||
|
||||
|
||||
def test_parse_slot_v2_explicit_use_security_wins():
|
||||
raw = {
|
||||
's': {
|
||||
'server_id': '1',
|
||||
'url': 'u',
|
||||
'use_security': False,
|
||||
'cert_path': '/legacy/path',
|
||||
'tags': {
|
||||
'ns=2;i=1': {
|
||||
'frequency': '1000',
|
||||
'names': [{'collection': 'c', 'tag_name': 't', 'model_id': '1'}],
|
||||
def test_parse_slot_defaults_active_read_disabled():
|
||||
tag = parse_slot(_server())['s']['tags']['ns=2;i=1']
|
||||
assert tag['active_read'] == {'enabled': False}
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'raw',
|
||||
[
|
||||
pytest.param(_server(tags={'ns=2;i=1': {'frequency': '1000'}}), id='tag-without-names'),
|
||||
pytest.param(
|
||||
_server(tags={'ns=2;i=1': {'frequency': '1', 'names': [{'collection': 'c'}]}}),
|
||||
id='name-without-tag-name',
|
||||
),
|
||||
pytest.param(
|
||||
_server(
|
||||
tags={
|
||||
'ns=2;i=1': {
|
||||
'frequency': '1',
|
||||
'names': [{'collection': 'c', 'tag_name': 't'}],
|
||||
}
|
||||
}
|
||||
),
|
||||
id='name-without-model-id',
|
||||
),
|
||||
pytest.param(_server(security=None), id='no-security'),
|
||||
# Slot v1 (an Orchestrator before SIENTIAPDE-1974): no longer accepted
|
||||
pytest.param(
|
||||
{
|
||||
'kepserver': {
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert_path': '/app/certs/opc-ingestor-cert.pem',
|
||||
'tags': {
|
||||
'ns=2;s=A': {'tag_name': 'A', 'frequency': 30000, 'topics': ['raw_x']}
|
||||
},
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
assert parse_slot(raw)['s']['use_security'] is False
|
||||
|
||||
|
||||
def test_parse_slot_rejects_v1_tag_without_tag_name():
|
||||
raw = {
|
||||
's': {
|
||||
'server_id': '1',
|
||||
'url': 'u',
|
||||
'tags': {'ns=2;i=1': {'frequency': '1', 'topics': ['c']}},
|
||||
}
|
||||
}
|
||||
import pytest
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
id='slot-v1',
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_parse_slot_rejects_invalid_slots(raw):
|
||||
with pytest.raises((KeyError, ValueError)):
|
||||
parse_slot(raw)
|
||||
|
||||
@@ -9,7 +9,6 @@ from ingestor.config import GatewayConfig
|
||||
from ingestor.connection.pool import OpcConnectionPool
|
||||
from ingestor.io.slots import parse_slot
|
||||
from ingestor.managers.ingestor_manager import IngestorManager
|
||||
from ingestor.managers.opc_manager import OpcManager
|
||||
from ingestor.managers.server_reader import ServerReader
|
||||
|
||||
metadata = {
|
||||
@@ -22,17 +21,27 @@ metadata = {
|
||||
}
|
||||
|
||||
|
||||
def raw_slot_v1(tag_name='Counter', address='ns=2;i=2'):
|
||||
"""A slot as the current (v1) Orchestrator writes it to Redis."""
|
||||
SECURITY = {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}
|
||||
CERTIFICATE = {
|
||||
'certificate_id': 1,
|
||||
'application_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/p/cert.pem'},
|
||||
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/p/key.pem'},
|
||||
}
|
||||
|
||||
|
||||
def raw_slot(tag_name='Counter', address='ns=2;i=2'):
|
||||
"""A slot v2 as the Orchestrator writes it to Redis (contract C1)."""
|
||||
name = {'collection': 'raw_a', 'tag_name': tag_name, 'model_id': '1', 'model_name': 'm'}
|
||||
return {
|
||||
'server1': {
|
||||
'name': 'server1',
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://localhost:4840',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert_path': '/app/certs/opc-ingestor-cert.pem',
|
||||
'subscription_period_ms': 500.0,
|
||||
'tags': {address: {'tag_name': tag_name, 'frequency': '1000', 'topics': ['raw_a']}},
|
||||
'security': SECURITY,
|
||||
'certificate': CERTIFICATE,
|
||||
'tags': {address: {'frequency': '1000', 'names': [name]}},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -124,7 +133,6 @@ def test___init__(
|
||||
# Read path (SIENTIAPDE-1974): pool + read sink built here, nothing connected yet.
|
||||
assert isinstance(ingestor.pool, OpcConnectionPool)
|
||||
assert ingestor.pool.cert_provider is ingestor.cert_provider
|
||||
assert ingestor.cert_provider.mongo is data_manager_mock.return_value.mongo_repository
|
||||
assert ingestor.sink.data_manager is data_manager_mock.return_value
|
||||
assert ingestor.sink.postgres is None
|
||||
|
||||
@@ -143,10 +151,9 @@ def test___init___cert_provider_uses_gateway_config(resource_manager_mock, data_
|
||||
notification_handler=MagicMock(),
|
||||
metadata=metadata['metadata'],
|
||||
metrics_controller=MagicMock(),
|
||||
gateway_config=GatewayConfig(cert_bucket='my-bucket', cert_cache_dir='certs-cache'),
|
||||
gateway_config=GatewayConfig(cert_cache_dir='certs-cache'),
|
||||
)
|
||||
|
||||
assert ingestor.cert_provider.bucket == 'my-bucket'
|
||||
assert ingestor.cert_provider.cache_dir == 'certs-cache'
|
||||
|
||||
|
||||
@@ -157,9 +164,9 @@ async def test_initialize_opc_from_config(server_reader, ingestor_manager):
|
||||
'name': 'server1',
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://localhost:4840',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'use_security': True,
|
||||
'subscription_period_ms': 1000,
|
||||
'security': SECURITY,
|
||||
'certificate': CERTIFICATE,
|
||||
}
|
||||
|
||||
server_reader.return_value = MagicMock(connect=AsyncMock())
|
||||
@@ -191,9 +198,9 @@ async def test_initialize_opc_from_config_exception(
|
||||
'name': 'server1',
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://localhost:4840',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'use_security': True,
|
||||
'subscription_period_ms': 1000,
|
||||
'security': SECURITY,
|
||||
'certificate': CERTIFICATE,
|
||||
}
|
||||
|
||||
ingestor_manager.logger.error = MagicMock()
|
||||
@@ -483,13 +490,13 @@ async def test_get_number_of_slots_empty(metrics, ingestor_manager):
|
||||
@mark.asyncio
|
||||
async def test_get_slot_leases_1_success(ingestor_manager):
|
||||
ingestor_manager.resource_manager.lease_tag = AsyncMock(return_value=True)
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(return_value=raw_slot_v1())
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(return_value=raw_slot())
|
||||
|
||||
ingestor_manager.number_of_slots = 1
|
||||
result = await ingestor_manager.get_slot_leases()
|
||||
|
||||
# Stored already normalized to slot v2 (v1 converted by parse_slot).
|
||||
assert result == {'1': parse_slot(raw_slot_v1())}
|
||||
assert result == {'1': parse_slot(raw_slot())}
|
||||
assert ingestor_manager.managed_tags == result
|
||||
|
||||
|
||||
@@ -497,22 +504,22 @@ async def test_get_slot_leases_1_success(ingestor_manager):
|
||||
async def test_get_slot_leases_2_success(ingestor_manager):
|
||||
ingestor_manager.resource_manager.lease_tag = AsyncMock(side_effect=[True, True])
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
|
||||
side_effect=[raw_slot_v1('Counter'), raw_slot_v1('Ramp', 'ns=2;i=3')]
|
||||
side_effect=[raw_slot('Counter'), raw_slot('Ramp', 'ns=2;i=3')]
|
||||
)
|
||||
|
||||
ingestor_manager.number_of_slots = 2
|
||||
result = await ingestor_manager.get_slot_leases(max_slots=2)
|
||||
|
||||
assert result == {
|
||||
'1': parse_slot(raw_slot_v1('Counter')),
|
||||
'2': parse_slot(raw_slot_v1('Ramp', 'ns=2;i=3')),
|
||||
'1': parse_slot(raw_slot('Counter')),
|
||||
'2': parse_slot(raw_slot('Ramp', 'ns=2;i=3')),
|
||||
}
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_get_slot_leases_skips_malformed_slot(ingestor_manager):
|
||||
ingestor_manager.resource_manager.lease_tag = AsyncMock(return_value=True)
|
||||
# tag without names/topics/frequency
|
||||
# tag without names/frequency
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
|
||||
return_value={'server1': {'server_id': '1', 'url': 'u', 'tags': {'ns=2;i=2': {}}}}
|
||||
)
|
||||
@@ -624,8 +631,8 @@ async def test_unsubscribe_slot_reader_failure_does_not_raise(ingestor_manager,
|
||||
|
||||
@mark.asyncio
|
||||
async def test_unsubscribe_slot_with_real_readers(ingestor_manager):
|
||||
"""Contract: both real readers keep `subscriptions` keyed by slot, and
|
||||
unsubscribe_slot closes the slot's subscription on each of them"""
|
||||
"""Contract: the real reader keeps `subscriptions` keyed by slot, and
|
||||
unsubscribe_slot closes the slot's subscription"""
|
||||
v2_subscription = AsyncMock()
|
||||
client = MagicMock(create_subscription=AsyncMock(return_value=v2_subscription))
|
||||
pool = MagicMock(get=AsyncMock(return_value=client), evict=AsyncMock())
|
||||
@@ -646,35 +653,13 @@ async def test_unsubscribe_slot_with_real_readers(ingestor_manager):
|
||||
server_reader.engine.emit_metric = AsyncMock()
|
||||
await server_reader.create_subscription('slot1')
|
||||
|
||||
with patch('ingestor.managers.opc_manager.metrics'):
|
||||
legacy_reader = OpcManager(
|
||||
name='legacy',
|
||||
url='opc.tcp://legacy:4840',
|
||||
data_manager=AsyncMock(),
|
||||
subscription_period_ms=1000,
|
||||
logger=MagicMock(),
|
||||
server_uri='opc.tcp://legacy:4840',
|
||||
notification_handler=MagicMock(),
|
||||
metadata={},
|
||||
metrics_controller=AsyncMock(),
|
||||
)
|
||||
legacy_reader.emit_metric = AsyncMock()
|
||||
legacy_subscription = AsyncMock()
|
||||
legacy_reader.client = MagicMock(
|
||||
create_subscription=AsyncMock(return_value=legacy_subscription)
|
||||
)
|
||||
await legacy_reader.create_subscription('slot1')
|
||||
|
||||
assert 'slot1' in server_reader.subscriptions
|
||||
assert 'slot1' in legacy_reader.subscriptions
|
||||
ingestor_manager.opc_managers = {'kepserver': server_reader, 'legacy': legacy_reader}
|
||||
ingestor_manager.opc_managers = {'kepserver': server_reader}
|
||||
|
||||
await ingestor_manager.unsubscribe_slot('slot1')
|
||||
|
||||
v2_subscription.delete.assert_awaited_once()
|
||||
legacy_subscription.delete.assert_awaited_once()
|
||||
assert 'slot1' not in server_reader.subscriptions
|
||||
assert 'slot1' not in legacy_reader.subscriptions
|
||||
pool.pin.assert_called_once_with('1')
|
||||
pool.unpin.assert_called_once_with('1')
|
||||
|
||||
@@ -683,13 +668,13 @@ async def test_unsubscribe_slot_with_real_readers(ingestor_manager):
|
||||
async def test_update_slot_config_unsubscribes_removed_slot(ingestor_manager, reader_double):
|
||||
"""A slot deleted from Redis is unsubscribed right away, not left to the main loop"""
|
||||
ingestor_manager.managed_tags = {
|
||||
'slot1': parse_slot(raw_slot_v1('Same')),
|
||||
'slot2': parse_slot(raw_slot_v1('Same')),
|
||||
'slot1': parse_slot(raw_slot('Same')),
|
||||
'slot2': parse_slot(raw_slot('Same')),
|
||||
}
|
||||
reader = reader_double('slot1', 'slot2')
|
||||
ingestor_manager.opc_managers = {'test_server': reader}
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
|
||||
side_effect=lambda slot: raw_slot_v1('Same') if slot == 'slot1' else None
|
||||
side_effect=lambda slot: raw_slot('Same') if slot == 'slot1' else None
|
||||
)
|
||||
ingestor_manager.resource_manager.renew_tag_lease = AsyncMock()
|
||||
|
||||
@@ -703,20 +688,20 @@ async def test_update_slot_config_unsubscribes_removed_slot(ingestor_manager, re
|
||||
@mark.asyncio
|
||||
async def test_update_slot_config(ingestor_manager):
|
||||
ingestor_manager.managed_tags = {
|
||||
'slot1': parse_slot(raw_slot_v1('Old')),
|
||||
'slot2': parse_slot(raw_slot_v1('Same')),
|
||||
'slot3': parse_slot(raw_slot_v1('Old')),
|
||||
'slot1': parse_slot(raw_slot('Old')),
|
||||
'slot2': parse_slot(raw_slot('Same')),
|
||||
'slot3': parse_slot(raw_slot('Old')),
|
||||
}
|
||||
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
|
||||
side_effect=[raw_slot_v1('Updated'), raw_slot_v1('Same'), None]
|
||||
side_effect=[raw_slot('Updated'), raw_slot('Same'), None]
|
||||
)
|
||||
ingestor_manager.resource_manager.renew_tag_lease = AsyncMock()
|
||||
|
||||
await ingestor_manager.update_slot_config()
|
||||
|
||||
assert ingestor_manager.managed_tags['slot1'] == parse_slot(raw_slot_v1('Updated'))
|
||||
assert ingestor_manager.managed_tags['slot2'] == parse_slot(raw_slot_v1('Same'))
|
||||
assert ingestor_manager.managed_tags['slot1'] == parse_slot(raw_slot('Updated'))
|
||||
assert ingestor_manager.managed_tags['slot2'] == parse_slot(raw_slot('Same'))
|
||||
assert 'slot3' not in ingestor_manager.managed_tags
|
||||
|
||||
ingestor_manager.resource_manager.renew_tag_lease.assert_any_call('slot1')
|
||||
@@ -783,8 +768,8 @@ async def test_manage_server_create_subscription_failure(ingestor_manager):
|
||||
|
||||
@mark.asyncio
|
||||
async def test_manage_server_subscribe_and_unsubscribe_failure(ingestor_manager):
|
||||
"""subscribe() fails and so does the cleanup unsubscribe (legacy reader, session
|
||||
gone): manage_server reports the failure and nothing escapes to the main loop"""
|
||||
"""subscribe() fails and so does the cleanup unsubscribe (session gone):
|
||||
manage_server reports the failure and nothing escapes to the main loop"""
|
||||
reader = AsyncMock()
|
||||
reader.subscriptions = {}
|
||||
reader.subscribe.side_effect = Exception('Subscription error')
|
||||
@@ -886,7 +871,7 @@ async def test_subscribe_to_tags(ingestor_manager):
|
||||
|
||||
@mark.asyncio
|
||||
async def test_update_slot_config_keeps_last_good_config_when_malformed(ingestor_manager):
|
||||
good = parse_slot(raw_slot_v1('Counter'))
|
||||
good = parse_slot(raw_slot('Counter'))
|
||||
ingestor_manager.managed_tags = {'slot1': good}
|
||||
ingestor_manager.resource_manager.get_tag_slot = AsyncMock(
|
||||
return_value={'server1': {'server_id': '1', 'url': 'u', 'tags': {'ns=2;i=2': {}}}}
|
||||
@@ -922,7 +907,7 @@ def test___init___ignores_malformed_write_env(
|
||||
"""A malformed write/gRPC-only env var must not stop the read path (review #1)."""
|
||||
monkeypatch.setenv('GRPC_PORT', '')
|
||||
monkeypatch.setenv('WRITE_MIN_PAUSE_S', '0,5')
|
||||
monkeypatch.setenv('CERT_BUCKET', 'streamlit-connectors')
|
||||
monkeypatch.setenv('CERT_CACHE_DIR', 'certs-cache')
|
||||
|
||||
ingestor = IngestorManager(
|
||||
redis_data={'host': 'localhost', 'port': 6379},
|
||||
@@ -937,147 +922,4 @@ def test___init___ignores_malformed_write_env(
|
||||
metrics_controller=MagicMock(),
|
||||
)
|
||||
|
||||
assert ingestor.cert_provider.bucket == 'streamlit-connectors'
|
||||
|
||||
|
||||
def test_parse_slot_config_warns_once_about_server_cert_path(ingestor_manager):
|
||||
raw = raw_slot_v1()
|
||||
raw['server1']['server_cert_path'] = '/app/certs/kepserver.der'
|
||||
|
||||
ingestor_manager.parse_slot_config('1', raw)
|
||||
ingestor_manager.parse_slot_config('1', raw) # every loop refreshes the slot
|
||||
|
||||
warnings = [
|
||||
c for c in ingestor_manager.logger.warning.call_args_list if 'server_cert_path' in str(c)
|
||||
]
|
||||
assert len(warnings) == 1
|
||||
|
||||
|
||||
# --- READ_ENGINE=legacy: plan B with the untouched rc3 OpcManager -------------------
|
||||
|
||||
|
||||
def make_manager(read_engine=None):
|
||||
with (
|
||||
patch('ingestor.managers.ingestor_manager.DataManager'),
|
||||
patch('ingestor.managers.ingestor_manager.ResourceManager'),
|
||||
):
|
||||
manager = IngestorManager(
|
||||
redis_data={'host': 'localhost', 'port': 6379},
|
||||
lease_ttl=60,
|
||||
heartbeat_ttl=60,
|
||||
poll_interval=5,
|
||||
mongo_connection_string='mongodb://localhost:27017',
|
||||
mongo_database='sientia',
|
||||
logger=MagicMock(),
|
||||
notification_handler=MagicMock(),
|
||||
metadata=metadata['metadata'],
|
||||
metrics_controller=MagicMock(),
|
||||
read_engine=read_engine,
|
||||
)
|
||||
manager.send_notification = MagicMock()
|
||||
manager.emit_metric = AsyncMock()
|
||||
return manager
|
||||
|
||||
|
||||
def test_read_engine_defaults_to_v2(monkeypatch):
|
||||
monkeypatch.delenv('READ_ENGINE', raising=False)
|
||||
|
||||
assert make_manager().read_engine == 'v2'
|
||||
|
||||
|
||||
def test_read_engine_from_env(monkeypatch):
|
||||
monkeypatch.setenv('READ_ENGINE', 'Legacy ')
|
||||
|
||||
assert make_manager().read_engine == 'legacy'
|
||||
|
||||
|
||||
def test_invalid_read_engine_falls_back_to_v2(monkeypatch):
|
||||
monkeypatch.setenv('READ_ENGINE', 'v3')
|
||||
|
||||
manager = make_manager()
|
||||
|
||||
assert manager.read_engine == 'v2'
|
||||
assert any('Invalid READ_ENGINE' in str(c) for c in manager.logger.warning.call_args_list)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('ingestor.managers.ingestor_manager.ServerReader')
|
||||
@patch('ingestor.managers.ingestor_manager.OpcManager')
|
||||
async def test_legacy_initialize_builds_rc3_opc_manager(opc_manager, server_reader):
|
||||
manager = make_manager('legacy')
|
||||
server_config = {
|
||||
'name': 'server1',
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://localhost:4840',
|
||||
'subscription_period_ms': 1000,
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert_path': '/app/certs/opc-ingestor-cert.pem',
|
||||
'private_key_path': '/app/certs/opc-ingestor-key.pem',
|
||||
'server_cert_path': None,
|
||||
}
|
||||
opc_manager.return_value = MagicMock(connect=AsyncMock())
|
||||
|
||||
result = await manager.initialize_opc_from_config(server_config)
|
||||
|
||||
server_reader.assert_not_called()
|
||||
opc_manager.assert_called_once_with(
|
||||
name='server1',
|
||||
url='opc.tcp://localhost:4840',
|
||||
subscription_period_ms=1000,
|
||||
data_manager=manager.data_manager,
|
||||
logger=manager.logger,
|
||||
server_uri='urn:sientia:opc-ingestor',
|
||||
notification_handler=manager.notification_handler,
|
||||
metadata=metadata['metadata'],
|
||||
cert_path='/app/certs/opc-ingestor-cert.pem',
|
||||
private_key_path='/app/certs/opc-ingestor-key.pem',
|
||||
server_cert_path=None,
|
||||
metrics_controller=manager.metrics_controller,
|
||||
)
|
||||
assert result is opc_manager.return_value
|
||||
result.connect.assert_awaited_once()
|
||||
|
||||
|
||||
def test_legacy_keeps_raw_slot_v1():
|
||||
manager = make_manager('legacy')
|
||||
raw = raw_slot_v1()
|
||||
|
||||
# OpcManager reads tag_name/topics and the file cert paths straight from the slot
|
||||
assert manager.parse_slot_config('1', raw) is raw
|
||||
|
||||
|
||||
def test_legacy_rejects_slot_v2():
|
||||
manager = make_manager('legacy')
|
||||
raw_v2 = {
|
||||
'server1': {
|
||||
'name': 'server1',
|
||||
'server_id': '1',
|
||||
'url': 'opc.tcp://localhost:4840',
|
||||
'use_security': True,
|
||||
'subscription_period_ms': 500,
|
||||
'tags': {
|
||||
'ns=2;i=2': {
|
||||
'frequency': '1000',
|
||||
'names': [{'collection': 'raw_a', 'tag_name': 'Counter', 'model_id': '1'}],
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
assert manager.parse_slot_config('1', raw_v2) is None
|
||||
_, kwargs = manager.send_notification.call_args
|
||||
assert kwargs['notification_id'] == 'OPC_SLOT_CONFIG_ERROR_1'
|
||||
assert 'READ_ENGINE=legacy' in kwargs['message']
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_legacy_remove_server_calls_plain_shutdown():
|
||||
manager = make_manager('legacy')
|
||||
opc_manager = MagicMock(shutdown=AsyncMock())
|
||||
manager.opc_managers = {'server1': opc_manager}
|
||||
manager.managed_tags = {'slot1': {'server1': {}}}
|
||||
|
||||
await manager.remove_server('server1', evict=False)
|
||||
|
||||
opc_manager.shutdown.assert_awaited_once_with()
|
||||
assert manager.opc_managers == {}
|
||||
assert ingestor.cert_provider.cache_dir == 'certs-cache'
|
||||
|
||||
@@ -1,627 +0,0 @@
|
||||
import json
|
||||
from datetime import datetime
|
||||
from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
|
||||
|
||||
import pytest
|
||||
from asyncua.crypto.security_policies import SecurityPolicyBasic256
|
||||
from pytest import fixture, mark
|
||||
from sientia_do.notifications.models import NotificationLevel
|
||||
|
||||
from ingestor.managers.opc_manager import OpcManager
|
||||
|
||||
tags = {
|
||||
'ns=3;i=1001': {
|
||||
'aggregation_function': 'LTS',
|
||||
'frequency': 1000,
|
||||
'max_value': 100,
|
||||
'min_value': 0,
|
||||
'tag_name': 'Counter',
|
||||
},
|
||||
'ns=3;i=1003': {
|
||||
'aggregation_function': 'AVG',
|
||||
'frequency': 1000,
|
||||
'max_value': 100,
|
||||
'min_value': 0,
|
||||
'tag_name': 'Random',
|
||||
},
|
||||
'ns=3;i=1004': {
|
||||
'aggregation_function': 'MDN',
|
||||
'frequency': 1000,
|
||||
'max_value': 100,
|
||||
'min_value': 0,
|
||||
'tag_name': 'Sawtooth',
|
||||
},
|
||||
}
|
||||
|
||||
metadata = {
|
||||
'metadata': {
|
||||
'model_id': 'test_model',
|
||||
'model_name': 'test_model',
|
||||
'workflow_name': 'test_workflow',
|
||||
'schema_name': 'test_schedule',
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@fixture
|
||||
@patch('ingestor.managers.opc_manager.metrics')
|
||||
def raw_opc_manager(mock_metrics):
|
||||
opc_manager = OpcManager(
|
||||
name='TestConnector',
|
||||
url='opc.tcp://localhost:4840',
|
||||
data_manager=AsyncMock(),
|
||||
subscription_period_ms=1000,
|
||||
logger=MagicMock(),
|
||||
server_uri='opc.tcp://localhost:4840',
|
||||
notification_handler=MagicMock(),
|
||||
metadata=metadata['metadata'],
|
||||
metrics_controller=AsyncMock(),
|
||||
)
|
||||
|
||||
opc_manager.emit_metric = AsyncMock()
|
||||
opc_manager.send_notification_async = AsyncMock()
|
||||
opc_manager.send_notification = MagicMock()
|
||||
|
||||
return opc_manager
|
||||
|
||||
|
||||
@fixture
|
||||
def opc_manager(raw_opc_manager):
|
||||
raw_opc_manager.client = AsyncMock()
|
||||
raw_opc_manager.cert_path = 'cert.pem'
|
||||
raw_opc_manager.private_key_path = 'private_key.pem'
|
||||
raw_opc_manager.server_cert_path = 'server_cert.pem'
|
||||
|
||||
return raw_opc_manager
|
||||
|
||||
|
||||
@fixture
|
||||
def opc_manager_subscribed(opc_manager):
|
||||
opc_manager.subscriptions['sub1'] = AsyncMock()
|
||||
|
||||
return opc_manager
|
||||
|
||||
|
||||
def test___str__(opc_manager):
|
||||
assert (
|
||||
str(opc_manager)
|
||||
== 'OpcManager(name=TestConnector, url=opc.tcp://localhost:4840, server_uri=opc.tcp://localhost:4840)\nnodes={}, subscriptions={}'
|
||||
)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_shutdown_success(opc_manager):
|
||||
opc_manager.disconnect = AsyncMock()
|
||||
|
||||
await opc_manager.shutdown()
|
||||
|
||||
opc_manager.disconnect.assert_called_once()
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_shutdown_error(opc_manager):
|
||||
opc_manager.disconnect = AsyncMock(side_effect=Exception('Test error'))
|
||||
|
||||
await opc_manager.shutdown()
|
||||
|
||||
opc_manager.logger.error.assert_called_once_with('Error during cleanup: Test error')
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_set_security_success(opc_manager):
|
||||
await opc_manager.set_security()
|
||||
|
||||
assert opc_manager.client.application_uri == opc_manager.server_uri
|
||||
|
||||
opc_manager.client.set_security.assert_called_once_with(
|
||||
SecurityPolicyBasic256,
|
||||
certificate=opc_manager.cert_path,
|
||||
private_key=opc_manager.private_key_path,
|
||||
server_certificate=opc_manager.server_cert_path,
|
||||
)
|
||||
|
||||
assert opc_manager.client.secure_channel_timeout == 10000000
|
||||
assert opc_manager.client.session_timeout == 10000000
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_set_security_no_cert(opc_manager):
|
||||
opc_manager.cert_path = None
|
||||
opc_manager.private_key_path = None
|
||||
|
||||
try:
|
||||
await opc_manager.set_security()
|
||||
except ValueError as e:
|
||||
assert str(e) == 'Certificate and private key paths must be provided for secure connection.'
|
||||
else:
|
||||
raise AssertionError('ValueError not raised')
|
||||
|
||||
assert opc_manager.client.set_security.call_count == 0
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('ingestor.managers.opc_manager.metrics')
|
||||
@patch('ingestor.managers.opc_manager.Client')
|
||||
async def test_connect_no_security(client, mock_metrics, raw_opc_manager):
|
||||
raw_opc_manager.set_security = AsyncMock()
|
||||
client.return_value = AsyncMock()
|
||||
|
||||
await raw_opc_manager.connect()
|
||||
|
||||
client.assert_called_once_with(raw_opc_manager.url, timeout=10, watchdog_intervall=3600000)
|
||||
raw_opc_manager.client.connect.assert_called_once()
|
||||
raw_opc_manager.set_security.assert_not_called()
|
||||
raw_opc_manager.emit_metric.assert_has_calls(
|
||||
[
|
||||
call(
|
||||
metric_object=mock_metrics.OPC_CONNECTIONS_TOTAL,
|
||||
tags={
|
||||
'pod_id': raw_opc_manager.pod_id,
|
||||
'server_name': raw_opc_manager.name,
|
||||
},
|
||||
),
|
||||
call(
|
||||
metric_object=mock_metrics.OPC_CONNECTION_STATUS,
|
||||
method='set',
|
||||
value=1,
|
||||
tags={
|
||||
'pod_id': raw_opc_manager.pod_id,
|
||||
'server_name': raw_opc_manager.name,
|
||||
'server_url': raw_opc_manager.url,
|
||||
},
|
||||
),
|
||||
],
|
||||
any_order=True,
|
||||
)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('ingestor.managers.opc_manager.Client')
|
||||
async def test_connect_with_security(client, raw_opc_manager):
|
||||
raw_opc_manager.cert_path = 'cert.pem'
|
||||
raw_opc_manager.private_key_path = 'private_key.pem'
|
||||
raw_opc_manager.server_cert_path = 'server_cert.pem'
|
||||
raw_opc_manager.set_security = AsyncMock()
|
||||
client.return_value = AsyncMock()
|
||||
|
||||
await raw_opc_manager.connect()
|
||||
|
||||
client.assert_called_once_with(raw_opc_manager.url, timeout=10, watchdog_intervall=3600000)
|
||||
raw_opc_manager.client.connect.assert_called_once()
|
||||
raw_opc_manager.set_security.assert_called_once()
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('ingestor.managers.opc_manager.Client')
|
||||
@patch('ingestor.managers.opc_manager.metrics')
|
||||
async def test_connect_exception_handling_and_metrics(
|
||||
mock_metrics_module, mock_opc_client_class, raw_opc_manager
|
||||
):
|
||||
mock_client_instance = mock_opc_client_class.return_value
|
||||
simulated_error_message = 'Erro de conexão simulado'
|
||||
mock_client_instance.connect.side_effect = Exception(simulated_error_message)
|
||||
|
||||
raw_opc_manager.disconnect = AsyncMock()
|
||||
|
||||
opc_manager_instance = raw_opc_manager
|
||||
opc_manager_instance.cert_path = None
|
||||
|
||||
with pytest.raises(Exception, match=simulated_error_message):
|
||||
await opc_manager_instance.connect()
|
||||
|
||||
opc_manager_instance.disconnect.assert_called_once()
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_create_subscription_no_client(raw_opc_manager):
|
||||
try:
|
||||
await raw_opc_manager.create_subscription('sub1')
|
||||
except ValueError as e:
|
||||
assert str(e) == 'Client not connected. Call connect first.'
|
||||
else:
|
||||
raise AssertionError('ValueError not raised')
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_create_subscription_success_has_period(opc_manager):
|
||||
await opc_manager.create_subscription('sub1')
|
||||
|
||||
opc_manager.client.create_subscription.assert_called_once_with(
|
||||
opc_manager.subscription_period_ms, opc_manager
|
||||
)
|
||||
assert opc_manager.subscriptions['sub1'] is not None
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_create_subscription_success_no_period(opc_manager):
|
||||
await opc_manager.create_subscription('sub1')
|
||||
|
||||
opc_manager.client.create_subscription.assert_called_once_with(
|
||||
opc_manager.subscription_period_ms, opc_manager
|
||||
)
|
||||
assert opc_manager.subscriptions['sub1'] is not None
|
||||
|
||||
|
||||
@patch('ingestor.managers.opc_manager.metrics')
|
||||
@mark.asyncio
|
||||
async def test_create_subscription_with_metrics(metrics, opc_manager):
|
||||
await opc_manager.create_subscription('sub1')
|
||||
|
||||
opc_manager.emit_metric.assert_called_once_with(
|
||||
metric_object=metrics.OPC_SUBSCRIPTIONS_CREATED,
|
||||
method='inc',
|
||||
value=1,
|
||||
tags={
|
||||
'pod_id': opc_manager.pod_id,
|
||||
'server_name': opc_manager.name,
|
||||
'slot_name': 'sub1',
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@patch('ingestor.managers.opc_manager.metrics')
|
||||
@mark.asyncio
|
||||
async def test_create_subscription_exception_during_client_call(
|
||||
mock_metrics_module, raw_opc_manager
|
||||
):
|
||||
opc_manager_instance = raw_opc_manager
|
||||
opc_manager_instance.client = AsyncMock()
|
||||
|
||||
subscription_name = 'test_sub_client_error'
|
||||
simulated_error_message = 'Falha ao criar subscrição no cliente OPC'
|
||||
|
||||
opc_manager_instance.client.create_subscription.side_effect = Exception(simulated_error_message)
|
||||
|
||||
with pytest.raises(Exception, match=simulated_error_message):
|
||||
await opc_manager_instance.create_subscription(subscription_name)
|
||||
|
||||
opc_manager_instance.client.create_subscription.assert_called_once_with(
|
||||
opc_manager_instance.subscription_period_ms, opc_manager_instance
|
||||
)
|
||||
|
||||
opc_manager_instance.logger.error.assert_called_once_with(
|
||||
f'Failed to create subscription {subscription_name} on {opc_manager_instance.name}: {simulated_error_message}'
|
||||
)
|
||||
|
||||
mock_metrics_module.OPC_SUBSCRIPTIONS_CREATED.labels.assert_not_called()
|
||||
|
||||
|
||||
@patch('ingestor.managers.opc_manager.metrics')
|
||||
@mark.asyncio
|
||||
async def test_subscribe_no_subscription(metrics, opc_manager):
|
||||
try:
|
||||
await opc_manager.subscribe('sub1', tags, 1000)
|
||||
except ValueError as e:
|
||||
assert str(e) == 'Subscription not created. Call create_subscription first.'
|
||||
else:
|
||||
raise AssertionError('ValueError not raised')
|
||||
metrics.OPC_TAGS_SUBSCRIBED.labels.assert_not_called()
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_subscribe_success(opc_manager_subscribed):
|
||||
opc_manager_subscribed.client.get_node = MagicMock()
|
||||
opc_manager_subscribed.nodes = {'ns=3;i=1001': 'data'}
|
||||
|
||||
await opc_manager_subscribed.subscribe('sub1', tags, 1000)
|
||||
|
||||
assert opc_manager_subscribed.nodes == tags
|
||||
opc_manager_subscribed.subscriptions['sub1'].subscribe_data_change.assert_called_once_with(
|
||||
[opc_manager_subscribed.client.get_node(n) for n in tags]
|
||||
)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_unsubscribe_no_subscription(opc_manager):
|
||||
await opc_manager.unsubscribe('sub1')
|
||||
|
||||
opc_manager.logger.warning.assert_called_once_with(
|
||||
"Subscription 'sub1' not found. Cannot unsubscribe."
|
||||
)
|
||||
assert opc_manager.subscriptions.get('sub1') is None
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_unsubscribe_success(opc_manager_subscribed):
|
||||
await opc_manager_subscribed.unsubscribe('sub1')
|
||||
|
||||
assert opc_manager_subscribed.subscriptions.get('sub1') is None
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_disconnection_fallback_success(opc_manager):
|
||||
opc_manager.client = AsyncMock()
|
||||
opc_manager.client.disconnect.return_value = True
|
||||
result = await opc_manager.disconnection_fallback()
|
||||
assert result == []
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_disconnection_fallback_fail(opc_manager):
|
||||
opc_manager.client = AsyncMock()
|
||||
opc_manager.client.disconnect.side_effect = Exception('Test error')
|
||||
result = await opc_manager.disconnection_fallback()
|
||||
assert result == [
|
||||
{'attempt': 1, 'error': 'Test error', 'traceback': ANY},
|
||||
{'attempt': 2, 'error': 'Test error', 'traceback': ANY},
|
||||
{'attempt': 3, 'error': 'Test error', 'traceback': ANY},
|
||||
{'attempt': 4, 'error': 'Test error', 'traceback': ANY},
|
||||
{'attempt': 5, 'error': 'Test error', 'traceback': ANY},
|
||||
]
|
||||
assert opc_manager.client.disconnect.call_count == 5
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_disconnect_success(opc_manager_subscribed):
|
||||
opc_manager_subscribed.client = MagicMock()
|
||||
|
||||
await opc_manager_subscribed.disconnect()
|
||||
|
||||
opc_manager_subscribed.subscriptions['sub1'].delete.assert_called_once()
|
||||
assert opc_manager_subscribed.client is None
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_disconnect_no_client(opc_manager_subscribed):
|
||||
opc_manager_subscribed.client = None
|
||||
assert await opc_manager_subscribed.disconnect() is None
|
||||
|
||||
opc_manager_subscribed.logger.warning.assert_has_calls(
|
||||
[
|
||||
call('Client already disconnected.'),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_disconnect_error_unsubscribe(opc_manager_subscribed):
|
||||
opc_manager_subscribed.client = MagicMock(disconnect=AsyncMock())
|
||||
opc_manager_subscribed.subscriptions['sub1'] = MagicMock(
|
||||
delete=AsyncMock(side_effect=Exception('Test error'))
|
||||
)
|
||||
|
||||
await opc_manager_subscribed.disconnect()
|
||||
|
||||
opc_manager_subscribed.subscriptions['sub1'].delete.assert_called_once()
|
||||
opc_manager_subscribed.client = None
|
||||
opc_manager_subscribed.logger.error.assert_called_once_with(
|
||||
'Failed to clean up subscription: Test error'
|
||||
)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_disconnect_error(opc_manager_subscribed):
|
||||
opc_manager_subscribed.client = MagicMock()
|
||||
opc_manager_subscribed.client.disconnect = MagicMock(side_effect=Exception('Test error'))
|
||||
|
||||
await opc_manager_subscribed.disconnect()
|
||||
|
||||
opc_manager_subscribed.subscriptions['sub1'].delete.assert_called_once()
|
||||
opc_manager_subscribed.client = None
|
||||
|
||||
|
||||
@patch('ingestor.managers.opc_manager.metrics')
|
||||
@mark.asyncio
|
||||
async def test_disconnect_metrics_on_successful_path(mock_metrics_module, raw_opc_manager):
|
||||
mock_metrics_module.OPC_CONNECTION_STATUS.reset_mock()
|
||||
mock_metrics_module.OPC_TAGS_SUBSCRIBED.reset_mock()
|
||||
|
||||
raw_opc_manager.client = MagicMock()
|
||||
mock_sub1 = MagicMock()
|
||||
mock_sub2 = MagicMock()
|
||||
raw_opc_manager.subscriptions = {'sub1': mock_sub1, 'sub2': mock_sub2}
|
||||
|
||||
await raw_opc_manager.disconnect()
|
||||
|
||||
raw_opc_manager.emit_metric.assert_has_calls(
|
||||
[
|
||||
call(
|
||||
metric_object=mock_metrics_module.OPC_CONNECTION_STATUS,
|
||||
method='set',
|
||||
value=0,
|
||||
tags={
|
||||
'pod_id': raw_opc_manager.pod_id,
|
||||
'server_name': raw_opc_manager.name,
|
||||
'server_url': raw_opc_manager.url,
|
||||
},
|
||||
),
|
||||
call(
|
||||
metric_object=mock_metrics_module.OPC_TAGS_SUBSCRIBED,
|
||||
method='set',
|
||||
value=0,
|
||||
tags={
|
||||
'pod_id': raw_opc_manager.pod_id,
|
||||
'server_name': raw_opc_manager.name,
|
||||
},
|
||||
),
|
||||
],
|
||||
any_order=True,
|
||||
)
|
||||
|
||||
|
||||
@patch('ingestor.managers.opc_manager.metrics')
|
||||
@mark.asyncio
|
||||
async def test_datachange_notification(metrics, opc_manager_subscribed):
|
||||
data = MagicMock(
|
||||
monitored_item=MagicMock(
|
||||
Value=MagicMock(
|
||||
Value=MagicMock(Value=42),
|
||||
SourceTimestamp=datetime.strptime('2021-01-01T00:00:00', '%Y-%m-%dT%H:%M:%S'),
|
||||
)
|
||||
)
|
||||
)
|
||||
opc_manager_subscribed.nodes = {
|
||||
'ns=3;i=1001': {
|
||||
'tag_name': 'Counter',
|
||||
'cycle_rule': {'cycle_increment': 1.0, 'cycle_count': 2},
|
||||
'topics': ['topic1', 'topic2'],
|
||||
}
|
||||
}
|
||||
|
||||
await opc_manager_subscribed.datachange_notification('ns=3;i=1001', None, data)
|
||||
|
||||
opc_manager_subscribed.data_manager.publish.assert_any_call(
|
||||
'topic1',
|
||||
{
|
||||
'tag': 'ns=3;i=1001',
|
||||
'name': 'Counter',
|
||||
'timestamp': '2021-01-01 00:00:00-0300',
|
||||
'value': 42,
|
||||
},
|
||||
)
|
||||
opc_manager_subscribed.data_manager.publish.assert_any_call(
|
||||
'topic2',
|
||||
{
|
||||
'tag': 'ns=3;i=1001',
|
||||
'name': 'Counter',
|
||||
'timestamp': '2021-01-01 00:00:00-0300',
|
||||
'value': 42,
|
||||
},
|
||||
)
|
||||
assert opc_manager_subscribed.nodes['ns=3;i=1001']['cycle_rule']['cycle_count'] == 0
|
||||
|
||||
opc_manager_subscribed.emit_metric.assert_called_once_with(
|
||||
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
|
||||
method='set',
|
||||
value=0,
|
||||
tags={
|
||||
'pod_id': opc_manager_subscribed.pod_id,
|
||||
'server_name': opc_manager_subscribed.name,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_check_cycles_no_notification(opc_manager):
|
||||
# Setup: node with cycle_count just below threshold
|
||||
opc_manager.nodes = {
|
||||
'ns=3;i=1001': {
|
||||
'tag_name': 'Counter',
|
||||
'cycle_rule': {'cycle_increment': 1.0, 'cycle_count': 3.0},
|
||||
}
|
||||
}
|
||||
|
||||
await opc_manager.check_cycles()
|
||||
|
||||
# After one increment, cycle_count = 4.0, still below threshold
|
||||
assert opc_manager.nodes['ns=3;i=1001']['cycle_rule']['cycle_count'] == pytest.approx(4.0)
|
||||
opc_manager.send_notification_async.assert_not_called()
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_check_cycles_triggers_notification(opc_manager):
|
||||
# Setup: node with cycle_count just below threshold, increment will cross threshold
|
||||
opc_manager.nodes = {
|
||||
'ns=3;i=1001': {
|
||||
'tag_name': 'Counter',
|
||||
'cycle_rule': {'cycle_increment': 2.5, 'cycle_count': 3.0},
|
||||
}
|
||||
}
|
||||
opc_manager.notification_handler.build_and_send_notification = MagicMock()
|
||||
|
||||
await opc_manager.check_cycles()
|
||||
|
||||
# After increment, cycle_count = 5.5, should trigger notification
|
||||
assert opc_manager.nodes['ns=3;i=1001']['cycle_rule']['cycle_count'] == pytest.approx(5.5)
|
||||
opc_manager.send_notification_async.assert_called_once_with(
|
||||
notification_id='TAG_ns=3;i=1001:Counter_LISTENNING_STOPPED',
|
||||
message='5.5 cycles without receive from ns=3;i=1001:Counter',
|
||||
block='opc_manager',
|
||||
level=NotificationLevel.WARNING,
|
||||
metadata=metadata['metadata'],
|
||||
)
|
||||
|
||||
|
||||
@patch('ingestor.managers.opc_manager.metrics')
|
||||
@mark.asyncio
|
||||
async def test_check_opc_listenning_no_notification(metrics, opc_manager):
|
||||
opc_manager.non_receive_count = 3
|
||||
|
||||
result = await opc_manager.check_opc_listenning()
|
||||
|
||||
assert opc_manager.non_receive_count == 4
|
||||
opc_manager.send_notification_async.assert_not_called()
|
||||
assert result is False
|
||||
|
||||
opc_manager.emit_metric.assert_called_once_with(
|
||||
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
|
||||
method='set',
|
||||
value=opc_manager.non_receive_count,
|
||||
tags={
|
||||
'pod_id': opc_manager.pod_id,
|
||||
'server_name': opc_manager.name,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_check_opc_listenning_warning_notification(opc_manager):
|
||||
opc_manager.non_receive_count = 4
|
||||
|
||||
result = await opc_manager.check_opc_listenning()
|
||||
|
||||
assert opc_manager.non_receive_count == 5
|
||||
opc_manager.send_notification_async.assert_called_once_with(
|
||||
notification_id=f'OPC_LISTENNING_STOPPED__{opc_manager.name}',
|
||||
message=f'5 cycles without receive from OPC {opc_manager.name}. Tags: {json.dumps(opc_manager.nodes)}',
|
||||
block='opc_manager',
|
||||
level=NotificationLevel.ERROR,
|
||||
metadata=metadata['metadata'],
|
||||
)
|
||||
assert result is False
|
||||
|
||||
|
||||
@patch('ingestor.managers.opc_manager.metrics')
|
||||
@mark.asyncio
|
||||
async def test_check_opc_listenning_error_notification_and_retry(metrics, opc_manager):
|
||||
opc_manager.non_receive_count = 14
|
||||
|
||||
result = await opc_manager.check_opc_listenning()
|
||||
|
||||
assert opc_manager.non_receive_count == 15
|
||||
# Should be called twice: once for 5, once for 15
|
||||
assert opc_manager.send_notification_async.call_count == 2
|
||||
calls = opc_manager.send_notification_async.call_args_list
|
||||
# First call: 5 cycles warning
|
||||
assert calls[0].kwargs == {
|
||||
'notification_id': f'OPC_LISTENNING_STOPPED__{opc_manager.name}',
|
||||
'message': f'15 cycles without receive from OPC {opc_manager.name}. Tags: {json.dumps(opc_manager.nodes)}',
|
||||
'block': 'opc_manager',
|
||||
'level': NotificationLevel.ERROR,
|
||||
'metadata': metadata['metadata'],
|
||||
}
|
||||
# Second call: 15 cycles retry
|
||||
assert calls[1].kwargs == {
|
||||
'notification_id': f'OPC_CONNECTION_RETRY__{opc_manager.name}',
|
||||
'message': f'Retrying to connect to server {opc_manager.name}',
|
||||
'block': 'opc_manager',
|
||||
'level': NotificationLevel.ERROR,
|
||||
'metadata': metadata['metadata'],
|
||||
}
|
||||
assert result is True
|
||||
|
||||
opc_manager.emit_metric.assert_has_calls(
|
||||
[
|
||||
call(
|
||||
metric_object=metrics.OPC_CYCLES_WITHOUT_DATA,
|
||||
method='set',
|
||||
value=opc_manager.non_receive_count,
|
||||
tags={
|
||||
'pod_id': opc_manager.pod_id,
|
||||
'server_name': opc_manager.name,
|
||||
},
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
opc_manager.emit_metric.assert_has_calls(
|
||||
[
|
||||
call(
|
||||
metric_object=metrics.OPC_RECONNECTIONS_TOTAL,
|
||||
tags={
|
||||
'pod_id': opc_manager.pod_id,
|
||||
'server_name': opc_manager.name,
|
||||
},
|
||||
),
|
||||
]
|
||||
)
|
||||
@@ -9,9 +9,14 @@ SERVER_CONFIG = {
|
||||
'name': 'kepserver',
|
||||
'server_id': 1, # the slot may carry it as int; the pool keys by str
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'use_security': True,
|
||||
'subscription_period_ms': 15000.0,
|
||||
'security': {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'},
|
||||
'certificate': {
|
||||
'certificate_id': 1,
|
||||
'application_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert': {'bucket': 'opc-certificates', 'object_name': 'client/p/cert.pem'},
|
||||
'private_key': {'bucket': 'opc-certificates', 'object_name': 'client/p/key.pem'},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -45,10 +50,8 @@ def test_server_cfg_keeps_only_connection_fields():
|
||||
|
||||
assert reader.server_id == '1'
|
||||
assert reader.server_cfg == {
|
||||
'url': 'opc.tcp://kepserver:49320',
|
||||
'server_uri': 'urn:sientia:opc-ingestor',
|
||||
'use_security': True,
|
||||
'subscription_period_ms': 15000.0,
|
||||
key: SERVER_CONFIG[key]
|
||||
for key in ('url', 'subscription_period_ms', 'security', 'certificate')
|
||||
}
|
||||
|
||||
|
||||
@@ -58,7 +61,7 @@ async def test_connect_opens_pooled_connection():
|
||||
|
||||
await reader.connect()
|
||||
|
||||
# use_security=True: the pool fetches the certificate from MinIO (CertProvider)
|
||||
# With a certificate, the pool fetches it from MinIO (CertProvider)
|
||||
pool.get.assert_awaited_once_with('1', reader.server_cfg)
|
||||
|
||||
|
||||
|
||||
@@ -8,7 +8,6 @@ _ENV_VARS = (
|
||||
'NATS_QUEUE',
|
||||
'GRPC_ENABLED',
|
||||
'GRPC_PORT',
|
||||
'CERT_BUCKET',
|
||||
'CERT_CACHE_DIR',
|
||||
'WRITE_TIMESTAMP',
|
||||
'WRITE_QUALITY',
|
||||
@@ -31,7 +30,6 @@ def test_from_env_defaults():
|
||||
assert cfg.nats_queue == 'gateway-writers'
|
||||
assert cfg.grpc_enabled is False
|
||||
assert cfg.grpc_port == 50051
|
||||
assert cfg.cert_bucket == 'streamlit-connectors'
|
||||
assert cfg.cert_cache_dir == '/tmp/opc-certs' # noqa: S108
|
||||
assert cfg.write_timestamp is True
|
||||
assert cfg.write_quality is True
|
||||
@@ -76,12 +74,10 @@ def test_certs_from_env_ignores_malformed_numeric_vars(monkeypatch):
|
||||
|
||||
monkeypatch.setenv('GRPC_PORT', '')
|
||||
monkeypatch.setenv('IDLE_CONN_MAX_S', 'abc')
|
||||
monkeypatch.setenv('CERT_BUCKET', 'my-bucket')
|
||||
monkeypatch.setenv('CERT_CACHE_DIR', 'certs-cache')
|
||||
|
||||
cfg = GatewayConfig.certs_from_env()
|
||||
|
||||
assert cfg.cert_bucket == 'my-bucket'
|
||||
assert cfg.cert_cache_dir == 'certs-cache'
|
||||
with pytest.raises(ValueError):
|
||||
GatewayConfig.from_env() # the full config still fails, only the runtime uses it
|
||||
|
||||
Reference in New Issue
Block a user