SIENTIAPDE-1973

SIENTIAPDE-1973: Implement per-connector OPC client certificates and remove legacy read engine.
This update overhauls certificate management, transitioning from a single platform certificate to individual client certificates defined in the slot configuration. The CertProvider now downloads certificates from MinIO based on certificate_id, applying dynamic security policies (mode, policy) from the slot. The legacy READ_ENGINE=legacy option and OpcManager have been entirely removed, standardizing on the ServerReader with strict slot v2 enforcement. Write command resolution (ServerConfigResolver) now consistently uses connectors joined with opc_certificates. The CERT_BUCKET environment variable is no longer used.

README.md

@@ -63,8 +63,8 @@ Gateway for on-demand and command-driven interactions with OPC UA servers:
 - **Scheduled active reads**: a polling read engine (`ingestor/engines/active_read.py`)
   for tags that need periodic reads outside the subscription pipeline;
   orchestrator-side wiring lands in SIENTIAPDE-1974.
-- **Certificate management**: client certs for secure OPC connections are
-  fetched from MinIO + a Mongo `platform_certificates` collection
+- **Certificate management**: each OPC connector has its own client certificate
+  (`opc_certificates`), named in the slot and downloaded from MinIO
   (`ingestor/connection/certs.py`) rather than shipped in the image.
 
 See `docs/deploy-notes.md` for the deployment-facing details (env vars,
@@ -83,12 +83,12 @@ The Laborious publishes one command per tag (SIENTIAPDE-1975):
 ```
 
 - **Server connection**: the command carries only `server_id`. The connection is
-  resolved by `ServerConfigResolver` (`ingestor/connection/servers.py`): `url`/
-  `server_uri`/`use_security` in the command when present (override), else the
-  live pooled connection opened by the reads, else the `connectors` document
-  (`type: opc_ua`, `status: Active`), else the `opc-servers` document
-  (`use_security = bool(cert_path)`) — the orchestrator's rule for the read slots.
-  Lookups are cached for 60 s. An unknown server is a failed write (`server_not_found`).
+  resolved by `ServerConfigResolver` (`ingestor/connection/servers.py`): the live
+  pooled connection opened by the reads, else the `connectors` document (`type: opc_ua`,
+  `status: Active`) joined with `opc_certificates`, with the orchestrator's rule for the
+  read slots (`ingestor/connection/certificates.py`), in the slot format (`url`,
+  `security`, `certificate`). Lookups are cached for 60 s. An unknown or incomplete
+  server is a failed write (`server_not_found`).
 - **Outcome**: `UPDATE ... SET write_status` on the prediction row (by `timestamp` +
   `model_id`), keeping the worst outcome: `ok` < `error` < `session_bad`, and
   `pending`/NULL are replaced by any answer, so the order of the answers does not
@@ -227,48 +227,42 @@ The feeder creates sample OPC tag configurations in Redis that the ingestor can
 
 ## 🔐 OPC Client Certificate
 
-OPC UA servers that require secure connections validate the gateway with a single,
-shared platform X.509 client certificate. Every OPC connection — subscription reads,
-NATS writes and gRPC reads — uses it with `Basic256Sha256` (SIENTIAPDE-1974).
+Each OPC connector has its own X.509 client certificate (SIENTIAPDE-1974): the
+connector's `security.client_certificate_id` points to an `opc_certificates` document
+(`kind: client`, `status: active`). The Orchestrator joins them and writes, for each
+server of the slot, its `security` (`mode`, `policy`) and its `certificate`:
+
+```json
+"certificate": {
+  "certificate_id": 1,
+  "application_uri": "urn:sientia:opc-ingestor",
+  "cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-cert.pem"},
+  "private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-key.pem"}
+}
+```
 
 The certificate is **never** read from the repository or the image. On connect, the
-`CertProvider` (`ingestor/connection/certs.py`):
-1. reads the single document of the Mongo collection `platform_certificates`:
-   `{"client_cert": {"minio_object_name": "..."}, "private_key": {"minio_object_name": "..."}}`;
-2. downloads both objects from the MinIO bucket `CERT_BUCKET` (default
-   `streamlit-connectors`) into `CERT_CACHE_DIR` (default `/tmp/opc-certs`).
+`CertProvider` (`ingestor/connection/certs.py`) downloads both objects from MinIO into
+`CERT_CACHE_DIR/<certificate_id>/` (default `/tmp/opc-certs`), and caches them per
+certificate. When the slot names other objects for the same certificate (it was
+replaced), the server config changes, the connection is re-opened and the files are
+downloaded again. A failed secured connection also forces a new download.
 
-Requirements for a server to accept the connection:
-- the server's slot config has `use_security: true` (a slot v1 server with a
-  `cert_path` is treated the same way; the path itself is ignored);
-- the certificate's SAN URI equals the server's `server_uri` in the slot — it is sent
-  as the client `application_uri` (KEPServer rejects a mismatch with
-  `BadCertificateUriInvalid`);
-- the server exposes a `Basic256Sha256` endpoint and trusts the certificate (e.g.
-  KEPServer: *OPC UA Configuration Manager → Trusted Clients → Trust*, then restart
-  its runtime).
+- **Security only with a certificate.** With a `certificate`, the connection uses the
+  slot's `security.mode` (`Sign` / `SignAndEncrypt`) and `security.policy`
+  (`Basic256Sha256`, `Aes128Sha256RsaOaep`, `Aes256Sha256RsaPss`, `Basic256`,
+  `Basic128Rsa15`); an unknown one fails the connection. Without a certificate
+  (`null`), or with `mode: None`, it connects without security, and a server that
+  requires a certificate refuses the connection (the error shows in the log and the
+  connection metrics). There is no default certificate.
+- The certificate's `application_uri` (its SAN URI) is sent as the client application
+  URI (KEPServer rejects a mismatch with `BadCertificateUriInvalid`).
+- The server must trust the certificate (e.g. KEPServer: *OPC UA Configuration Manager
+  → Trusted Clients → Trust*, then restart its runtime).
 
-MinIO is only contacted when a certificate is first needed, so a MinIO outage only
-affects secured servers, which are retried on the next main-loop iteration.
-
-### Plan B: legacy read engine (`READ_ENGINE=legacy`)
-
-Until the end of SIENTIAPDE-1974 the legacy reader is kept for validation and as a
-fallback. `READ_ENGINE` selects the read path (default `v2`; an invalid value falls
-back to `v2` with a warning):
-
-| `READ_ENGINE` | Reader | Slot | Certificate |
-|---|---|---|---|
-| `v2` (default) | `ServerReader` + `SubscriptionEngine` over the shared pool | v1 or v2 | MinIO + Mongo (`CertProvider`), `Basic256Sha256` |
-| `legacy` | the rc3 `OpcManager`, unchanged, with its own connection | **v1 only** (a v2 slot is rejected and notified) | files from the repository the pod clones (`cert_path` / `private_key_path` in the slot), `Basic256` |
-
-Writes and gRPC always use the pool (MinIO certificate), whatever the read engine.
-The legacy engine needs the certificate files in the Gitea branch the pod clones;
-`scripts/generate-and-push-opc-cert.sh` generates them and pushes them to
-`certs/` (`GITEA_PASSWORD='<senha>' BRANCH=<branch> ./scripts/generate-and-push-opc-cert.sh`).
-Each version deploy creates a new Gitea branch without `certs/`, so they have to be
-copied again for the legacy engine. Remove the legacy engine (OpcManager, its tests
-and the script) once it is unused and the test suite passes the same without it.
+MinIO is only contacted when a certificate is first needed (one client per bucket), so
+a MinIO outage only affects secured servers, which are retried on the next main-loop
+iteration.
 
 ## 🧪 Testing
 
@@ -349,7 +343,7 @@ The OPC Ingestor exposes comprehensive Prometheus metrics for monitoring:
 | `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
 
 See the `# OPC Gateway` block in `.env.example` (`NATS_URL`, `GRPC_ENABLED`,
-`CERT_BUCKET`, `WRITE_TIMESTAMP`, ...) and `docs/deploy-notes.md` for the
+`CERT_CACHE_DIR`, `WRITE_TIMESTAMP`, ...) and `docs/deploy-notes.md` for the
 gateway-specific environment variables.
 
 ### OPC Server Configuration
@@ -357,8 +351,8 @@ gateway-specific environment variables.
 OPC servers are configured by the Orchestrator through Redis, one key per slot
 (`slot:opc_tags:<n>`). The gateway reads the **slot v2** structure (contract C1):
 one entry per address, with one `names` item per collection / tag name / model, so
-the same address can feed several models under different names. For secured
-connections set `use_security: true` (the certificate comes from MinIO, see
+the same address can feed several models under different names. Each server carries
+its `security` and its `certificate` (`null` = no security, see
 [OPC Client Certificate](#-opc-client-certificate)):
 
 ```json
@@ -366,11 +360,16 @@ connections set `use_security: true` (the certificate comes from MinIO, see
   "slot:opc_tags:1": {
     "server1": {
       "name": "server1",
-      "server_id": "1",
+      "server_id": "kepserver",
       "url": "opc.tcp://localhost:4841",
-      "server_uri": "urn:sientia:opc-ingestor",
-      "use_security": true,
       "subscription_period_ms": 7500,
+      "security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256"},
+      "certificate": {
+        "certificate_id": 1,
+        "application_uri": "urn:sientia:opc-ingestor",
+        "cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-cert.pem"},
+        "private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-key.pem"}
+      },
       "tags": {
         "ns=2;i=2": {
           "frequency": "15000",
@@ -386,11 +385,10 @@ connections set `use_security: true` (the certificate comes from MinIO, see
 }
 ```
 
-The legacy **slot v1** (each tag with `tag_name` + `topics`, the server with
-`cert_path`) is still accepted and converted on read (`ingestor/io/slots.py`): one
-`names` item per topic, and `use_security` is true when `cert_path` is set (the path
-itself is ignored). This lets the gateway and the Orchestrator be deployed in any
-order. Documents written from a v1 slot have no `model_id`, exactly as before.
+Only the slot v2 is accepted (`ingestor/io/slots.py`): a slot without `security`, a
+tag without `names`, or a name without `collection`, `tag_name` or `model_id` is
+rejected and notified (`OPC_SLOT_CONFIG_ERROR_<slot>`), and the last good config keeps
+being read. The gateway and the Orchestrator of SIENTIAPDE-1974 are deployed together.
 
 ## 🔧 Development
 
@@ -401,14 +399,13 @@ sientia-dataops-opc-ingestor/
 │   ├── managers/            # Component managers
 │   │   ├── data_manager.py # Data persistence and export
 │   │   ├── ingestor_manager.py # Main coordination
-│   │   ├── opc_manager.py  # Legacy reader, plan B (READ_ENGINE=legacy)
 │   │   ├── resource_manager.py # Resource coordination
 │   │   └── server_reader.py # OPC UA server reads (SubscriptionEngine + pool)
 │   ├── app.py              # Main application entry point
 │   ├── ingestor.py         # Core ingestor logic
 │   └── metrics.py          # Prometheus metrics definitions
 ├── simulator/               # OPC simulation and testing tools
-├── scripts/                 # Ops scripts (e2e seed; legacy OPC cert generation)
+├── scripts/                 # Ops scripts (e2e seed)
 ├── tests/                   # Test suite
 └── requirements.txt         # Python dependencies
 ```