gitea_admin/sientia-dataops-opc-ingestor
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

SIENTIAPDE-1974: feat(read): slot v2 reads via SubscriptionEngine, OPC cert from MinIO

Browse Source 
Swaps the legacy OpcManager read path for the gateway's SubscriptionEngine over
the shared OpcConnectionPool. The OPC client certificate now always comes from
MinIO + Mongo (CertProvider, Basic256Sha256) — no more certs/ shipped in the
Gitea repository, so a new version branch no longer breaks the reads.

- io/slots.py: parse_slot accepts slot v2 (names[]) and converts the legacy
  v1 (tag_name + topics; cert_path => use_security, path dropped), so gateway
  and Orchestrator can be deployed in any order.
- managers/server_reader.py: new per-server reader with the OpcManager
  interface the IngestorManager already drives (connect, create_subscription,
  subscribe, unsubscribe, check_cycles, check_opc_listenning, shutdown).
- IngestorManager owns the read path: lazy-MinIO CertProvider, the pool and a
  read DataSink, built without touching the network. Slots from Redis are
  normalized on lease and refresh; a malformed slot is notified and the last
  good config is kept.
- GatewayRuntime borrows the pool (writes/gRPC) instead of building it, and no
  longer closes it on stop: a Postgres/NATS failure can't stop the reads.
- pool: pin/unpin per active subscription so release_idle never evicts a
  subscribed connection; public evict() for lost servers / changed config.
- SubscriptionEngine pins on subscribe, unpins on unsubscribe, and drops a
  subscription locally even when the server-side delete fails.
- DataSink: model_id only written when the name has one, keeping v1 documents
  identical to the legacy ones.
- config: CERT_BUCKET default is the legacy streamlit-connectors bucket (D7).
- Removes OpcManager, its tests and scripts/generate-and-push-opc-cert.sh;
  README/CLAUDE.md document the MinIO certificate and the slot v2 format.

Tests: 218 unit tests pass; ruff check/format clean. Local end-to-end against
an asyncua server: v1 slot with two topics -> one doc per collection, legacy
doc shape, idle sweep keeps the subscription, lost-server shutdown and
re-initialization work.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Eduardo Rios
2026-10-06 13:27:18 -03:00
parent 4c85a9c5c2
commit 50578fdba8
25 changed files with 957 additions and 1534 deletions
Download Patch File Download Diff File Expand all files Collapse all files

97
README.md
View File

@@ -123,9 +123,9 @@ health probe, NetworkPolicy, cert cache) and the `# OPC Gateway` block in
# Ensure services are accessible on localhost with appropriate ports
```
6. **Generate an OPC UA client certificate** (only if connecting to a secured OPC UA server)
6. **Register the OPC UA client certificate** (only if connecting to a secured OPC UA server)
See [Certificate Generation](#-certificate-generation) below.
See [OPC Client Certificate](#-opc-client-certificate) below.
## Usage
@@ -193,37 +193,31 @@ pytest --cov=ingestor --cov-report=html
The feeder creates sample OPC tag configurations in Redis that the ingestor can discover and manage.
## 🔐 Certificate Generation
## 🔐 OPC Client Certificate
OPC UA servers that require secure connections (e.g. KEPServer with `Basic256` policy) validate the ingestor via an X.509 client certificate. Use `scripts/generate-and-push-opc-cert.sh` to generate that certificate and push it to the Gitea repo the pod clones on boot.
OPC UA servers that require secure connections validate the gateway with a single,
shared platform X.509 client certificate. Every OPC connection — subscription reads,
NATS writes and gRPC reads — uses it with `Basic256Sha256` (SIENTIAPDE-1974).
```bash
GITEA_PASSWORD='<senha-do-gitea>' ./scripts/generate-and-push-opc-cert.sh
```
The certificate is **never** read from the repository or the image. On connect, the
`CertProvider` (`ingestor/connection/certs.py`):
1. reads the single document of the Mongo collection `platform_certificates`:
`{"client_cert": {"minio_object_name": "..."}, "private_key": {"minio_object_name": "..."}}`;
2. downloads both objects from the MinIO bucket `CERT_BUCKET` (default
`streamlit-connectors`) into `CERT_CACHE_DIR` (default `/tmp/opc-certs`).
Environment variables (all optional except `GITEA_PASSWORD`):
Requirements for a server to accept the connection:
- the server's slot config has `use_security: true` (a slot v1 server with a
`cert_path` is treated the same way; the path itself is ignored);
- the certificate's SAN URI equals the server's `server_uri` in the slot — it is sent
as the client `application_uri` (KEPServer rejects a mismatch with
`BadCertificateUriInvalid`);
- the server exposes a `Basic256Sha256` endpoint and trusts the certificate (e.g.
KEPServer: *OPC UA Configuration Manager → Trusted Clients → Trust*, then restart
its runtime).
| Variable | Description | Default |
|----------|-------------|---------|
| `GITEA_URL` | Gitea in-cluster URL | `https://git.sientia.ai` |
| `GITEA_USER` | Gitea user | `gitea_admin` |
| `GITEA_PASSWORD` | Gitea password (**required**) | — |
| `GITEA_REPO` | Repo the ingestor pod clones | `gitea_admin/sientia-dataops-opc-ingestor` |
| `BRANCH` | Branch to push the cert to | `main` |
| `APP_URI` | SAN URI of the cert — **must match** the `uri` field of the corresponding document in the `OPC_servers` Mongo collection, since the ingestor uses it as `application_uri` (KEPServer rejects with `BadCertificateUriInvalid` on mismatch) | `urn:sientia:opc-ingestor` |
| `CN` | Certificate common name | `sientia-opc-ingestor` |
| `DAYS` | Certificate validity (days) | `3650` |
| `FORCE` | Set to `1` to overwrite an existing cert already pushed to the repo | unset |
What the script does:
1. Generates a self-signed RSA-2048 X.509 cert + key with `openssl`, with the SAN URI, `keyUsage`, and `extendedKeyUsage` extensions OPC UA / KEPServer require.
2. Clones the ingestor's Gitea repo, copies the cert/key (`.pem`) and a `.der` copy (for manual import) into `certs/`, commits, and pushes.
3. Prints the paths the pod will see after cloning (`/app/certs/opc-ingestor-cert.pem`, `/app/certs/opc-ingestor-key.pem`) and the next manual steps.
After pushing, you still need to:
- Set `cert_path` / `private_key_path` (and `uri`) on the server's document in the `OPC_servers` collection so `OpcManager` picks them up (see [OPC Server Configuration](#opc-server-configuration)).
- Restart the ingestor deployment so it re-clones the repo: `kubectl -n sientia rollout restart deployment/opc-ingestor`.
- On first connection, the server rejects the new cert — trust it in the OPC UA server's console (e.g. KEPServer: *OPC UA Configuration Manager → Trusted Clients → Trust*) and restart its runtime.
MinIO is only contacted when a certificate is first needed, so a MinIO outage only
affects secured servers, which are retried on the next main-loop iteration.
## 🧪 Testing
@@ -306,7 +300,12 @@ gateway-specific environment variables.
### OPC Server Configuration
OPC servers are configured through Redis with the following structure. `cert_path` / `private_key_path` are only required for secured connections (see [Certificate Generation](#-certificate-generation)):
OPC servers are configured by the Orchestrator through Redis, one key per slot
(`slot:opc_tags:<n>`). The gateway reads the **slot v2** structure (contract C1):
one entry per address, with one `names` item per collection / tag name / model, so
the same address can feed several models under different names. For secured
connections set `use_security: true` (the certificate comes from MinIO, see
[OPC Client Certificate](#-opc-client-certificate)):
```json
{
@@ -315,30 +314,30 @@ OPC servers are configured through Redis with the following structure. `cert_pat
"name": "server1",
"server_id": "1",
"url": "opc.tcp://localhost:4841",
"server_uri": "http://opcua-server.simulator",
"cert_path": "/app/certs/opc-ingestor-cert.pem",
"private_key_path": "/app/certs/opc-ingestor-key.pem",
"server_uri": "urn:sientia:opc-ingestor",
"use_security": true,
"subscription_period_ms": 7500,
"tags": {
"ns=2;i=2": {
"aggr_func": "avg",
"data_range": [
-100,
100
],
"frequency": "15000",
"server_id": "1",
"tag_address": "ns=2;i=2",
"tag_name": "Counter",
"topics": [
"raw_scouter-opcua-orchestrated-pipeline",
"raw_scouter-basic-sum-model"
]
"frequency": "15000",
"active_read": {"enabled": false},
"names": [
{"collection": "raw_scouter-opcua-orchestrated-pipeline", "tag_name": "Counter", "model_name": "m1", "model_id": "1"},
{"collection": "raw_scouter-basic-sum-model", "tag_name": "Contador", "model_name": "m2", "model_id": "2"}
]
}
}
}
}
}
```
The legacy **slot v1** (each tag with `tag_name` + `topics`, the server with
`cert_path`) is still accepted and converted on read (`ingestor/io/slots.py`): one
`names` item per topic, and `use_security` is true when `cert_path` is set (the path
itself is ignored). This lets the gateway and the Orchestrator be deployed in any
order. Documents written from a v1 slot have no `model_id`, exactly as before.
## 🔧 Development
### Project Structure
@@ -348,13 +347,13 @@ sientia-dataops-opc-ingestor/
│ ├── managers/ # Component managers
│ │ ├── data_manager.py # Data persistence and export
│ │ ├── ingestor_manager.py # Main coordination
│ │ ├── opc_manager.py # OPC UA server management
│ │ └── resource_manager.py # Resource coordination
│ │ ├── resource_manager.py # Resource coordination
│ │ └── server_reader.py # OPC UA server reads (SubscriptionEngine + pool)
│ ├── app.py # Main application entry point
│ ├── ingestor.py # Core ingestor logic
│ └── metrics.py # Prometheus metrics definitions
├── simulator/ # OPC simulation and testing tools
├── scripts/ # Ops scripts (e.g. OPC UA cert generation)
├── scripts/ # Ops scripts (e.g. e2e seed)
├── tests/ # Test suite
└── requirements.txt # Python dependencies
```