gitea_admin/sientia-dataops-orchestrator_temporal
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

SIENTIAPDE-1974

Browse Source 
SIENTIAPDE-1974-connectors-pi-web-api: Implement OPC UA client certificate handling and update connector security model. Connectors now define security using security.mode and security.policy. Client certificate details (MinIO paths) are fetched from the opc_certificates collection via a MongoDB $lookup and included in the OPC slot. A warning is issued if a connector's client_certificate_id does not resolve to an active client certificate. Documentation (README, CLAUDE.md) and tests have been updated.
This commit is contained in:
Vitor Pimentel dos Santos
2026-10-29 12:21:11 -03:00
parent 5b47726a64
commit 0b928a0ea3
19 changed files with 595 additions and 88 deletions
Download Patch File Download Diff File Expand all files Collapse all files

163
README.md
View File

@@ -103,7 +103,7 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
4. **Parallel Operations**: Executes slot deletions, slot updates, schedule deletions, schedule creations, and schedule updates concurrently
5. **Parallel Reports & Timestamps**: Generates orchestration reports and updates MongoDB timestamps for created/updated/deleted pipelines
**Input Parameters**:
**Input Parameters** (minimal: each pipeline must already carry `model.name`; see "Workflow Configuration" for the `$lookup` on `models` that adds it):
```json
{
"schedule_name": "hourly_orchestration",
@@ -125,8 +125,19 @@ Optional: `connectors_query` and `connector_tags_query` override the tag databas
"collection": "connectors",
"aggregation": [
{"$match": {"type": "opc_ua", "status": "Active"}},
{"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "server_uri": 1,
"use_security": 1, "subscription_period_ms": 1}}
{"$lookup": {"from": "opc_certificates",
"let": {"certificate_id": "$security.client_certificate_id"},
"pipeline": [
{"$match": {"$expr": {"$eq": ["$certificate_id", "$$certificate_id"]},
"status": "active", "kind": "client"}},
{"$project": {"_id": 0, "certificate_id": 1, "application_uri": 1,
"cert.bucket": 1, "cert.object_name": 1,
"private_key.bucket": 1, "private_key.object_name": 1}}
],
"as": "certificates"}},
{"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "subscription_period_ms": 1,
"security.mode": 1, "security.policy": 1,
"security.client_certificate_id": 1, "certificates": 1}}
]
},
"connector_tags_query": {
@@ -145,13 +156,19 @@ The referenced tags are loaded whatever their category and status: the code uses
#### OPC slots (slot v2, contract C1)
`process_slots` writes one slot per active ingestor to Redis (`slot:opc_tags:{n}`). The OPC Gateway reads it (SIENTIAPDE-1974). Each server carries its connection fields, and each tag address carries the list of names it is written under:
`process_slots` writes one slot per active ingestor to Redis (`slot:opc_tags:{n}`). The OPC Gateway reads it (SIENTIAPDE-1974). Each server carries its connection fields, its security and the MinIO objects of its client certificate, and each tag address carries the list of names it is written under:
```json
{
"kepserver": {
"server_id": "1", "name": "kepserver", "url": "opc.tcp://host:49320",
"server_uri": "urn:...", "use_security": true, "subscription_period_ms": 7500.0,
"subscription_period_ms": 7500.0,
"security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256"},
"certificate": {
"certificate_id": 1, "application_uri": "urn:sientia:opc-ingestor",
"cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-cert.pem"},
"private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-key.pem"}
},
"tags": {
"ns=2;s=Channel1.Device1.Tag1": {
"frequency": 15000,
@@ -171,7 +188,8 @@ Each scouter `read_tags` entry references its tag by `tag_id` (`{"tag_id": 7, "f
| Field | Source |
|---|---|
| Address | `connector_tags.address` |
| Server (`server_id`, `name`, `url`, `server_uri`, `use_security`, `subscription_period_ms`) | The tag's `connector_tags.connector_id` in `connectors` (`server_id` = `connector_id`) |
| Server (`server_id`, `name`, `url`, `subscription_period_ms`, `security.mode`, `security.policy`) | The tag's `connector_tags.connector_id` in `connectors` (`server_id` = `connector_id`) |
| `certificate` | The `opc_certificates` document of the connector's `security.client_certificate_id`, when it is `active` and of kind `client` (`connectors_query`'s `$lookup`) |
| Name | The read tag's `tag_name`, else the `connector_tags` one |
Rules:
@@ -180,14 +198,16 @@ Rules:
- **`frequency`:** the lowest among the pipelines of the address, as an integer. Every read tag needs one.
- **`tag_id`:** required, an integer like in `connector_tags`. A read tag whose `tag_id` is missing, `null` or not an integer (`"7"` included) is left out and notified as invalid.
- **`subscription_period_ms`:** the connector's value. A connector without one gets half its lowest tag frequency in the slot.
- **`use_security`:** the connector's value. The certificate comes from MinIO on the gateway, so the slot carries no certificate path.
- **`security`:** the connector's `mode` and `policy`, the source of the OPC security (`use_security` and `server_uri` no longer exist). A connector without them is left out like an incomplete one.
- **`certificate`:** `certificate_id`, `application_uri` (the SAN URI of the certificate, used as the client application URI) and the `bucket`/`object_name` of `cert` and `private_key`; the gateway downloads them from MinIO. The certificate dates, `sha256` and audit fields stay out of the slot. There is no default certificate: `null` means the gateway connects without security, and a server that requires a certificate refuses the connection. A connector whose `client_certificate_id` is set but has no active `client` certificate (missing id, `retired`, another kind or an incomplete document) still goes into the slot with `certificate: null`, and is reported (see below).
- **`active_read`:** off by default. It is on when any pipeline sets `read_tags[].active_read.enabled`, with the lowest `cycle_multiple`.
- **Server keys:** two different connectors with the same name get distinct slot keys (`name`, then `name:server_id`).
- **Slot division:** the addresses are split across the slots, sorted by server and address, and the names of an address always stay in the same slot.
- **Unresolved tags:** a read tag that cannot be resolved is left out of the slot, and the rest of the slot is still written. It is notified once per run:
- `ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the `tag_id` is not an active Read tag, or its document lacks `connector_id`, `address` or `tag_name`;
- `ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the connector is missing or inactive, or lacks `name`, `url` or `server_uri`;
- `ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the connector is missing or inactive, or lacks `name`, `url`, `security.mode` or `security.policy`;
- `ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION`: the read tag has no integer `tag_id`, no integer `frequency`, or an enabled `active_read` without an integer `cycle_multiple`; or its pipeline lacks `schedule_name`, `model_id` or `model.name` (the value is then the `schedule_name`).
- `ORCHESTRATOR_CERTIFICATE_NOT_FOUND_DURING_SLOT_CONFIGURATION` (`WARNING`): the connector's `client_certificate_id` has no active `client` certificate, so its tags go into the slot without a certificate. Most likely a registration error.
- **Only changed slots are written.** The slot is built in a stable order, so unchanged pipelines produce an identical slot, and the gateway does not re-subscribe.
#### Write tags (predictions_batch, through the tag database)
@@ -668,6 +688,112 @@ python -m orchestrator.worker.worker
| `LOG_LEVEL` | Application logging level | `INFO` | No |
| `KAFKA_BOOTSTRAP_SERVERS` | Kafka bootstrap servers | - | No |
### MongoDB Collections
| Collection | Read / written by | Purpose |
|---|---|---|
| `pipelines` | Orchestrator (`pipelines_query`) | One document per Temporal schedule (`schedule_name`, `workflow_type`, `model_id`, `active`, `frequency`, ...) |
| `models` | Orchestrator (`$lookup` in `pipelines_query`) | Gives each pipeline its `model` (`name`, `active`, `model_config`) |
| `connectors` | Orchestrator (`connectors_query`) | Tag database: the OPC UA servers (`connector_id`, `type`, `name`, `url`, `security`, `subscription_period_ms`, `status`) |
| `connector_tags` | Orchestrator (`connector_tags_query`) | Tag database: the tags (`tag_id`, `connector_id`, `address`, `tag_name`, `category`, `data_type`, `status`) |
| `opc_certificates` | Orchestrator (`$lookup` in `connectors_query`) | The client certificate of each connector (`security.client_certificate_id` → `certificate_id`), only when `active` and of kind `client`: `application_uri` and the MinIO objects of `cert` and `private_key` go into the slot |
| `orchestrated_schedules` | Orchestrator (read and written) | Per deployed schedule: `schedule_name`, `namespace`, `updated_at` (last deploy) and `config_hash` |
| `raw_{schedule_name}` | Orchestrator (created) | One per scouter `topic`, where the OPC Gateway writes its tags. Created with a TTL index on `inserted_at` (`MONGODB_TTL_INDEX_HOURS`) |
| `notification_queue` | Alerts / Reports (read) | Notifications (`notification_id`, `trigger`, `level`, `model_id`, `timestamp`, ...), loaded incrementally by `timestamp` |
| `receiver_groups` | Alerts / Reports (read, `active: true`) | Who receives what (see below) |
#### Tags in a pipeline
Pipelines reference tags only by `tag_id` (an integer, as in `connector_tags`). Servers, addresses and data types come from the tag database, never from the pipeline.
Scouter (`scouter` namespace), `workflow_type: "scouter"`:
```json
{
"schedule_name": "scouter-linreg",
"workflow_type": "scouter",
"model_id": "1",
"active": true,
"frequency": "1m",
"read_tags": [
{
"tag_id": 7,
"frequency": 15000,
"tag_name": "Pressure",
"aggr_func": "avg",
"data_range": [0, 100],
"active_read": {"enabled": true, "cycle_multiple": 2}
}
]
}
```
| Field | Required | Meaning |
|---|---|---|
| `tag_id` | Yes | An active `Read` tag of `connector_tags` |
| `frequency` | Yes | Read period in ms, an integer. The slot uses the lowest among the pipelines of the address |
| `tag_name` | No | Overrides the `connector_tags` name: the name the gateway writes and the scouter looks for |
| `aggr_func` | No | Default `lts` |
| `data_range` | No | Default `[-100, 100]` |
| `active_read` | No | Off by default. When `enabled`, `cycle_multiple` must be an integer (default `1`) |
Laborious (`laborious` namespace): only `predictions_batch` and `xgboost_predictions_batch` reference tags; `minimal_retrain`, `drift` and `simple_metrics` have none.
```json
{
"schedule_name": "pred-linreg",
"workflow_type": "predictions_batch",
"model_id": "1",
"active": true,
"frequency": "1m",
"query": "SELECT ...",
"write_tags": [{"tag_id": 101}, {"tag_id": 102}]
}
```
Each `write_tags` entry is only a `tag_id`, of an active `Write` (prediction) or `Prediction Confidence` tag. The Orchestrator builds `opc_output_config` from them (see "Write tags" above) and overwrites any `opc_output_config` stored in the pipeline.
The `connector_tags` documents those pipelines reference:
```json
{"tag_id": 7, "connector_id": "plant", "address": "ns=3;i=1001", "tag_name": "Pressure", "category": "Read", "status": "Active"}
{"tag_id": 101, "connector_id": "plant", "address": "ns=3;i=2001", "tag_name": "PRED-101", "category": "Write", "data_type": "double", "status": "Active"}
{"tag_id": 102, "connector_id": "plant", "address": "ns=3;i=2002", "tag_name": "CONF-101", "category": "Prediction Confidence", "status": "Active"}
```
`pi_web_api_scouter` (`scouter` namespace) does not use the tag database: its `read_tags` is an object keyed by tag name, and it is not part of the OPC slot:
```json
{
"workflow_type": "pi_web_api_scouter",
"read_tags": {"Pressure": {"webid": "F1DP...", "aggr_func": "lts", "data_range": [0, 100]}},
"pi_web_api_config": {"endpoint": "https://pi/piwebapi", "period": "*-1d", "max_count": 1, "api_timeout": 30}
}
```
#### Receiver groups
```json
{
"group_name": "admin_team",
"active": true,
"members": ["admin@example.com"],
"contents": ["core_alerts", "persistent_alerts", "reports"],
"levels": ["ERROR", "WARNING"],
"model_ids": ["1"],
"ignore_models_id": ["2"],
"ignore": ["SOME_NOTIFICATION_ID"]
}
```
| Field | Meaning |
|---|---|
| `contents` | What the group gets: `core_alerts` / `persistent_alerts` (Alerts workflow), `reports` (Reports workflow) |
| `levels` | Optional allow-list of notification levels. Absent = every level the workflow loads |
| `model_ids` | Optional allow-list of `model_id`s |
| `ignore_models_id` | Optional deny-list of `model_id`s (applies together with `model_ids`) |
| `ignore` | Optional deny-list of `notification_id`s |
### Workflow Configuration
Temporal input configuration sample:
@@ -695,7 +821,7 @@ Temporal input configuration sample:
},
{
"$addFields": {
"models": {
"model": {
"$arrayElemAt": [
"$model_docs",
0
@@ -705,7 +831,7 @@ Temporal input configuration sample:
},
{
"$match": {
"models.active": True
"model.active": True
}
},
{
@@ -719,8 +845,19 @@ Temporal input configuration sample:
"collection": "connectors",
"aggregation": [
{"$match": {"type": "opc_ua", "status": "Active"}},
{"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "server_uri": 1,
"use_security": 1, "subscription_period_ms": 1}}
{"$lookup": {"from": "opc_certificates",
"let": {"certificate_id": "$security.client_certificate_id"},
"pipeline": [
{"$match": {"$expr": {"$eq": ["$certificate_id", "$$certificate_id"]},
"status": "active", "kind": "client"}},
{"$project": {"_id": 0, "certificate_id": 1, "application_uri": 1,
"cert.bucket": 1, "cert.object_name": 1,
"private_key.bucket": 1, "private_key.object_name": 1}}
],
"as": "certificates"}},
{"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "subscription_period_ms": 1,
"security.mode": 1, "security.policy": 1,
"security.client_certificate_id": 1, "certificates": 1}}
]
},
"connector_tags_query": {
@@ -735,6 +872,8 @@ Temporal input configuration sample:
`connectors_query` and `connector_tags_query` are optional: the values above are the defaults in the code (see "OPC slots" and "Write tags" above).
The `$lookup` must end in a `model` field (singular): every pipeline type reads `model.name` (and `model.model_config`, when present), so a pipeline without `model` fails to build its schedule.
#### Simple metrics: parallel windows & adaptive frequency
One `pipelines` doc per window, all with the same `model_id`. Name them `<model>-simple_metrics-<interval_minutes>m`; each doc becomes its own Temporal schedule. Thresholds are shared through `model.model_config`. The adaptive state is per doc.