gitea_admin/sientia-dataops-orchestrator_temporal
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

SIENTIAPDE-1974

Browse Source 
SIENTIAPDE-1974-connectors-pi-web-api: Consolidate OPC server and tag data sourcing to the tag database. This change removes the legacy opc-servers collection and enforces tag_id as the sole reference for all read and write tags via connectors and connector_tags. The Orchestrator workflow now exclusively loads OPC server and tag information from the tag database, simplifying data flow and removing conditional logic for legacy data sources. Improved error reporting for pipeline timestamp updates and updated documentation, examples, and tests are also included.
This commit is contained in:
Vitor Pimentel dos Santos
2026-10-29 10:25:51 -03:00
parent 9ba2fc74c4
commit 0c9eaab3f5
30 changed files with 1082 additions and 3051 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
README.md
View File

@@ -97,7 +97,7 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
- **Infrastructure Management**: Creates, updates, and deletes workflow schedules
**Execution Flow**:
1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`, for the slot and the schedules)
1. **Parallel Data Loading**: Concurrently loads pipelines, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`, the only source of the OPC servers and tags of the slot and the schedules)
2. **Parallel Processing**: Formats orchestrated schedules and processes new schedules and slots
3. **Parallel Config Creation**: Creates schedule and slot action configurations, normalizes schedules, and creates collections with TTL indexes
4. **Parallel Operations**: Executes slot deletions, slot updates, schedule deletions, schedule creations, and schedule updates concurrently
@@ -113,10 +113,6 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
{"$match": {"active": true}},
{"$sort": {"updated_at": -1}}
]
},
"opc_servers_query": {
"collection": "opc_servers",
"filters": {"active": true}
}
}
```
@@ -137,15 +133,15 @@ Optional: `connectors_query` and `connector_tags_query` override the tag databas
"collection": "connector_tags",
"aggregation": [
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1,
"category": 1, "data_type": 1, "status": 1, "updated_at": 1}}
"category": 1, "data_type": 1, "status": 1}}
]
}
}
```
The tag database is read only when a scouter read tag or a `predictions_batch` write tag carries a `tag_id`, and the workflow puts `{"$match": {"tag_id": {"$in": [...]}}}` in front of `connector_tags_query`, so it loads only the referenced tags. Environments without a tag database, or whose pipelines do not use `tag_id`, never query it. An override should keep a `$project` of these fields: any other attribute may hold a BSON type a Temporal payload cannot carry (such as `ObjectId`). `updated_at` is turned into text by the load.
The workflow puts `{"$match": {"tag_id": {"$in": [...]}}}` in front of `connector_tags_query`, so it loads only the tags the scouter read tags and `predictions_batch` write tags reference. When no pipeline references a `tag_id`, the tag database is not queried. An override should keep a `$project` of these fields: any other attribute may hold a BSON type a Temporal payload cannot carry (such as `ObjectId`).
The referenced tags are loaded whatever their category and status: the code uses only the active ones (`Read` for the slot and the scouter names, `Write` / `Prediction Confidence` for the write tags), and every referenced tag's `updated_at` redeploys its pipelines (see "Redeploy on tag database changes" below). A document without `status` counts as active.
The referenced tags are loaded whatever their category and status: the code uses only the active ones (`Read` for the slot and the scouter names, `Write` / `Prediction Confidence` for the write tags), and a change to a referenced tag that reaches a schedule redeploys it through its `config_hash` (see "Redeploy on changes" below). A document without `status` counts as active.
#### OPC slots (slot v2, contract C1)
@@ -170,31 +166,29 @@ The referenced tags are loaded whatever their category and status: the code uses
}
```
How each scouter `read_tags` entry is resolved:
Each scouter `read_tags` entry references its tag by `tag_id` (`{"tag_id": 7, "frequency": 15000}`). `connectors` is the only source of the OPC servers:
| Read tag | Address and server | Name |
|---|---|---|
| With `tag_id` | `connector_tags` (`address`, `connector_id`) and `connectors` (`url`, `server_uri`, `use_security`, `subscription_period_ms`) | The read tag's `tag_name`, else the `connector_tags` one |
| Without `tag_id` | `server_id` + `tag_address`, server from `opc-servers` | The read tag's `tag_name` |
| Field | Source |
|---|---|
| Address | `connector_tags.address` |
| Server (`server_id`, `name`, `url`, `server_uri`, `use_security`, `subscription_period_ms`) | The tag's `connector_tags.connector_id` in `connectors` (`server_id` = `connector_id`) |
| Name | The read tag's `tag_name`, else the `connector_tags` one |
Rules:
- **One entry per address.** An address read by N pipelines is read once and written to N collections: one `names` item per pipeline (`collection` = `raw_{schedule_name}`).
- **Same name source as the scouter.** `resolve_tag_names` names both the slot's `names` and the scouter's `model_tags`, so the gateway always writes the name the scouter looks for.
- **`frequency`:** the lowest among the pipelines of the address, as an integer. Every read tag needs one, with or without `tag_id`.
- **`tag_id`:** a missing, `null` or empty `tag_id` means the read tag does not use the tag database.
- **`subscription_period_ms`:** the connector's value. For `opc-servers`, it is half the server's lowest tag frequency in the slot, as before.
- **`use_security`:** the connector's value. For `opc-servers`, it is `true` when the server has a `cert_path`. The certificate itself comes from MinIO on the gateway, so the slot carries no certificate path.
- **`frequency`:** the lowest among the pipelines of the address, as an integer. Every read tag needs one.
- **`tag_id`:** required, an integer like in `connector_tags`. A read tag whose `tag_id` is missing, `null` or not an integer (`"7"` included) is left out and notified as invalid.
- **`subscription_period_ms`:** the connector's value. A connector without one gets half its lowest tag frequency in the slot.
- **`use_security`:** the connector's value. The certificate comes from MinIO on the gateway, so the slot carries no certificate path.
- **`active_read`:** off by default. It is on when any pipeline sets `read_tags[].active_read.enabled`, with the lowest `cycle_multiple`.
- **Same `server_id` in both sources:** a server known through `connectors` and `opc-servers` under the same id is one server, and the `connectors` fields win. Two different servers with the same name get distinct slot keys (`name`, then `name:server_id`).
- **Server keys:** two different connectors with the same name get distinct slot keys (`name`, then `name:server_id`).
- **Slot division:** the addresses are split across the slots, sorted by server and address, and the names of an address always stay in the same slot.
- **Unresolved tags:** a read tag that cannot be resolved is left out of the slot, and the rest of the slot is still written. It is notified once per run:
- `ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the `tag_id` is not an active Read tag, or its document lacks `connector_id`, `address` or `tag_name`;
- `ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the connector is missing or inactive, or lacks `name`, `url` or `server_uri`;
- `ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the `server_id` is not in `opc-servers`;
- `ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION`: the read tag has no integer `frequency`, has an enabled `active_read` without an integer `cycle_multiple` or, without `tag_id`, lacks `server_id`, `tag_address` or `tag_name`; or its pipeline lacks `schedule_name`, `model_id` or `model.name` (the value is then the `schedule_name`).
- **Only changed slots are written.** The slot is built in a stable order, so unchanged pipelines produce an identical slot, and the gateway does not re-subscribe. A slot still in the v1 format (`topics`, `cert_path`) is replaced once.
**Gateway plan B:** the gateway's legacy read engine (`READ_ENGINE=legacy`, kept until the end of SIENTIAPDE-1974) reads the slot v1 only. Once this version writes the slot v2, switching the gateway to the legacy engine also requires rolling the Orchestrator back to a version that writes v1.
- `ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION`: the read tag has no integer `tag_id`, no integer `frequency`, or an enabled `active_read` without an integer `cycle_multiple`; or its pipeline lacks `schedule_name`, `model_id` or `model.name` (the value is then the `schedule_name`).
- **Only changed slots are written.** The slot is built in a stable order, so unchanged pipelines produce an identical slot, and the gateway does not re-subscribe.
#### Write tags (predictions_batch, through the tag database)
@@ -219,10 +213,10 @@ A `predictions_batch` (or `xgboost_predictions_batch`) references its write tags
| Address | `connector_tags.address` |
| `data_type` | `connector_tags.data_type` (`float`, `double`, `real`, `int`, `integer`, `bool`, `boolean`, `str`, `string`), else `float` |
The legacy write tags (`server_id`, `type`, `addr`, `data_type` in the pipeline) are no longer read: migrate them to `tag_id` before deploying this version (see "Deploying the tag database write tags" below). A write tag that cannot be resolved is left out, the rest of the pipeline is still scheduled, and it is notified once per run (values are `schedule_name/tag_id`):
A write tag that cannot be resolved is left out, the rest of the pipeline is still scheduled, and it is notified once per run (values are `schedule_name/tag_id`):
- `ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the `tag_id` is not an active `Write` / `Prediction Confidence` tag, or its document lacks `connector_id`, `address`, `tag_name` or `category`;
- `ORCHESTRATOR_WRITE_TAG_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the tag's connector is missing, inactive or incomplete in `connectors`;
- `ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION`: the entry has no `tag_id` (a legacy write tag), or the tag's `data_type` is not one the gateway accepts.
- `ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION`: the entry has no integer `tag_id`, or the tag's `data_type` is not one the gateway accepts.
#### Redeploy on changes (`updated_at` and `config_hash`)
@@ -233,10 +227,10 @@ The legacy write tags (`server_id`, `type`, `addr`, `data_type` in the pipeline)
A change that does not reach the schedule (e.g. a tag's `description`) does not redeploy. The slot is compared by content and follows on its own. A pipeline without `updated_at` is still redeployed on every run. Schedules deployed before `config_hash` existed have none stored, so the first run of this version redeploys all of them once and records it.
#### Deploying the tag database write tags
#### Deploying
1. Run the migration that adds `data_type` to the `connector_tags` validator (`sientia-dataops-database-migrations`).
2. Make sure every `predictions_batch` / `xgboost_predictions_batch` declares its write tags as `[{tag_id}]`, with each tag in `connector_tags` (category `Write` or `Prediction Confidence`, and its `data_type`). Entries still in the legacy form (`server_id` / `type` / `addr`) are left out and notified. Convert them together with the deploy: the previous version cannot read `[{tag_id}]`.
1. Run the migrations that create the tag database and add `data_type` to the `connector_tags` validator (`sientia-dataops-database-migrations`).
2. Every scouter `read_tags` and every `predictions_batch` / `xgboost_predictions_batch` `write_tags` entry is a `tag_id` of `connector_tags`, as an integer. Anything else is left out and notified.
3. Deploy this version. Its first run redeploys every schedule once, to record their `config_hash` (see above).
#### Architecture
@@ -245,7 +239,7 @@ A change that does not reach the schedule (e.g. a tag's `description`) does not
flowchart LR
subgraph "Parallel Data Loading"
A[1. aggregate_documents_in_mongodb]
B[2. find_documents_in_mongodb<br/>OPC Servers]
B[2. aggregate_documents_in_mongodb<br/>connectors + connector_tags]
C[3. find_documents_in_mongodb<br/>Orchestrated Schedules]
D[4. load_opc_slots]
E[5. load_active_ingestors]
@@ -721,12 +715,6 @@ Temporal input configuration sample:
}
]
},
"opc_servers_query": {
"collection": "opc-servers",
"filters": {
}
},
"connectors_query": {
"collection": "connectors",
"aggregation": [
@@ -739,7 +727,7 @@ Temporal input configuration sample:
"collection": "connector_tags",
"aggregation": [
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1,
"category": 1, "data_type": 1, "status": 1, "updated_at": 1}}
"category": 1, "data_type": 1, "status": 1}}
]
}
}