gitea_admin/sientia-dataops-orchestrator_temporal
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

SIENTIAPDE-1974

Browse Source 
SIENTIAPDE-1974  Write OPC tags through the tag database and drop the PI Web API path

- predictions_batch write_tags are now [{tag_id}] only: resolve_write_tags builds
  the opc_output_config from connector_tags + connectors (server = connector_id,
  Write / Prediction Confidence -> prediction_tags / confidence_tags, address and
  data_type from the tag, float by default). The legacy server_id/type/addr write
  tags are no longer read and are notified as invalid.
- A pipeline's effective updated_at is the latest of its own and of the
  connector_tags it references, so renaming, moving or deactivating a tag
  redeploys it (removes the rename limitation).
- The connector_tags query loads the referenced tags of every category and
  status; the code filters them. aggregate_documents_in_mongodb formats only
  datetime timestamp fields.
- Remove the PI Web API tag database path (connectors-pi-web-api patch, PI loads,
  resolution, connection and held schedules): PI pipelines behave as on main.
- tag_id is canonical text and the workflow-side id collection never raises;
  a malformed pipeline no longer fails the whole OPC slot.
- Replay test against histories recorded on the e2e stack (main and this
  version), e2e scenarios for write tags and tag changes, and the runbook
  script scripts/sientiapde_1974_backfill_write_tags.js.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Vitor Pimentel dos Santos
2026-10-29 08:25:01 -03:00
parent 9eaec54fc8
commit 2c331b5f16
28 changed files with 4451 additions and 1385 deletions
Download Patch File Download Diff File Expand all files Collapse all files

86
README.md
View File

@@ -97,7 +97,7 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
- **Infrastructure Management**: Creates, updates, and deletes workflow schedules
**Execution Flow**:
1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`, OPC UA for the slot and PI Web API for the schedules)
1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`, for the slot and the schedules)
2. **Parallel Processing**: Formats orchestrated schedules and processes new schedules and slots
3. **Parallel Config Creation**: Creates schedule and slot action configurations, normalizes schedules, and creates collections with TTL indexes
4. **Parallel Operations**: Executes slot deletions, slot updates, schedule deletions, schedule creations, and schedule updates concurrently
@@ -136,14 +136,16 @@ Optional: `connectors_query` and `connector_tags_query` override the tag databas
"connector_tags_query": {
"collection": "connector_tags",
"aggregation": [
{"$match": {"category": "Read", "status": "Active"}},
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1}}
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1,
"category": 1, "data_type": 1, "status": 1, "updated_at": 1}}
]
}
}
```
The tag database is read only when a scouter read tag carries a `tag_id`, and the workflow puts `{"$match": {"tag_id": {"$in": [...]}}}` in front of `connector_tags_query`, so it loads only the referenced tags. Environments without a tag database, or whose pipelines do not use `tag_id`, never query it. An override should keep a `$project` of these fields: any other attribute may hold a BSON type a Temporal payload cannot carry (such as `ObjectId`).
The tag database is read only when a scouter read tag or a `predictions_batch` write tag carries a `tag_id`, and the workflow puts `{"$match": {"tag_id": {"$in": [...]}}}` in front of `connector_tags_query`, so it loads only the referenced tags. Environments without a tag database, or whose pipelines do not use `tag_id`, never query it. An override should keep a `$project` of these fields: any other attribute may hold a BSON type a Temporal payload cannot carry (such as `ObjectId`). `updated_at` is turned into text by the load.
The referenced tags are loaded whatever their category and status: the code uses only the active ones (`Read` for the slot and the scouter names, `Write` / `Prediction Confidence` for the write tags), and every referenced tag's `updated_at` redeploys its pipelines (see "Redeploy on tag database changes" below). A document without `status` counts as active.
#### OPC slots (slot v2, contract C1)
@@ -189,56 +191,48 @@ Rules:
- `ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the `tag_id` is not an active Read tag, or its document lacks `connector_id`, `address` or `tag_name`;
- `ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the connector is missing or inactive, or lacks `name`, `url` or `server_uri`;
- `ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the `server_id` is not in `opc-servers`;
- `ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION`: the read tag has no integer `frequency` or, without `tag_id`, lacks `server_id`, `tag_address` or `tag_name`.
- `ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION`: the read tag has no integer `frequency`, has an enabled `active_read` without an integer `cycle_multiple` or, without `tag_id`, lacks `server_id`, `tag_address` or `tag_name`; or its pipeline lacks `schedule_name`, `model_id` or `model.name` (the value is then the `schedule_name`).
- **Only changed slots are written.** The slot is built in a stable order, so unchanged pipelines produce an identical slot, and the gateway does not re-subscribe. A slot still in the v1 format (`topics`, `cert_path`) is replaced once.
**Gateway plan B:** the gateway's legacy read engine (`READ_ENGINE=legacy`, kept until the end of SIENTIAPDE-1974) reads the slot v1 only. Once this version writes the slot v2, switching the gateway to the legacy engine also requires rolling the Orchestrator back to a version that writes v1.
**Known limitation:** scouter schedules are only redeployed when the pipeline's `updated_at` changes. If a tag is renamed in `connector_tags`, the slot picks up the new name on the next run, but the scouter keeps the old `model_tags` until the pipeline is updated. When renaming, also update the pipeline, or keep the name in `read_tags[].tag_name`.
#### Write tags (predictions_batch, through the tag database)
#### PI Web API through the tag database
The tag database also holds PI Web API servers: `connectors` documents with `type: "pi_web_api"`, whose `url` is the PI Web API base URL and whose `pi_web_api` block carries `auth` (`type`: `basic` | `bearer`, `secret`: `{name, key}` of a Kubernetes secret, never the credential) and optional connection defaults (`period`, `max_count`, `api_timeout`, `data_server_webid`). Their tags are ordinary `connector_tags` whose `address` is the PI WebID. The slot never reads them: they only feed `process_schedules`.
The workflow loads them behind the `connectors-pi-web-api` patch, with two more optional overrides (defaults in `orchestrator/workflows/orchestrator.py`):
A `predictions_batch` (or `xgboost_predictions_batch`) references its write tags by `tag_id` only, exactly like the read tags. Everything else comes from the tag database, so the pipeline does not say which server, address or type a tag has:
```json
{
"pi_connectors_query": {
"collection": "connectors",
"aggregation": [
{"$match": {"type": "pi_web_api", "status": "Active"}},
{"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "pi_web_api": 1}}
]
},
"pi_connector_tags_query": {
"collection": "connector_tags",
"aggregation": [
{"$match": {"status": "Active"}},
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1, "category": 1}}
]
}
}
{"workflow_type": "predictions_batch", "write_tags": [{"tag_id": 101}, {"tag_id": 102}]}
```
As for OPC, they are read only when a PI Web API tag carries a `tag_id`, and the workflow puts the referenced `tag_id`s in front of `pi_connector_tags_query`. Tags of every category are loaded, since `predictions_batch` writes to `Write` and `Prediction Confidence` tags.
`process_schedules` turns them into the `opc_output_config` the Laborious already reads:
How the PI Web API pipelines are resolved:
```json
{"plant": {
"prediction_tags": {"ns=3;i=2001": {"data_type": "double"}},
"confidence_tags": {"ns=3;i=2002": {"data_type": "float"}}}}
```
| Pipeline field | With `tag_id` | Without `tag_id` |
|---|---|---|
| `pi_web_api_scouter` `read_tags` (`{model_tag: {...}}`) | `webid` = the `Read` tag's `address`; the key, `aggr_func` and `data_range` stay the pipeline's | `webid` from the pipeline, as before |
| `predictions_batch` `pi_web_api_output_config.prediction_tags` / `confidence_tags` given as a list | `{tag_name: address}` of the `Write` / `Prediction Confidence` tag | A mapping already given as `{tag_name: webid}` passes through unchanged |
| Field | Source |
|---|---|
| Server key | `connector_tags.connector_id`. The Laborious publishes it as `server_id` on NATS, and the OPC Gateway resolves it in `connectors` |
| `prediction_tags` / `confidence_tags` | `connector_tags.category`: `Write` / `Prediction Confidence` |
| Address | `connector_tags.address` |
| `data_type` | `connector_tags.data_type` (`float`, `double`, `real`, `int`, `integer`, `bool`, `boolean`, `str`, `string`), else `float` |
Rules:
- **Connection.** When a `pi_web_api_scouter` resolves at least one tag through the tag database, its schedule input gets `connection: {connector_id, base_url, auth: {type, secret: {name, key}}}` (plus `data_server_webid` when set), and the connector's `period`, `max_count` and `api_timeout` become defaults under the pipeline's `pi_web_api_config`: pipeline values win, and `api_timeout` is still capped at the frequency. `connection` is additive: the scouter keeps reading `pi_web_api_query` until its own card. Only the secret reference is copied; the orchestrator never resolves the secret.
- **Legacy pipelines are unchanged.** Without any `tag_id`, the schedule input is byte-identical to before.
- **One connector per pipeline.** A `pi_web_api_scouter` whose tags belong to two connectors is held: its existing schedule is neither updated nor deleted (and none is created), and the conflict is notified.
- **Unresolved tags** are left out, the rest of the pipeline is still scheduled, and they are notified once per run (values are `schedule_name/tag_id`):
- `ORCHESTRATOR_PI_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the `tag_id` is not an active tag, or its document lacks `connector_id`, `address` or `tag_name`;
- `ORCHESTRATOR_PI_TAG_WRONG_CATEGORY_DURING_SCHEDULE_CONFIGURATION`: the tag's category does not fit (`Read` for read tags, `Write` for `prediction_tags`, `Prediction Confidence` for `confidence_tags`);
- `ORCHESTRATOR_PI_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the tag's connector is not an active `pi_web_api` connector with `url` and `pi_web_api.auth` (`type`, `secret.name`, `secret.key`);
- `ORCHESTRATOR_PI_MIXED_CONNECTORS_DURING_SCHEDULE_CONFIGURATION`: the held pipelines above.
The legacy write tags (`server_id`, `type`, `addr`, `data_type` in the pipeline) are no longer read: migrate them to `tag_id` before deploying this version (see "Deploying the tag database write tags" below). A write tag that cannot be resolved is left out, the rest of the pipeline is still scheduled, and it is notified once per run (values are `schedule_name/tag_id`):
- `ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the `tag_id` is not an active `Write` / `Prediction Confidence` tag, or its document lacks `connector_id`, `address`, `tag_name` or `category`;
- `ORCHESTRATOR_WRITE_TAG_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the tag's connector is missing, inactive or incomplete in `connectors`;
- `ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION`: the entry has no `tag_id` (a legacy write tag), or the tag's `data_type` is not one the gateway accepts.
#### Redeploy on tag database changes
A schedule is redeployed when its `updated_at` is newer than the last time it was orchestrated. That `updated_at` is the latest of the pipeline's own and of the `updated_at` of every `connector_tags` document it references (read and write tags, active or not). So renaming a read tag, moving or retyping a write tag, or deactivating a tag redeploys the pipelines that use it on the next run, without touching them. The slot is compared by content and follows on its own. A pipeline without `updated_at` is still redeployed on every run. A deleted tag document (as opposed to a deactivated one) is not detected: deactivate tags instead of deleting them.
#### Deploying the tag database write tags
1. Run the migration that adds `data_type` to the `connector_tags` validator (`sientia-dataops-database-migrations`).
2. Run `scripts/sientiapde_1974_backfill_write_tags.js`, first with `DRY_RUN=1` (the default). It creates the `connector_tags` of the legacy write tags, rewrites `write_tags` as `[{tag_id}]` (keeping the original in `legacy_write_tags`) and bumps the pipelines' `updated_at`.
3. Deploy this version.
#### Architecture
@@ -739,14 +733,14 @@ Temporal input configuration sample:
"connector_tags_query": {
"collection": "connector_tags",
"aggregation": [
{"$match": {"category": "Read", "status": "Active"}},
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1}}
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1,
"category": 1, "data_type": 1, "status": 1, "updated_at": 1}}
]
}
}
```
`connectors_query` and `connector_tags_query` are optional: the values above are the defaults in the code (see "OPC slots" above).
`connectors_query` and `connector_tags_query` are optional: the values above are the defaults in the code (see "OPC slots" and "Write tags" above).
#### Simple metrics: parallel windows & adaptive frequency
@@ -960,7 +954,7 @@ orchestrator/
│ ├── connectors_config.py # Database configuration
│ ├── email_builder.py # Email template generation
│ ├── pipeline_configs.py # Schedule input builders per pipeline type
│ ├── tag_database.py # Tag database indexes, OPC names and PI Web API resolution
│ ├── tag_database.py # Tag database indexes, OPC read names and write tags
│ ├── opc_slots.py # OPC slot v2 (contract C1)
│ ├── converters.py # Data type conversion utilities
│ └── templates/ # Email HTML templates