SIENTIAPDE-1974

Resolve PI Web API pipeline tags through the tag database

- Load active `pi_web_api` connectors and referenced tags behind the
  `connectors-pi-web-api` patch, so replays of older runs are unaffected.
- Resolve `pi_web_api_scouter` read tags and `predictions_batch` output
  tags by `tag_id`; legacy `webid` pipelines produce identical input.
- Add a `connection` block to scouter schedules with the secret reference
  only, plus connector defaults for period, max_count and api_timeout.
- Hold pipelines whose tags span several connectors and notify unresolved
  tags, wrong categories and missing connectors once per run.
- Share the issue notification logic between slots and schedules.
This commit is contained in:
2026-10-28 15:46:09 -03:00
parent 744ca647f7
commit 7e6f1a8782
7 changed files with 1170 additions and 57 deletions

View File

@@ -97,7 +97,7 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
- **Infrastructure Management**: Creates, updates, and deletes workflow schedules
**Execution Flow**:
1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`)
1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`, OPC UA for the slot and PI Web API for the schedules)
2. **Parallel Processing**: Formats orchestrated schedules and processes new schedules and slots
3. **Parallel Config Creation**: Creates schedule and slot action configurations, normalizes schedules, and creates collections with TTL indexes
4. **Parallel Operations**: Executes slot deletions, slot updates, schedule deletions, schedule creations, and schedule updates concurrently
@@ -196,6 +196,50 @@ Rules:
**Known limitation:** scouter schedules are only redeployed when the pipeline's `updated_at` changes. If a tag is renamed in `connector_tags`, the slot picks up the new name on the next run, but the scouter keeps the old `model_tags` until the pipeline is updated. When renaming, also update the pipeline, or keep the name in `read_tags[].tag_name`.
#### PI Web API through the tag database
The tag database also holds PI Web API servers: `connectors` documents with `type: "pi_web_api"`, whose `url` is the PI Web API base URL and whose `pi_web_api` block carries `auth` (`type`: `basic` | `bearer`, `secret`: `{name, key}` of a Kubernetes secret, never the credential) and optional connection defaults (`period`, `max_count`, `api_timeout`, `data_server_webid`). Their tags are ordinary `connector_tags` whose `address` is the PI WebID. The slot never reads them: they only feed `process_schedules`.
The workflow loads them behind the `connectors-pi-web-api` patch, with two more optional overrides (defaults in `orchestrator/workflows/orchestrator.py`):
```json
{
"pi_connectors_query": {
"collection": "connectors",
"aggregation": [
{"$match": {"type": "pi_web_api", "status": "Active"}},
{"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "pi_web_api": 1}}
]
},
"pi_connector_tags_query": {
"collection": "connector_tags",
"aggregation": [
{"$match": {"status": "Active"}},
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1, "category": 1}}
]
}
}
```
As for OPC, they are read only when a PI Web API tag carries a `tag_id`, and the workflow puts the referenced `tag_id`s in front of `pi_connector_tags_query`. Tags of every category are loaded, since `predictions_batch` writes to `Write` and `Prediction Confidence` tags.
How the PI Web API pipelines are resolved:
| Pipeline field | With `tag_id` | Without `tag_id` |
|---|---|---|
| `pi_web_api_scouter` `read_tags` (`{model_tag: {...}}`) | `webid` = the `Read` tag's `address`; the key, `aggr_func` and `data_range` stay the pipeline's | `webid` from the pipeline, as before |
| `predictions_batch` `pi_web_api_output_config.prediction_tags` / `confidence_tags` given as a list | `{tag_name: address}` of the `Write` / `Prediction Confidence` tag | A mapping already given as `{tag_name: webid}` passes through unchanged |
Rules:
- **Connection.** When a `pi_web_api_scouter` resolves at least one tag through the tag database, its schedule input gets `connection: {connector_id, base_url, auth: {type, secret: {name, key}}}` (plus `data_server_webid` when set), and the connector's `period`, `max_count` and `api_timeout` become defaults under the pipeline's `pi_web_api_config`: pipeline values win, and `api_timeout` is still capped at the frequency. `connection` is additive: the scouter keeps reading `pi_web_api_query` until its own card. Only the secret reference is copied; the orchestrator never resolves the secret.
- **Legacy pipelines are unchanged.** Without any `tag_id`, the schedule input is byte-identical to before.
- **One connector per pipeline.** A `pi_web_api_scouter` whose tags belong to two connectors is held: its existing schedule is neither updated nor deleted (and none is created), and the conflict is notified.
- **Unresolved tags** are left out, the rest of the pipeline is still scheduled, and they are notified once per run (values are `schedule_name/tag_id`):
- `ORCHESTRATOR_PI_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the `tag_id` is not an active tag, or its document lacks `connector_id`, `address` or `tag_name`;
- `ORCHESTRATOR_PI_TAG_WRONG_CATEGORY_DURING_SCHEDULE_CONFIGURATION`: the tag's category does not fit (`Read` for read tags, `Write` for `prediction_tags`, `Prediction Confidence` for `confidence_tags`);
- `ORCHESTRATOR_PI_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the tag's connector is not an active `pi_web_api` connector with `url` and `pi_web_api.auth` (`type`, `secret.name`, `secret.key`);
- `ORCHESTRATOR_PI_MIXED_CONNECTORS_DURING_SCHEDULE_CONFIGURATION`: the held pipelines above.
#### Architecture
```mermaid