SIENTIAPDE-1974
Resolve PI Web API pipeline tags through the tag database - Load active `pi_web_api` connectors and referenced tags behind the `connectors-pi-web-api` patch, so replays of older runs are unaffected. - Resolve `pi_web_api_scouter` read tags and `predictions_batch` output tags by `tag_id`; legacy `webid` pipelines produce identical input. - Add a `connection` block to scouter schedules with the secret reference only, plus connector defaults for period, max_count and api_timeout. - Hold pipelines whose tags span several connectors and notify unresolved tags, wrong categories and missing connectors once per run. - Share the issue notification logic between slots and schedules.
This commit is contained in:
46
README.md
46
README.md
@@ -97,7 +97,7 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
|
||||
- **Infrastructure Management**: Creates, updates, and deletes workflow schedules
|
||||
|
||||
**Execution Flow**:
|
||||
1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`)
|
||||
1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`, OPC UA for the slot and PI Web API for the schedules)
|
||||
2. **Parallel Processing**: Formats orchestrated schedules and processes new schedules and slots
|
||||
3. **Parallel Config Creation**: Creates schedule and slot action configurations, normalizes schedules, and creates collections with TTL indexes
|
||||
4. **Parallel Operations**: Executes slot deletions, slot updates, schedule deletions, schedule creations, and schedule updates concurrently
|
||||
@@ -196,6 +196,50 @@ Rules:
|
||||
|
||||
**Known limitation:** scouter schedules are only redeployed when the pipeline's `updated_at` changes. If a tag is renamed in `connector_tags`, the slot picks up the new name on the next run, but the scouter keeps the old `model_tags` until the pipeline is updated. When renaming, also update the pipeline, or keep the name in `read_tags[].tag_name`.
|
||||
|
||||
#### PI Web API through the tag database
|
||||
|
||||
The tag database also holds PI Web API servers: `connectors` documents with `type: "pi_web_api"`, whose `url` is the PI Web API base URL and whose `pi_web_api` block carries `auth` (`type`: `basic` | `bearer`, `secret`: `{name, key}` of a Kubernetes secret, never the credential) and optional connection defaults (`period`, `max_count`, `api_timeout`, `data_server_webid`). Their tags are ordinary `connector_tags` whose `address` is the PI WebID. The slot never reads them: they only feed `process_schedules`.
|
||||
|
||||
The workflow loads them behind the `connectors-pi-web-api` patch, with two more optional overrides (defaults in `orchestrator/workflows/orchestrator.py`):
|
||||
|
||||
```json
|
||||
{
|
||||
"pi_connectors_query": {
|
||||
"collection": "connectors",
|
||||
"aggregation": [
|
||||
{"$match": {"type": "pi_web_api", "status": "Active"}},
|
||||
{"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "pi_web_api": 1}}
|
||||
]
|
||||
},
|
||||
"pi_connector_tags_query": {
|
||||
"collection": "connector_tags",
|
||||
"aggregation": [
|
||||
{"$match": {"status": "Active"}},
|
||||
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1, "category": 1}}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
As for OPC, they are read only when a PI Web API tag carries a `tag_id`, and the workflow puts the referenced `tag_id`s in front of `pi_connector_tags_query`. Tags of every category are loaded, since `predictions_batch` writes to `Write` and `Prediction Confidence` tags.
|
||||
|
||||
How the PI Web API pipelines are resolved:
|
||||
|
||||
| Pipeline field | With `tag_id` | Without `tag_id` |
|
||||
|---|---|---|
|
||||
| `pi_web_api_scouter` `read_tags` (`{model_tag: {...}}`) | `webid` = the `Read` tag's `address`; the key, `aggr_func` and `data_range` stay the pipeline's | `webid` from the pipeline, as before |
|
||||
| `predictions_batch` `pi_web_api_output_config.prediction_tags` / `confidence_tags` given as a list | `{tag_name: address}` of the `Write` / `Prediction Confidence` tag | A mapping already given as `{tag_name: webid}` passes through unchanged |
|
||||
|
||||
Rules:
|
||||
- **Connection.** When a `pi_web_api_scouter` resolves at least one tag through the tag database, its schedule input gets `connection: {connector_id, base_url, auth: {type, secret: {name, key}}}` (plus `data_server_webid` when set), and the connector's `period`, `max_count` and `api_timeout` become defaults under the pipeline's `pi_web_api_config`: pipeline values win, and `api_timeout` is still capped at the frequency. `connection` is additive: the scouter keeps reading `pi_web_api_query` until its own card. Only the secret reference is copied; the orchestrator never resolves the secret.
|
||||
- **Legacy pipelines are unchanged.** Without any `tag_id`, the schedule input is byte-identical to before.
|
||||
- **One connector per pipeline.** A `pi_web_api_scouter` whose tags belong to two connectors is held: its existing schedule is neither updated nor deleted (and none is created), and the conflict is notified.
|
||||
- **Unresolved tags** are left out, the rest of the pipeline is still scheduled, and they are notified once per run (values are `schedule_name/tag_id`):
|
||||
- `ORCHESTRATOR_PI_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the `tag_id` is not an active tag, or its document lacks `connector_id`, `address` or `tag_name`;
|
||||
- `ORCHESTRATOR_PI_TAG_WRONG_CATEGORY_DURING_SCHEDULE_CONFIGURATION`: the tag's category does not fit (`Read` for read tags, `Write` for `prediction_tags`, `Prediction Confidence` for `confidence_tags`);
|
||||
- `ORCHESTRATOR_PI_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the tag's connector is not an active `pi_web_api` connector with `url` and `pi_web_api.auth` (`type`, `secret.name`, `secret.key`);
|
||||
- `ORCHESTRATOR_PI_MIXED_CONNECTORS_DURING_SCHEDULE_CONFIGURATION`: the held pipelines above.
|
||||
|
||||
#### Architecture
|
||||
|
||||
```mermaid
|
||||
|
||||
Reference in New Issue
Block a user