Merge pull request #51 from Aignosi/feature/SIENTIAPDE-1974-connectors-pi-web-api
SIENTIAPDE-1974: OPC tag database for read and write tags (slot v2, write_tags by tag_id)
This commit is contained in:
GitHub
@@ -47,7 +47,8 @@ CI quality gate (`.github/workflows/quality-gate.yml`) calls the shared reusable
|
||||
- **`orchestrator/worker/worker.py`** — entrypoint (`python -m orchestrator.worker.worker`). Builds a single `Activities` instance, connects to Temporal, starts Prometheus (`HTTP_METRICS_PORT`) and Temporal SDK metrics (`HTTP_SDK_METRICS_PORT`) servers, then runs three `Worker`s concurrently via `asyncio.gather`, each on its own task queue: `orchestrator-queue`, `alerts-queue`, `reports-queue`. `prepare_worker.py` derives the queue name from the workflow class name (`camel_to_snake`) and reads per-queue concurrency/poller tuning from env vars prefixed with the workflow name (e.g. `ORCHESTRATOR_MAX_CONCURRENT_ACTIVITIES`).
|
||||
- **`orchestrator/workflows/`** — Temporal workflow definitions: `orchestrator.py` (pipeline/slot/schedule orchestration), `alerts.py` (real-time ERROR notifications), `reports.py` (scheduled all-level reports). `alerts.py` and `reports.py` both delegate to the same two subworkflows in `workflows/subworkflows/`: `load_notification_package.py` (load notifications + receiver configs from Mongo, incremental via Redis timestamp) and `process_notifications.py` (build HTML email, send, log to Postgres). The difference between alerts and reports is which filter activity they call (`filter_notification_alerts` vs `filter_notification_reports`) and their base data filter (`level: ERROR` by default vs all levels) — `load_notification_package.py`'s `_compute_query_time_filter` can widen either beyond that default to the union of active receiver groups' own `levels` (SIENTIAPDE-2011); the ERROR-only default only holds when no active group narrows further. Receiver groups also support a `model_ids` allow-list (SIENTIAPDE-2149), independent of the pre-existing `ignore_models_id` deny-list — both apply together (AND) when a group configures both, and `model_ids` follows the same union-at-query-time / precise-per-group-slice split as `levels`. `load_latest_data`'s Mongo query is capped at `NOTIFICATION_QUEUE_BATCH_SIZE` documents per run (default `0`, meaning unlimited — unchanged from pre-SIENTIAPDE-2013 behavior unless a deployment opts in), sorted by `timestamp` ascending so a capped run always returns the oldest pending batch first (SIENTIAPDE-2013) — a hard cap with no drain loop, since the incremental Redis cursor picks up any remainder on the next run.
|
||||
- **`orchestrator/activities/activities.py`** — `Activities` is a single class combining every activity mixin via multiple inheritance: `TemporalManager, SlotManager, Formatters, MongoDB, Email, Postgres` (Postgres comes from `sientia_do.temporal.activities.postgres`, the external `sientia-dataops-library` dependency). One instance is constructed in `worker.py` and its bound methods are handed to each `Worker` as the activity list — activities are plain methods on this shared object, not separate classes. Each mixin's `__init__` takes its own config dict plus shared `logger`, `notification_handler`, `metrics_controller`. `Activities.shutdown()` closes every mixin's connections.
|
||||
- **`orchestrator/utils/`** — `connectors_config.py` builds each config dict from env vars (Temporal/Redis/MongoDB/Postgres/Email); `email_builder.py` renders Jinja2 templates (`utils/templates/email_template.html`, `general_template.html`); `orchestrator_functions.py` transforms pipeline documents into pipeline-type-specific configs (`scouter`, `pi_web_api_scouter`, `predictions_batch`, `minimal_retrain`, `drift`, `simple_metrics`); `converters.py` handles Temporal schedule frequency parsing.
|
||||
- **`orchestrator/utils/`** — `connectors_config.py` builds each config dict from env vars (Temporal/Redis/MongoDB/Postgres/Email); `email_builder.py` renders Jinja2 templates (`utils/templates/email_template.html`, `general_template.html`); `pipeline_configs.py` transforms pipeline documents into pipeline-type-specific configs (`scouter`, `pi_web_api_scouter`, `predictions_batch`, `minimal_retrain`, `drift`, `simple_metrics`); `tag_database.py` indexes the tag database (`connectors` + `connector_tags`) and resolves tags by `tag_id` (scouter read tag names and the `predictions_batch` write tags → `opc_output_config` keyed by `connector_id`, via `resolve_write_tags`; tag changes reach the schedules through `config_hash`, see below); `opc_slots.py` builds the OPC slot v2 the gateway reads (`gather_read_tags` / `build_tag_config`, contract C1: one entry per address with a `names[]` list, and per server `security` {mode, policy} plus the `certificate` from `tag_database.resolve_certificate`; see the README section "OPC slots"). The certificate comes from `opc_certificates` through the `$lookup` in `DEFAULT_CONNECTORS_QUERY` (only `active` + `client`); no certificate means the gateway connects without security, and a `client_certificate_id` without one is a `WARNING` (SIENTIAPDE-1974, D5). `resolve_tag_names` (in `tag_database.py`) is the single source of tag names for both the scouter's `model_tags` and the slot's `names` (SIENTIAPDE-1974 SC1); `converters.py` handles Temporal schedule frequency parsing.
|
||||
- **Schedule redeploys** — `orchestrated_schedules` stores, per schedule, the last deploy time (`updated_at`) and the `config_hash` it was deployed with (`formatters.schedule_config_hash`: SHA-256 of the built schedule without `updated_at`). `compare_config_timestamps` redeploys when the pipeline's `updated_at` is newer (manual force) or the hash differs (any content change, including tag database edits/deletes made without bumping `updated_at`). `create_schedules`/`update_schedules` pop `config_hash` before sending the schedule to Temporal and return it in their report for `create_pipelines_timestamps`/`update_pipelines_timestamps`.
|
||||
|
||||
### Temporal-specific conventions (all workflow/activity files follow this)
|
||||
- Every workflow and activity-consuming module wraps non-deterministic/side-effecting imports in `with workflow.unsafe.imports_passed_through(): ...` right after `from temporalio import workflow`, because Temporal's sandbox restricts what workflow code can import directly. When adding a new workflow file, follow the existing pattern (see `orchestrator/workflows/orchestrator.py`, `orchestrator/activities/activities.py`) rather than importing at module top level.
|
||||
|
||||
294
README.md
294
README.md
@@ -97,13 +97,13 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
|
||||
- **Infrastructure Management**: Creates, updates, and deletes workflow schedules
|
||||
|
||||
**Execution Flow**:
|
||||
1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, and active ingestors
|
||||
1. **Parallel Data Loading**: Concurrently loads pipelines, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`, the only source of the OPC servers and tags of the slot and the schedules)
|
||||
2. **Parallel Processing**: Formats orchestrated schedules and processes new schedules and slots
|
||||
3. **Parallel Config Creation**: Creates schedule and slot action configurations, normalizes schedules, and creates collections with TTL indexes
|
||||
4. **Parallel Operations**: Executes slot deletions, slot updates, schedule deletions, schedule creations, and schedule updates concurrently
|
||||
5. **Parallel Reports & Timestamps**: Generates orchestration reports and updates MongoDB timestamps for created/updated/deleted pipelines
|
||||
|
||||
**Input Parameters**:
|
||||
**Input Parameters** (minimal: each pipeline must already carry `model.name`; see "Workflow Configuration" for the `$lookup` on `models` that adds it):
|
||||
```json
|
||||
{
|
||||
"schedule_name": "hourly_orchestration",
|
||||
@@ -113,21 +113,153 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
|
||||
{"$match": {"active": true}},
|
||||
{"$sort": {"updated_at": -1}}
|
||||
]
|
||||
},
|
||||
"opc_servers_query": {
|
||||
"collection": "opc_servers",
|
||||
"filters": {"active": true}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Optional: `connectors_query` and `connector_tags_query` override the tag database aggregations. The defaults live in the code (`orchestrator/workflows/orchestrator.py`), so the schedule input does not need them:
|
||||
|
||||
```json
|
||||
{
|
||||
"connectors_query": {
|
||||
"collection": "connectors",
|
||||
"aggregation": [
|
||||
{"$match": {"type": "opc_ua", "status": "Active"}},
|
||||
{"$lookup": {"from": "opc_certificates",
|
||||
"let": {"certificate_id": "$security.client_certificate_id"},
|
||||
"pipeline": [
|
||||
{"$match": {"$expr": {"$eq": ["$certificate_id", "$$certificate_id"]},
|
||||
"status": "active", "kind": "client"}},
|
||||
{"$project": {"_id": 0, "certificate_id": 1, "application_uri": 1,
|
||||
"cert.bucket": 1, "cert.object_name": 1,
|
||||
"private_key.bucket": 1, "private_key.object_name": 1}}
|
||||
],
|
||||
"as": "certificates"}},
|
||||
{"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "subscription_period_ms": 1,
|
||||
"security.mode": 1, "security.policy": 1,
|
||||
"security.client_certificate_id": 1, "certificates": 1}}
|
||||
]
|
||||
},
|
||||
"connector_tags_query": {
|
||||
"collection": "connector_tags",
|
||||
"aggregation": [
|
||||
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1,
|
||||
"category": 1, "data_type": 1, "status": 1}}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The workflow puts `{"$match": {"tag_id": {"$in": [...]}}}` in front of `connector_tags_query`, so it loads only the tags the scouter read tags and `predictions_batch` write tags reference. When no pipeline references a `tag_id`, the tag database is not queried. An override should keep a `$project` of these fields: any other attribute may hold a BSON type a Temporal payload cannot carry (such as `ObjectId`).
|
||||
|
||||
The referenced tags are loaded whatever their category and status: the code uses only the active ones (`Read` for the slot and the scouter names, `Write` / `Prediction Confidence` for the write tags), and a change to a referenced tag that reaches a schedule redeploys it through its `config_hash` (see "Redeploy on changes" below). A document without `status` counts as active.
|
||||
|
||||
#### OPC slots (slot v2, contract C1)
|
||||
|
||||
`process_slots` writes one slot per active ingestor to Redis (`slot:opc_tags:{n}`). The OPC Gateway reads it (SIENTIAPDE-1974). Each server carries its connection fields, its security and the MinIO objects of its client certificate, and each tag address carries the list of names it is written under:
|
||||
|
||||
```json
|
||||
{
|
||||
"kepserver": {
|
||||
"server_id": "1", "name": "kepserver", "url": "opc.tcp://host:49320",
|
||||
"subscription_period_ms": 7500.0,
|
||||
"security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256"},
|
||||
"certificate": {
|
||||
"certificate_id": 1, "application_uri": "urn:sientia:opc-ingestor",
|
||||
"cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-cert.pem"},
|
||||
"private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-key.pem"}
|
||||
},
|
||||
"tags": {
|
||||
"ns=2;s=Channel1.Device1.Tag1": {
|
||||
"frequency": 15000,
|
||||
"active_read": {"enabled": false},
|
||||
"names": [
|
||||
{"collection": "raw_model_a", "tag_name": "Counter", "model_id": "1", "model_name": "model a"},
|
||||
{"collection": "raw_model_b", "tag_name": "Contador", "model_id": "2", "model_name": "model b"}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Each scouter `read_tags` entry references its tag by `tag_id` (`{"tag_id": 7, "frequency": 15000}`). `connectors` is the only source of the OPC servers:
|
||||
|
||||
| Field | Source |
|
||||
|---|---|
|
||||
| Address | `connector_tags.address` |
|
||||
| Server (`server_id`, `name`, `url`, `subscription_period_ms`, `security.mode`, `security.policy`) | The tag's `connector_tags.connector_id` in `connectors` (`server_id` = `connector_id`) |
|
||||
| `certificate` | The `opc_certificates` document of the connector's `security.client_certificate_id`, when it is `active` and of kind `client` (`connectors_query`'s `$lookup`) |
|
||||
| Name | The read tag's `tag_name`, else the `connector_tags` one |
|
||||
|
||||
Rules:
|
||||
- **One entry per address.** An address read by N pipelines is read once and written to N collections: one `names` item per pipeline (`collection` = `raw_{schedule_name}`).
|
||||
- **Same name source as the scouter.** `resolve_tag_names` names both the slot's `names` and the scouter's `model_tags`, so the gateway always writes the name the scouter looks for.
|
||||
- **`frequency`:** the lowest among the pipelines of the address, as an integer. Every read tag needs one.
|
||||
- **`tag_id`:** required, an integer like in `connector_tags`. A read tag whose `tag_id` is missing, `null` or not an integer (`"7"` included) is left out and notified as invalid.
|
||||
- **`subscription_period_ms`:** the connector's value. A connector without one gets half its lowest tag frequency in the slot.
|
||||
- **`security`:** the connector's `mode` and `policy`, the source of the OPC security (`use_security` and `server_uri` no longer exist). A connector without them is left out like an incomplete one.
|
||||
- **`certificate`:** `certificate_id`, `application_uri` (the SAN URI of the certificate, used as the client application URI) and the `bucket`/`object_name` of `cert` and `private_key`; the gateway downloads them from MinIO. The certificate dates, `sha256` and audit fields stay out of the slot. There is no default certificate: `null` means the gateway connects without security, and a server that requires a certificate refuses the connection. A connector whose `client_certificate_id` is set but has no active `client` certificate (missing id, `retired`, another kind or an incomplete document) still goes into the slot with `certificate: null`, and is reported (see below).
|
||||
- **`active_read`:** off by default. It is on when any pipeline sets `read_tags[].active_read.enabled`, with the lowest `cycle_multiple`.
|
||||
- **Server keys:** two different connectors with the same name get distinct slot keys (`name`, then `name:server_id`).
|
||||
- **Slot division:** the addresses are split across the slots, sorted by server and address, and the names of an address always stay in the same slot.
|
||||
- **Unresolved tags:** a read tag that cannot be resolved is left out of the slot, and the rest of the slot is still written. It is notified once per run:
|
||||
- `ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the `tag_id` is not an active Read tag, or its document lacks `connector_id`, `address` or `tag_name`;
|
||||
- `ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the connector is missing or inactive, or lacks `name`, `url`, `security.mode` or `security.policy`;
|
||||
- `ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION`: the read tag has no integer `tag_id`, no integer `frequency`, or an enabled `active_read` without an integer `cycle_multiple`; or its pipeline lacks `schedule_name`, `model_id` or `model.name` (the value is then the `schedule_name`).
|
||||
- `ORCHESTRATOR_CERTIFICATE_NOT_FOUND_DURING_SLOT_CONFIGURATION` (`WARNING`): the connector's `client_certificate_id` has no active `client` certificate, so its tags go into the slot without a certificate. Most likely a registration error.
|
||||
- **Only changed slots are written.** The slot is built in a stable order, so unchanged pipelines produce an identical slot, and the gateway does not re-subscribe.
|
||||
|
||||
#### Write tags (predictions_batch, through the tag database)
|
||||
|
||||
A `predictions_batch` (or `xgboost_predictions_batch`) references its write tags by `tag_id` only, exactly like the read tags. Everything else comes from the tag database, so the pipeline does not say which server, address or type a tag has:
|
||||
|
||||
```json
|
||||
{"workflow_type": "predictions_batch", "write_tags": [{"tag_id": 101}, {"tag_id": 102}]}
|
||||
```
|
||||
|
||||
`process_schedules` turns them into the `opc_output_config` the Laborious already reads:
|
||||
|
||||
```json
|
||||
{"plant": {
|
||||
"prediction_tags": {"ns=3;i=2001": {"data_type": "double"}},
|
||||
"confidence_tags": {"ns=3;i=2002": {"data_type": "float"}}}}
|
||||
```
|
||||
|
||||
| Field | Source |
|
||||
|---|---|
|
||||
| Server key | `connector_tags.connector_id`. The Laborious publishes it as `server_id` on NATS, and the OPC Gateway resolves it in `connectors` |
|
||||
| `prediction_tags` / `confidence_tags` | `connector_tags.category`: `Write` / `Prediction Confidence` |
|
||||
| Address | `connector_tags.address` |
|
||||
| `data_type` | `connector_tags.data_type` (`float`, `double`, `real`, `int`, `integer`, `bool`, `boolean`, `str`, `string`), else `float` |
|
||||
|
||||
A write tag that cannot be resolved is left out, the rest of the pipeline is still scheduled, and it is notified once per run (values are `schedule_name/tag_id`):
|
||||
- `ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the `tag_id` is not an active `Write` / `Prediction Confidence` tag, or its document lacks `connector_id`, `address`, `tag_name` or `category`;
|
||||
- `ORCHESTRATOR_WRITE_TAG_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the tag's connector is missing, inactive or incomplete in `connectors`;
|
||||
- `ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION`: the entry has no integer `tag_id`, or the tag's `data_type` is not one the gateway accepts.
|
||||
|
||||
#### Redeploy on changes (`updated_at` and `config_hash`)
|
||||
|
||||
`orchestrated_schedules` keeps, per schedule, when it was last deployed (`updated_at`) and the `config_hash` it was deployed with: a SHA-256 of the schedule as the Orchestrator builds it (the workflow input, frequency and offset, with tags and connectors already resolved), without `updated_at`. A schedule is redeployed when either:
|
||||
|
||||
- the pipeline's `updated_at` is newer than its last deploy. Bumping `updated_at` by hand forces a redeploy even when nothing else changed; or
|
||||
- its `config_hash` differs from the stored one. Any change that reaches the schedule redeploys it, whether or not someone bumped an `updated_at`: a pipeline edited by hand, a tag renamed, moved, retyped, deactivated or deleted in `connector_tags`, a connector changed, or a new Orchestrator version that builds the schedule differently.
|
||||
|
||||
A change that does not reach the schedule (e.g. a tag's `description`) does not redeploy. The slot is compared by content and follows on its own. A pipeline without `updated_at` is still redeployed on every run. Schedules deployed before `config_hash` existed have none stored, so the first run of this version redeploys all of them once and records it.
|
||||
|
||||
#### Deploying
|
||||
|
||||
1. Run the migrations that create the tag database and add `data_type` to the `connector_tags` validator (`sientia-dataops-database-migrations`).
|
||||
2. Every scouter `read_tags` and every `predictions_batch` / `xgboost_predictions_batch` `write_tags` entry is a `tag_id` of `connector_tags`, as an integer. Anything else is left out and notified.
|
||||
3. Deploy this version. Its first run redeploys every schedule once, to record their `config_hash` (see above).
|
||||
|
||||
#### Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph "Parallel Data Loading"
|
||||
A[1. aggregate_documents_in_mongodb]
|
||||
B[2. find_documents_in_mongodb<br/>OPC Servers]
|
||||
B[2. aggregate_documents_in_mongodb<br/>connectors + connector_tags]
|
||||
C[3. find_documents_in_mongodb<br/>Orchestrated Schedules]
|
||||
D[4. load_opc_slots]
|
||||
E[5. load_active_ingestors]
|
||||
@@ -556,6 +688,112 @@ python -m orchestrator.worker.worker
|
||||
| `LOG_LEVEL` | Application logging level | `INFO` | No |
|
||||
| `KAFKA_BOOTSTRAP_SERVERS` | Kafka bootstrap servers | - | No |
|
||||
|
||||
### MongoDB Collections
|
||||
|
||||
| Collection | Read / written by | Purpose |
|
||||
|---|---|---|
|
||||
| `pipelines` | Orchestrator (`pipelines_query`) | One document per Temporal schedule (`schedule_name`, `workflow_type`, `model_id`, `active`, `frequency`, ...) |
|
||||
| `models` | Orchestrator (`$lookup` in `pipelines_query`) | Gives each pipeline its `model` (`name`, `active`, `model_config`) |
|
||||
| `connectors` | Orchestrator (`connectors_query`) | Tag database: the OPC UA servers (`connector_id`, `type`, `name`, `url`, `security`, `subscription_period_ms`, `status`) |
|
||||
| `connector_tags` | Orchestrator (`connector_tags_query`) | Tag database: the tags (`tag_id`, `connector_id`, `address`, `tag_name`, `category`, `data_type`, `status`) |
|
||||
| `opc_certificates` | Orchestrator (`$lookup` in `connectors_query`) | The client certificate of each connector (`security.client_certificate_id` → `certificate_id`), only when `active` and of kind `client`: `application_uri` and the MinIO objects of `cert` and `private_key` go into the slot |
|
||||
| `orchestrated_schedules` | Orchestrator (read and written) | Per deployed schedule: `schedule_name`, `namespace`, `updated_at` (last deploy) and `config_hash` |
|
||||
| `raw_{schedule_name}` | Orchestrator (created) | One per scouter `topic`, where the OPC Gateway writes its tags. Created with a TTL index on `inserted_at` (`MONGODB_TTL_INDEX_HOURS`) |
|
||||
| `notification_queue` | Alerts / Reports (read) | Notifications (`notification_id`, `trigger`, `level`, `model_id`, `timestamp`, ...), loaded incrementally by `timestamp` |
|
||||
| `receiver_groups` | Alerts / Reports (read, `active: true`) | Who receives what (see below) |
|
||||
|
||||
#### Tags in a pipeline
|
||||
|
||||
Pipelines reference tags only by `tag_id` (an integer, as in `connector_tags`). Servers, addresses and data types come from the tag database, never from the pipeline.
|
||||
|
||||
Scouter (`scouter` namespace), `workflow_type: "scouter"`:
|
||||
|
||||
```json
|
||||
{
|
||||
"schedule_name": "scouter-linreg",
|
||||
"workflow_type": "scouter",
|
||||
"model_id": "1",
|
||||
"active": true,
|
||||
"frequency": "1m",
|
||||
"read_tags": [
|
||||
{
|
||||
"tag_id": 7,
|
||||
"frequency": 15000,
|
||||
"tag_name": "Pressure",
|
||||
"aggr_func": "avg",
|
||||
"data_range": [0, 100],
|
||||
"active_read": {"enabled": true, "cycle_multiple": 2}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Required | Meaning |
|
||||
|---|---|---|
|
||||
| `tag_id` | Yes | An active `Read` tag of `connector_tags` |
|
||||
| `frequency` | Yes | Read period in ms, an integer. The slot uses the lowest among the pipelines of the address |
|
||||
| `tag_name` | No | Overrides the `connector_tags` name: the name the gateway writes and the scouter looks for |
|
||||
| `aggr_func` | No | Default `lts` |
|
||||
| `data_range` | No | Default `[-100, 100]` |
|
||||
| `active_read` | No | Off by default. When `enabled`, `cycle_multiple` must be an integer (default `1`) |
|
||||
|
||||
Laborious (`laborious` namespace): only `predictions_batch` and `xgboost_predictions_batch` reference tags; `minimal_retrain`, `drift` and `simple_metrics` have none.
|
||||
|
||||
```json
|
||||
{
|
||||
"schedule_name": "pred-linreg",
|
||||
"workflow_type": "predictions_batch",
|
||||
"model_id": "1",
|
||||
"active": true,
|
||||
"frequency": "1m",
|
||||
"query": "SELECT ...",
|
||||
"write_tags": [{"tag_id": 101}, {"tag_id": 102}]
|
||||
}
|
||||
```
|
||||
|
||||
Each `write_tags` entry is only a `tag_id`, of an active `Write` (prediction) or `Prediction Confidence` tag. The Orchestrator builds `opc_output_config` from them (see "Write tags" above) and overwrites any `opc_output_config` stored in the pipeline.
|
||||
|
||||
The `connector_tags` documents those pipelines reference:
|
||||
|
||||
```json
|
||||
{"tag_id": 7, "connector_id": "plant", "address": "ns=3;i=1001", "tag_name": "Pressure", "category": "Read", "status": "Active"}
|
||||
{"tag_id": 101, "connector_id": "plant", "address": "ns=3;i=2001", "tag_name": "PRED-101", "category": "Write", "data_type": "double", "status": "Active"}
|
||||
{"tag_id": 102, "connector_id": "plant", "address": "ns=3;i=2002", "tag_name": "CONF-101", "category": "Prediction Confidence", "status": "Active"}
|
||||
```
|
||||
|
||||
`pi_web_api_scouter` (`scouter` namespace) does not use the tag database: its `read_tags` is an object keyed by tag name, and it is not part of the OPC slot:
|
||||
|
||||
```json
|
||||
{
|
||||
"workflow_type": "pi_web_api_scouter",
|
||||
"read_tags": {"Pressure": {"webid": "F1DP...", "aggr_func": "lts", "data_range": [0, 100]}},
|
||||
"pi_web_api_config": {"endpoint": "https://pi/piwebapi", "period": "*-1d", "max_count": 1, "api_timeout": 30}
|
||||
}
|
||||
```
|
||||
|
||||
#### Receiver groups
|
||||
|
||||
```json
|
||||
{
|
||||
"group_name": "admin_team",
|
||||
"active": true,
|
||||
"members": ["admin@example.com"],
|
||||
"contents": ["core_alerts", "persistent_alerts", "reports"],
|
||||
"levels": ["ERROR", "WARNING"],
|
||||
"model_ids": ["1"],
|
||||
"ignore_models_id": ["2"],
|
||||
"ignore": ["SOME_NOTIFICATION_ID"]
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `contents` | What the group gets: `core_alerts` / `persistent_alerts` (Alerts workflow), `reports` (Reports workflow) |
|
||||
| `levels` | Optional allow-list of notification levels. Absent = every level the workflow loads |
|
||||
| `model_ids` | Optional allow-list of `model_id`s |
|
||||
| `ignore_models_id` | Optional deny-list of `model_id`s (applies together with `model_ids`) |
|
||||
| `ignore` | Optional deny-list of `notification_id`s |
|
||||
|
||||
### Workflow Configuration
|
||||
|
||||
Temporal input configuration sample:
|
||||
@@ -583,7 +821,7 @@ Temporal input configuration sample:
|
||||
},
|
||||
{
|
||||
"$addFields": {
|
||||
"models": {
|
||||
"model": {
|
||||
"$arrayElemAt": [
|
||||
"$model_docs",
|
||||
0
|
||||
@@ -593,7 +831,7 @@ Temporal input configuration sample:
|
||||
},
|
||||
{
|
||||
"$match": {
|
||||
"models.active": True
|
||||
"model.active": True
|
||||
}
|
||||
},
|
||||
{
|
||||
@@ -603,15 +841,39 @@ Temporal input configuration sample:
|
||||
}
|
||||
]
|
||||
},
|
||||
"opc_servers_query": {
|
||||
"collection": "opc-servers",
|
||||
"filters": {
|
||||
|
||||
}
|
||||
"connectors_query": {
|
||||
"collection": "connectors",
|
||||
"aggregation": [
|
||||
{"$match": {"type": "opc_ua", "status": "Active"}},
|
||||
{"$lookup": {"from": "opc_certificates",
|
||||
"let": {"certificate_id": "$security.client_certificate_id"},
|
||||
"pipeline": [
|
||||
{"$match": {"$expr": {"$eq": ["$certificate_id", "$$certificate_id"]},
|
||||
"status": "active", "kind": "client"}},
|
||||
{"$project": {"_id": 0, "certificate_id": 1, "application_uri": 1,
|
||||
"cert.bucket": 1, "cert.object_name": 1,
|
||||
"private_key.bucket": 1, "private_key.object_name": 1}}
|
||||
],
|
||||
"as": "certificates"}},
|
||||
{"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "subscription_period_ms": 1,
|
||||
"security.mode": 1, "security.policy": 1,
|
||||
"security.client_certificate_id": 1, "certificates": 1}}
|
||||
]
|
||||
},
|
||||
"connector_tags_query": {
|
||||
"collection": "connector_tags",
|
||||
"aggregation": [
|
||||
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1,
|
||||
"category": 1, "data_type": 1, "status": 1}}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`connectors_query` and `connector_tags_query` are optional: the values above are the defaults in the code (see "OPC slots" and "Write tags" above).
|
||||
|
||||
The `$lookup` must end in a `model` field (singular): every pipeline type reads `model.name` (and `model.model_config`, when present), so a pipeline without `model` fails to build its schedule.
|
||||
|
||||
#### Simple metrics: parallel windows & adaptive frequency
|
||||
|
||||
One `pipelines` doc per window, all with the same `model_id`. Name them `<model>-simple_metrics-<interval_minutes>m`; each doc becomes its own Temporal schedule. Thresholds are shared through `model.model_config`. The adaptive state is per doc.
|
||||
@@ -823,7 +1085,9 @@ orchestrator/
|
||||
├── utils/ # Utility functions
|
||||
│ ├── connectors_config.py # Database configuration
|
||||
│ ├── email_builder.py # Email template generation
|
||||
│ ├── orchestrator_functions.py # Pipeline utilities
|
||||
│ ├── pipeline_configs.py # Schedule input builders per pipeline type
|
||||
│ ├── tag_database.py # Tag database indexes, OPC read names and write tags
|
||||
│ ├── opc_slots.py # OPC slot v2 (contract C1)
|
||||
│ ├── converters.py # Data type conversion utilities
|
||||
│ └── templates/ # Email HTML templates
|
||||
│ ├── email_template.html # Report email template
|
||||
|
||||
@@ -25,8 +25,8 @@ MONGO_COLLECTIONS = (
|
||||
'receiver_groups',
|
||||
'orchestrated_schedules',
|
||||
'pipelines',
|
||||
'opc_servers',
|
||||
'opc-servers',
|
||||
'connectors',
|
||||
'connector_tags',
|
||||
)
|
||||
|
||||
DATETIME_FORMAT_MS_WITH_TZ = '%Y-%m-%d %H:%M:%S.%f%z'
|
||||
@@ -143,19 +143,25 @@ def seed_pipelines(
|
||||
client.close()
|
||||
|
||||
|
||||
def seed_opc_servers(
|
||||
def seed_tag_database(
|
||||
mongo_uri: str,
|
||||
database: str,
|
||||
servers: list[dict[str, Any]],
|
||||
*,
|
||||
collection: str = 'opc_servers',
|
||||
connectors: list[dict[str, Any]],
|
||||
connector_tags: list[dict[str, Any]],
|
||||
opc_certificates: list[dict[str, Any]] | None = None,
|
||||
) -> None:
|
||||
"""Insert OPC server documents into the test Mongo database."""
|
||||
"""
|
||||
Insert tag database documents (connectors, connector_tags and the opc_certificates
|
||||
the connectors point to) into the test Mongo database.
|
||||
"""
|
||||
client = MongoClient(mongo_uri)
|
||||
try:
|
||||
coll = client[database][collection]
|
||||
if servers:
|
||||
coll.insert_many(servers)
|
||||
if connectors:
|
||||
client[database]['connectors'].insert_many(connectors)
|
||||
if connector_tags:
|
||||
client[database]['connector_tags'].insert_many(connector_tags)
|
||||
if opc_certificates:
|
||||
client[database]['opc_certificates'].insert_many(opc_certificates)
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
@@ -4,10 +4,6 @@
|
||||
"pipelines_query": {
|
||||
"collection": "pipelines",
|
||||
"aggregation": [{"$match": {"active": true}}]
|
||||
},
|
||||
"opc_servers_query": {
|
||||
"collection": "opc_servers",
|
||||
"filters": {"active": true}
|
||||
}
|
||||
},
|
||||
"pipelines": [
|
||||
@@ -22,13 +18,16 @@
|
||||
"frequency": "1m",
|
||||
"offset": "0m",
|
||||
"query": "SELECT 1",
|
||||
"write_tags": [
|
||||
{"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
|
||||
]
|
||||
"write_tags": [{"tag_id": 101}]
|
||||
}
|
||||
],
|
||||
"opc_servers": [
|
||||
{"id": "srv-1", "active": true}
|
||||
"connectors": [
|
||||
{"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
|
||||
"security": {"mode": "None", "policy": "None"}, "status": "Active"}
|
||||
],
|
||||
"connector_tags": [
|
||||
{"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
|
||||
"category": "Write", "status": "Active", "updated_at": "@now-1d"}
|
||||
],
|
||||
"active_ingestors": ["heartbeat:ingestor:1"]
|
||||
}
|
||||
|
||||
@@ -4,10 +4,6 @@
|
||||
"pipelines_query": {
|
||||
"collection": "pipelines",
|
||||
"aggregation": [{"$match": {"active": true}}]
|
||||
},
|
||||
"opc_servers_query": {
|
||||
"collection": "opc_servers",
|
||||
"filters": {"active": true}
|
||||
}
|
||||
},
|
||||
"pipelines": [
|
||||
@@ -22,13 +18,16 @@
|
||||
"frequency": "1m",
|
||||
"offset": "0m",
|
||||
"query": "SELECT 1",
|
||||
"write_tags": [
|
||||
{"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
|
||||
]
|
||||
"write_tags": [{"tag_id": 101}]
|
||||
}
|
||||
],
|
||||
"opc_servers": [
|
||||
{"id": "srv-1", "active": true}
|
||||
"connectors": [
|
||||
{"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
|
||||
"security": {"mode": "None", "policy": "None"}, "status": "Active"}
|
||||
],
|
||||
"connector_tags": [
|
||||
{"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
|
||||
"category": "Write", "status": "Active", "updated_at": "@now-1d"}
|
||||
],
|
||||
"active_ingestors": ["heartbeat:ingestor:1"]
|
||||
}
|
||||
|
||||
@@ -4,10 +4,6 @@
|
||||
"pipelines_query": {
|
||||
"collection": "pipelines",
|
||||
"aggregation": [{"$match": {"active": true}}]
|
||||
},
|
||||
"opc_servers_query": {
|
||||
"collection": "opc_servers",
|
||||
"filters": {"active": true}
|
||||
}
|
||||
},
|
||||
"pipelines": [
|
||||
@@ -24,8 +20,5 @@
|
||||
"interval_minutes": 60
|
||||
}
|
||||
],
|
||||
"opc_servers": [
|
||||
{"id": "srv-1", "active": true}
|
||||
],
|
||||
"active_ingestors": ["heartbeat:ingestor:1"]
|
||||
}
|
||||
|
||||
@@ -4,10 +4,6 @@
|
||||
"pipelines_query": {
|
||||
"collection": "pipelines",
|
||||
"aggregation": [{"$match": {"active": true}}]
|
||||
},
|
||||
"opc_servers_query": {
|
||||
"collection": "opc_servers",
|
||||
"filters": {"active": true}
|
||||
}
|
||||
},
|
||||
"pipelines": [
|
||||
@@ -33,6 +29,5 @@
|
||||
"chunk_period": "s"
|
||||
}
|
||||
],
|
||||
"opc_servers": [],
|
||||
"active_ingestors": ["heartbeat:ingestor:1"]
|
||||
}
|
||||
|
||||
@@ -4,13 +4,8 @@
|
||||
"pipelines_query": {
|
||||
"collection": "pipelines",
|
||||
"aggregation": [{"$match": {"active": true}}]
|
||||
},
|
||||
"opc_servers_query": {
|
||||
"collection": "opc_servers",
|
||||
"filters": {"active": true}
|
||||
}
|
||||
},
|
||||
"pipelines": [],
|
||||
"opc_servers": [],
|
||||
"active_ingestors": []
|
||||
}
|
||||
|
||||
@@ -4,10 +4,6 @@
|
||||
"pipelines_query": {
|
||||
"collection": "pipelines",
|
||||
"aggregation": [{"$match": {"active": true}}]
|
||||
},
|
||||
"opc_servers_query": {
|
||||
"collection": "opc_servers",
|
||||
"filters": {"active": true}
|
||||
}
|
||||
},
|
||||
"pipelines": [
|
||||
@@ -22,9 +18,7 @@
|
||||
"frequency": "1m",
|
||||
"offset": "0m",
|
||||
"query": "SELECT 1",
|
||||
"write_tags": [
|
||||
{"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
|
||||
]
|
||||
"write_tags": [{"tag_id": 101}]
|
||||
},
|
||||
{
|
||||
"schedule_name": "drift-gpu",
|
||||
@@ -39,8 +33,13 @@
|
||||
"interval_minutes": 60
|
||||
}
|
||||
],
|
||||
"opc_servers": [
|
||||
{"id": "srv-1", "active": true, "name": "opc-1"}
|
||||
"connectors": [
|
||||
{"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
|
||||
"security": {"mode": "None", "policy": "None"}, "status": "Active"}
|
||||
],
|
||||
"connector_tags": [
|
||||
{"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
|
||||
"category": "Write", "status": "Active", "updated_at": "@now-1d"}
|
||||
],
|
||||
"active_ingestors": ["heartbeat:ingestor:1"]
|
||||
}
|
||||
|
||||
@@ -4,10 +4,6 @@
|
||||
"pipelines_query": {
|
||||
"collection": "pipelines",
|
||||
"aggregation": [{"$match": {"active": true}}]
|
||||
},
|
||||
"opc_servers_query": {
|
||||
"collection": "opc_servers",
|
||||
"filters": {"active": true}
|
||||
}
|
||||
},
|
||||
"pipelines": [
|
||||
@@ -22,13 +18,16 @@
|
||||
"frequency": "1m",
|
||||
"offset": "0m",
|
||||
"query": "SELECT 1",
|
||||
"write_tags": [
|
||||
{"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
|
||||
]
|
||||
"write_tags": [{"tag_id": 101}]
|
||||
}
|
||||
],
|
||||
"opc_servers": [
|
||||
{"id": "srv-1", "active": true}
|
||||
"connectors": [
|
||||
{"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
|
||||
"security": {"mode": "None", "policy": "None"}, "status": "Active"}
|
||||
],
|
||||
"connector_tags": [
|
||||
{"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
|
||||
"category": "Write", "status": "Active", "updated_at": "@now-1d"}
|
||||
],
|
||||
"active_ingestors": ["heartbeat:ingestor:1"],
|
||||
"orchestrated_schedules": [
|
||||
|
||||
@@ -4,10 +4,6 @@
|
||||
"pipelines_query": {
|
||||
"collection": "pipelines",
|
||||
"aggregation": [{"$match": {"active": true}}]
|
||||
},
|
||||
"opc_servers_query": {
|
||||
"collection": "opc_servers",
|
||||
"filters": {"active": true}
|
||||
}
|
||||
},
|
||||
"pipelines": [
|
||||
@@ -40,6 +36,5 @@
|
||||
"interval_minutes": 1440
|
||||
}
|
||||
],
|
||||
"opc_servers": [],
|
||||
"active_ingestors": ["heartbeat:ingestor:1"]
|
||||
}
|
||||
|
||||
71
e2e/scenario_inputs/orchestrator_tag_database.json
Normal file
71
e2e/scenario_inputs/orchestrator_tag_database.json
Normal file
@@ -0,0 +1,71 @@
|
||||
{
|
||||
"workflow_input": {
|
||||
"schedule_name": "orchestrator-e2e-tag-database",
|
||||
"pipelines_query": {
|
||||
"collection": "pipelines",
|
||||
"aggregation": [{"$match": {"active": true}}]
|
||||
}
|
||||
},
|
||||
"connectors": [
|
||||
{
|
||||
"connector_id": "plant",
|
||||
"type": "opc_ua",
|
||||
"name": "plant",
|
||||
"url": "opc.tcp://plant:4840",
|
||||
"subscription_period_ms": 500,
|
||||
"security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256", "client_certificate_id": 1},
|
||||
"status": "Active",
|
||||
"updated_at": "@now-1d"
|
||||
}
|
||||
],
|
||||
"opc_certificates": [
|
||||
{"certificate_id": 1, "kind": "client", "name": "sientia-platform", "status": "active",
|
||||
"application_uri": "urn:sientia:opc-ingestor",
|
||||
"cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/cert.pem", "sha256": null},
|
||||
"private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/key.pem", "sha256": null},
|
||||
"thumbprint_sha1": null, "not_before": "@now-1d", "not_after": "@now+365d",
|
||||
"created_at": "@now-1d", "updated_at": "@now-1d", "created_by": "e2e", "updated_by": "e2e"},
|
||||
{"certificate_id": 2, "kind": "client", "name": "retired", "status": "retired",
|
||||
"application_uri": "urn:sientia:old",
|
||||
"cert": {"bucket": "opc-certificates", "object_name": "client/retired/cert.pem"},
|
||||
"private_key": {"bucket": "opc-certificates", "object_name": "client/retired/key.pem"},
|
||||
"created_at": "@now-1d", "updated_at": "@now-1d", "created_by": "e2e", "updated_by": "e2e"}
|
||||
],
|
||||
"connector_tags": [
|
||||
{"tag_id": 7, "connector_id": "plant", "address": "ns=3;i=1001", "tag_name": "Pressure", "category": "Read", "status": "Active", "updated_at": "@now-1d"},
|
||||
{"tag_id": 101, "connector_id": "plant", "address": "ns=3;i=2001", "tag_name": "PRED-101", "category": "Write", "data_type": "double", "status": "Active", "updated_at": "@now-1d"},
|
||||
{"tag_id": 102, "connector_id": "plant", "address": "ns=3;i=2002", "tag_name": "CONF-101", "category": "Prediction Confidence", "status": "Active", "updated_at": "@now-1d"},
|
||||
{"tag_id": 103, "connector_id": "plant", "address": "ns=3;i=2003", "tag_name": "PRED-OLD", "category": "Write", "status": "Inactive", "updated_at": "@now-1d"}
|
||||
],
|
||||
"pipelines": [
|
||||
{
|
||||
"schedule_name": "opc-tagdb",
|
||||
"workflow_type": "scouter",
|
||||
"model_id": "model-opc",
|
||||
"model": {"name": "Model model-opc"},
|
||||
"active": true,
|
||||
"updated_at": "@now-2d",
|
||||
"frequency": "1m",
|
||||
"read_tags": [{"tag_id": 7, "frequency": 1000, "aggr_func": "avg"}]
|
||||
},
|
||||
{
|
||||
"schedule_name": "pred-tagdb",
|
||||
"workflow_type": "predictions_batch",
|
||||
"model_id": "model-opc",
|
||||
"model": {"name": "Model model-opc"},
|
||||
"active": true,
|
||||
"updated_at": "@now-2d",
|
||||
"frequency": "1m",
|
||||
"query": "SELECT 1",
|
||||
"write_tags": [
|
||||
{"tag_id": 101},
|
||||
{"tag_id": 102},
|
||||
{"tag_id": 7},
|
||||
{"tag_id": 103},
|
||||
{"tag_id": 404},
|
||||
{"tag_name": "PRED-NO-ID"}
|
||||
]
|
||||
}
|
||||
],
|
||||
"active_ingestors": ["heartbeat:ingestor:1"]
|
||||
}
|
||||
@@ -4,10 +4,6 @@
|
||||
"pipelines_query": {
|
||||
"collection": "pipelines",
|
||||
"aggregation": [{"$match": {"active": true}}]
|
||||
},
|
||||
"opc_servers_query": {
|
||||
"collection": "opc_servers",
|
||||
"filters": {"active": true}
|
||||
}
|
||||
},
|
||||
"pipelines": [
|
||||
@@ -22,24 +18,17 @@
|
||||
"frequency": "1m",
|
||||
"offset": "0m",
|
||||
"read_tags": [
|
||||
{
|
||||
"server_id": "srv-1",
|
||||
"tag_name": "Read1",
|
||||
"tag_address": "ns=2;s=Read1",
|
||||
"aggr_func": "lts",
|
||||
"frequency": 1000
|
||||
}
|
||||
{"tag_id": 1, "aggr_func": "lts", "frequency": 1000}
|
||||
]
|
||||
}
|
||||
],
|
||||
"opc_servers": [
|
||||
{
|
||||
"id": "srv-1",
|
||||
"active": true,
|
||||
"server_name": "opc-1",
|
||||
"url": "opc.tcp://localhost:4840",
|
||||
"uri": "urn:opcfoundation:UA:DemoServer"
|
||||
}
|
||||
"connectors": [
|
||||
{"connector_id": "opc-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://localhost:4840",
|
||||
"security": {"mode": "None", "policy": "None"}, "status": "Active"}
|
||||
],
|
||||
"connector_tags": [
|
||||
{"tag_id": 1, "connector_id": "opc-1", "address": "ns=2;s=Read1", "tag_name": "Read1",
|
||||
"category": "Read", "status": "Active"}
|
||||
],
|
||||
"active_ingestors": ["heartbeat:ingestor:1"]
|
||||
}
|
||||
|
||||
@@ -4,10 +4,6 @@
|
||||
"pipelines_query": {
|
||||
"collection": "pipelines",
|
||||
"aggregation": [{"$match": {"active": true}}]
|
||||
},
|
||||
"opc_servers_query": {
|
||||
"collection": "opc_servers",
|
||||
"filters": {"active": true}
|
||||
}
|
||||
},
|
||||
"pipelines": [
|
||||
@@ -22,13 +18,16 @@
|
||||
"frequency": "5m",
|
||||
"offset": "0m",
|
||||
"query": "SELECT 1",
|
||||
"write_tags": [
|
||||
{"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
|
||||
]
|
||||
"write_tags": [{"tag_id": 101}]
|
||||
}
|
||||
],
|
||||
"opc_servers": [
|
||||
{"id": "srv-1", "active": true}
|
||||
"connectors": [
|
||||
{"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
|
||||
"security": {"mode": "None", "policy": "None"}, "status": "Active"}
|
||||
],
|
||||
"connector_tags": [
|
||||
{"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
|
||||
"category": "Write", "status": "Active", "updated_at": "@now-1d"}
|
||||
],
|
||||
"active_ingestors": ["heartbeat:ingestor:1"],
|
||||
"orchestrated_schedules": [
|
||||
|
||||
@@ -26,7 +26,9 @@ Source: `e2e/test_orchestrator_main_workflow.py`
|
||||
|
||||
### 1.1.1 Happy path
|
||||
|
||||
Pipelines in Mongo → Temporal schedule created in the `laborious` namespace, `orchestrated_schedules` entries written.
|
||||
Pipelines in Mongo → both Temporal schedules created in the `laborious` namespace, one
|
||||
`orchestrated_schedules` entry per pipeline, and the `predictions_batch` write tag (`tag_id`)
|
||||
resolved into `opc_output_config` through the tag database.
|
||||
|
||||
### 1.3.1 Create-only
|
||||
|
||||
@@ -64,6 +66,34 @@ Two `drift` pipelines on `frequency='1m'`: one without `interval_minutes` and wi
|
||||
`interval_minutes=60` and `chunk_period='s'` → both forwarded unchanged, no warning
|
||||
(SIENTIAPDE-2212).
|
||||
|
||||
### 1.9 OPC tag database: read and write tags by tag_id
|
||||
|
||||
Scenario input: `orchestrator_tag_database.json` (one `opc_ua` connector, its `connector_tags` of
|
||||
every category, one inactive, an OPC scouter reading by `tag_id` and a `predictions_batch` writing by
|
||||
`tag_id`, with a Read tag, an inactive tag, an unknown `tag_id` and an entry without `tag_id` among its
|
||||
write tags).
|
||||
|
||||
#### 1.9.1 Write tags through the tag database
|
||||
|
||||
`write_tags: [{tag_id}]` → `opc_output_config` keyed by `connector_id`, address and `data_type`
|
||||
from `connector_tags` (`float` when absent), `Write` → `prediction_tags`, `Prediction Confidence` →
|
||||
`confidence_tags`. The Read, inactive and unknown tags are left out and notified once
|
||||
(`ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`); the entry without `tag_id` is notified
|
||||
(`ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION`).
|
||||
|
||||
#### 1.9.2 A tag change redeploys its pipelines
|
||||
|
||||
A second run with nothing changed redeploys nothing (`orchestrated_schedules` is not stamped
|
||||
again). Then renaming the read tag, moving the write tag and deleting the confidence tag in
|
||||
`connector_tags` by hand (no `updated_at` bumped anywhere, pipelines untouched) → the next run
|
||||
redeploys both through their `config_hash`: the scouter `model_tags` and the slot `names` take
|
||||
the new name, the `opc_output_config` the new address without the confidence tag, and
|
||||
`orchestrated_schedules` records the new `config_hash`.
|
||||
|
||||
#### 1.9.3 OPC slot from the tag database
|
||||
|
||||
The slot v2 written to Redis holds only the active Read tag, with the connector's server fields.
|
||||
|
||||
---
|
||||
|
||||
## 2. Alerts workflow
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
"""E2E tests for the Orchestrator main workflow."""
|
||||
|
||||
import json
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import pytest
|
||||
@@ -14,9 +15,9 @@ from e2e.helpers import (
|
||||
load_scenario_input,
|
||||
make_workflow_id,
|
||||
seed_active_ingestors,
|
||||
seed_opc_servers,
|
||||
seed_orchestrated_schedules,
|
||||
seed_pipelines,
|
||||
seed_tag_database,
|
||||
start_and_await_workflow,
|
||||
)
|
||||
from orchestrator.workflows.orchestrator import Orchestrator
|
||||
@@ -41,7 +42,7 @@ def _apply_orchestrator_seeds(
|
||||
redis_client: Redis,
|
||||
) -> None:
|
||||
"""
|
||||
Seed Mongo and Redis with the pipelines/opc_servers/ingestors declared in a scenario.
|
||||
Seed Mongo and Redis with the pipelines/tag database/ingestors of a scenario.
|
||||
|
||||
Args:
|
||||
scenario: Scenario payload returned by ``load_scenario_input``.
|
||||
@@ -49,7 +50,13 @@ def _apply_orchestrator_seeds(
|
||||
redis_client: Redis client connected to the test instance.
|
||||
"""
|
||||
seed_pipelines(mongo_uri, E2E_DATABASE, scenario.get('pipelines', []))
|
||||
seed_opc_servers(mongo_uri, E2E_DATABASE, scenario.get('opc_servers', []))
|
||||
seed_tag_database(
|
||||
mongo_uri,
|
||||
E2E_DATABASE,
|
||||
scenario.get('connectors', []),
|
||||
scenario.get('connector_tags', []),
|
||||
scenario.get('opc_certificates', []),
|
||||
)
|
||||
seed_active_ingestors(redis_client, scenario.get('active_ingestors', []))
|
||||
if scenario.get('orchestrated_schedules'):
|
||||
seed_orchestrated_schedules(
|
||||
@@ -84,14 +91,19 @@ async def test_scenario_1_1_1_happy_path(
|
||||
try:
|
||||
tracked = list(mongo[E2E_DATABASE]['orchestrated_schedules'].find())
|
||||
names = {doc['schedule_name'] for doc in tracked}
|
||||
assert names, f'Expected orchestrated_schedules rows, got {tracked}'
|
||||
assert names == {'pred-legacy', 'drift-gpu'}, f'Unexpected orchestrated_schedules {tracked}'
|
||||
finally:
|
||||
mongo.close()
|
||||
|
||||
laborious_schedules = await _schedule_ids(temporal_host, 'laborious')
|
||||
assert 'pred-legacy' in laborious_schedules or 'drift-gpu' in laborious_schedules, (
|
||||
assert {'pred-legacy', 'drift-gpu'} <= set(laborious_schedules), (
|
||||
f'Expected Temporal schedules in laborious, got {laborious_schedules}'
|
||||
)
|
||||
# The write tag comes from the tag database, keyed by the connector the gateway writes to
|
||||
pred = await _schedule_input(temporal_host, 'pred-legacy')
|
||||
assert pred['opc_output_config'] == {
|
||||
'srv-1': {'prediction_tags': {'ns=2;s=Tag1': {'data_type': 'float'}}}
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.e2e
|
||||
@@ -315,9 +327,9 @@ async def test_scenario_1_7_2_simple_metrics_adaptive_frequency_round_trip(
|
||||
assert await _schedule_every(temporal_host, short) == timedelta(hours=1)
|
||||
|
||||
|
||||
async def _schedule_input(host: str, schedule_id: str) -> dict:
|
||||
"""Return the decoded workflow input of a laborious-namespace schedule."""
|
||||
client = await Client.connect(host, namespace='laborious')
|
||||
async def _schedule_input(host: str, schedule_id: str, namespace: str = 'laborious') -> dict:
|
||||
"""Return the decoded workflow input of a schedule (laborious namespace by default)."""
|
||||
client = await Client.connect(host, namespace=namespace)
|
||||
description = await client.get_schedule_handle(schedule_id).describe()
|
||||
action = description.schedule.action
|
||||
assert isinstance(action, ScheduleActionStartWorkflow)
|
||||
@@ -364,3 +376,195 @@ async def test_scenario_1_8_1_drift_window_and_chunk_period(
|
||||
)
|
||||
assert len(warnings) == 1
|
||||
assert warnings[0]['model_id'] == 'model-dr'
|
||||
|
||||
|
||||
# 1.9 OPC tag database: read and write tags by tag_id (SIENTIAPDE-1974)
|
||||
|
||||
|
||||
async def _run_tag_database_scenario(temporal_env: WorkflowEnvironment, tag: str) -> None:
|
||||
await start_and_await_workflow(
|
||||
temporal_env.client,
|
||||
Orchestrator.run,
|
||||
load_scenario_input('orchestrator_tag_database')['workflow_input'],
|
||||
make_workflow_id(f'orchestrator-tagdb-{tag}'),
|
||||
)
|
||||
|
||||
|
||||
def _notifications(notification_handler, notification_id: str) -> list[dict]:
|
||||
return list(notification_handler.mongo_collection.find({'notification_id': notification_id}))
|
||||
|
||||
|
||||
def _opc_slots(redis_client: Redis) -> dict[str, dict]:
|
||||
return {
|
||||
key if isinstance(key, str) else key.decode(): json.loads(redis_client.get(key))
|
||||
for key in redis_client.keys('slot:opc_tags:*')
|
||||
}
|
||||
|
||||
|
||||
def _orchestrated_schedules(mongo_uri: str) -> dict[str, dict]:
|
||||
"""orchestrated_schedules by schedule_name: updated_at (last deploy) and config_hash."""
|
||||
mongo = MongoClient(mongo_uri)
|
||||
try:
|
||||
return {
|
||||
doc['schedule_name']: {
|
||||
'updated_at': doc['updated_at'],
|
||||
'config_hash': doc.get('config_hash'),
|
||||
}
|
||||
for doc in mongo[E2E_DATABASE]['orchestrated_schedules'].find()
|
||||
}
|
||||
finally:
|
||||
mongo.close()
|
||||
|
||||
|
||||
@pytest.mark.e2e
|
||||
@pytest.mark.asyncio
|
||||
async def test_scenario_1_9_1_write_tags_through_tag_database(
|
||||
temporal_env: WorkflowEnvironment,
|
||||
orchestrator_worker,
|
||||
stub_workers,
|
||||
mongo_uri: str,
|
||||
temporal_host: str,
|
||||
redis_client: Redis,
|
||||
notification_handler,
|
||||
):
|
||||
"""1.9.1: write_tags by tag_id become the opc_output_config; the rest is notified."""
|
||||
_apply_orchestrator_seeds(
|
||||
load_scenario_input('orchestrator_tag_database'), mongo_uri, redis_client
|
||||
)
|
||||
|
||||
await _run_tag_database_scenario(temporal_env, 'write')
|
||||
|
||||
pred = await _schedule_input(temporal_host, 'pred-tagdb')
|
||||
assert pred['opc_output_config'] == {
|
||||
'plant': {
|
||||
'prediction_tags': {'ns=3;i=2001': {'data_type': 'double'}},
|
||||
'confidence_tags': {'ns=3;i=2002': {'data_type': 'float'}},
|
||||
}
|
||||
}
|
||||
|
||||
not_found = _notifications(
|
||||
notification_handler, 'ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION'
|
||||
)
|
||||
assert len(not_found) == 1
|
||||
# A Read tag, an inactive tag and an unknown tag_id
|
||||
for value in ('pred-tagdb/7', 'pred-tagdb/103', 'pred-tagdb/404'):
|
||||
assert value in not_found[0]['message']
|
||||
invalid = _notifications(
|
||||
notification_handler, 'ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION'
|
||||
)
|
||||
assert len(invalid) == 1
|
||||
assert 'PRED-NO-ID' in invalid[0]['message']
|
||||
|
||||
|
||||
@pytest.mark.e2e
|
||||
@pytest.mark.asyncio
|
||||
async def test_scenario_1_9_2_tag_change_redeploys_pipelines(
|
||||
temporal_env: WorkflowEnvironment,
|
||||
orchestrator_worker,
|
||||
stub_workers,
|
||||
mongo_uri: str,
|
||||
temporal_host: str,
|
||||
redis_client: Redis,
|
||||
):
|
||||
"""
|
||||
1.9.2: changing or deleting a tag in connector_tags, without touching any updated_at,
|
||||
redeploys the pipelines that reference it (config_hash); an unchanged run does not.
|
||||
"""
|
||||
_apply_orchestrator_seeds(
|
||||
load_scenario_input('orchestrator_tag_database'), mongo_uri, redis_client
|
||||
)
|
||||
await _run_tag_database_scenario(temporal_env, 'before-change')
|
||||
|
||||
scouter = await _schedule_input(temporal_host, 'opc-tagdb', namespace='scouter')
|
||||
assert set(scouter['model_tags']) == {'Pressure'}
|
||||
deployed = _orchestrated_schedules(mongo_uri)
|
||||
assert all(doc.get('config_hash') for doc in deployed.values()), deployed
|
||||
|
||||
# Nothing changed: no redeploy, so orchestrated_schedules is not stamped again
|
||||
await _run_tag_database_scenario(temporal_env, 'unchanged')
|
||||
assert _orchestrated_schedules(mongo_uri) == deployed
|
||||
|
||||
# As an edit by hand in Compass: rename the read tag, move a write tag and delete the
|
||||
# confidence tag, with no updated_at bumped anywhere
|
||||
mongo = MongoClient(mongo_uri)
|
||||
try:
|
||||
tags = mongo[E2E_DATABASE]['connector_tags']
|
||||
tags.update_one({'tag_id': 7}, {'$set': {'tag_name': 'Pressao'}})
|
||||
tags.update_one({'tag_id': 101}, {'$set': {'address': 'ns=3;i=3001'}})
|
||||
tags.delete_one({'tag_id': 102})
|
||||
finally:
|
||||
mongo.close()
|
||||
|
||||
await _run_tag_database_scenario(temporal_env, 'after-change')
|
||||
|
||||
scouter = await _schedule_input(temporal_host, 'opc-tagdb', namespace='scouter')
|
||||
assert set(scouter['model_tags']) == {'Pressao'}
|
||||
pred = await _schedule_input(temporal_host, 'pred-tagdb')
|
||||
assert pred['opc_output_config'] == {
|
||||
'plant': {'prediction_tags': {'ns=3;i=3001': {'data_type': 'double'}}}
|
||||
}
|
||||
redeployed = _orchestrated_schedules(mongo_uri)
|
||||
for name in ('opc-tagdb', 'pred-tagdb'):
|
||||
assert redeployed[name]['config_hash'] != deployed[name]['config_hash']
|
||||
assert redeployed[name]['updated_at'] > deployed[name]['updated_at']
|
||||
# The slot names the read tag like the scouter does (SC1)
|
||||
[slot] = _opc_slots(redis_client).values()
|
||||
[name] = slot['plant']['tags']['ns=3;i=1001']['names']
|
||||
assert name['tag_name'] == 'Pressao'
|
||||
|
||||
|
||||
@pytest.mark.e2e
|
||||
@pytest.mark.asyncio
|
||||
async def test_scenario_1_9_3_opc_slot_from_tag_database(
|
||||
temporal_env: WorkflowEnvironment,
|
||||
orchestrator_worker,
|
||||
stub_workers,
|
||||
mongo_uri: str,
|
||||
redis_client: Redis,
|
||||
):
|
||||
"""1.9.3: the slot v2 holds only the active Read tags of the tag database."""
|
||||
_apply_orchestrator_seeds(
|
||||
load_scenario_input('orchestrator_tag_database'), mongo_uri, redis_client
|
||||
)
|
||||
|
||||
await _run_tag_database_scenario(temporal_env, 'slot')
|
||||
|
||||
assert _opc_slots(redis_client) == {
|
||||
'slot:opc_tags:1': {
|
||||
'plant': {
|
||||
'server_id': 'plant',
|
||||
'name': 'plant',
|
||||
'url': 'opc.tcp://plant:4840',
|
||||
'subscription_period_ms': 500,
|
||||
# The active client certificate of client_certificate_id, without its
|
||||
# dates, sha256 or audit fields (D5)
|
||||
'security': {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'},
|
||||
'certificate': {
|
||||
'certificate_id': 1,
|
||||
'application_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert': {
|
||||
'bucket': 'opc-certificates',
|
||||
'object_name': 'client/sientia-platform/cert.pem',
|
||||
},
|
||||
'private_key': {
|
||||
'bucket': 'opc-certificates',
|
||||
'object_name': 'client/sientia-platform/key.pem',
|
||||
},
|
||||
},
|
||||
'tags': {
|
||||
'ns=3;i=1001': {
|
||||
'frequency': 1000,
|
||||
'active_read': {'enabled': False},
|
||||
'names': [
|
||||
{
|
||||
'collection': 'raw_opc-tagdb',
|
||||
'tag_name': 'Pressure',
|
||||
'model_id': 'model-opc',
|
||||
'model_name': 'Model model-opc',
|
||||
}
|
||||
],
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -177,14 +177,13 @@ class Email(SientiaMonitoring):
|
||||
self.logger.error(f'SMTP server disconnected: {e}')
|
||||
self.logger.info(f'Reconnecting to {self.smtp_server}:{self.smtp_port}')
|
||||
|
||||
if self.server:
|
||||
try:
|
||||
self.server.quit()
|
||||
except SMTPServerDisconnected as e:
|
||||
self.logger.info(f'Server already disconnected: {e}')
|
||||
except Exception as e:
|
||||
self.logger.error(f'Failed to quit server: {e}')
|
||||
raise e
|
||||
try:
|
||||
self.server.quit()
|
||||
except SMTPServerDisconnected as e:
|
||||
self.logger.info(f'Server already disconnected: {e}')
|
||||
except Exception as e:
|
||||
self.logger.error(f'Failed to quit server: {e}')
|
||||
raise e
|
||||
|
||||
self.server = smtplib.SMTP(self.smtp_server, self.smtp_port, timeout=20)
|
||||
if self.sender_password:
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from temporalio import activity, workflow
|
||||
|
||||
with workflow.unsafe.imports_passed_through():
|
||||
import hashlib
|
||||
import json
|
||||
from collections.abc import Callable
|
||||
from logging import Logger
|
||||
@@ -17,21 +18,95 @@ with workflow.unsafe.imports_passed_through():
|
||||
now,
|
||||
)
|
||||
|
||||
from orchestrator.utils.orchestrator_functions import (
|
||||
build_tag_config,
|
||||
from orchestrator.utils.opc_slots import build_tag_config, gather_read_tags
|
||||
from orchestrator.utils.pipeline_configs import (
|
||||
chunk_period_warning,
|
||||
drift,
|
||||
frequency_alert_warning,
|
||||
gather_read_tags,
|
||||
minimal_retrain,
|
||||
pi_web_api_scouter,
|
||||
predictions_batch,
|
||||
scouter,
|
||||
simple_metrics,
|
||||
)
|
||||
from orchestrator.utils.tag_database import (
|
||||
WRITE_CATEGORIES,
|
||||
WRITE_WORKFLOW_TYPES,
|
||||
index_connector_tags,
|
||||
index_connectors,
|
||||
resolve_tag_names,
|
||||
resolve_write_tags,
|
||||
)
|
||||
|
||||
topic_separator = '\n ========== \n'
|
||||
|
||||
# Schedule fields left out of config_hash: updated_at is compared on its own
|
||||
CONFIG_HASH_EXCLUDED_FIELDS = ('updated_at', 'config_hash')
|
||||
|
||||
# Read tags left out of the slots, by gather_read_tags issue kind:
|
||||
# (kind, notification_id, message, level)
|
||||
SLOT_ISSUES = (
|
||||
(
|
||||
'tags',
|
||||
'ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION',
|
||||
'Tags {} not found among the active, complete Read tags of connector_tags',
|
||||
NotificationLevel.ERROR,
|
||||
),
|
||||
(
|
||||
'connectors',
|
||||
'ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION',
|
||||
'Connectors {} not found, not active or incomplete in connectors',
|
||||
NotificationLevel.ERROR,
|
||||
),
|
||||
(
|
||||
'invalid',
|
||||
'ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION',
|
||||
'Read tags {} have no integer tag_id, no integer frequency or an active_read without an '
|
||||
'integer cycle_multiple; or their pipeline lacks schedule_name, model_id or model.name',
|
||||
NotificationLevel.ERROR,
|
||||
),
|
||||
(
|
||||
'certificates',
|
||||
'ORCHESTRATOR_CERTIFICATE_NOT_FOUND_DURING_SLOT_CONFIGURATION',
|
||||
'Connectors {} have a security.client_certificate_id without an active client '
|
||||
'certificate in opc_certificates: they go into the slot without a certificate, so '
|
||||
'without security',
|
||||
NotificationLevel.WARNING,
|
||||
),
|
||||
)
|
||||
|
||||
# Write tags left out of the predictions_batch opc_output_config, by resolve_write_tags
|
||||
# issue kind; values are schedule_name/<tag_id or connector_id>
|
||||
WRITE_TAG_ISSUES = (
|
||||
(
|
||||
'tags',
|
||||
'ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION',
|
||||
'Write tags {} not found among the active, complete Write and Prediction Confidence '
|
||||
'tags of connector_tags',
|
||||
NotificationLevel.ERROR,
|
||||
),
|
||||
(
|
||||
'connectors',
|
||||
'ORCHESTRATOR_WRITE_TAG_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION',
|
||||
'Connectors {} of write tags not found, not active or incomplete in connectors',
|
||||
NotificationLevel.ERROR,
|
||||
),
|
||||
(
|
||||
'invalid',
|
||||
'ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION',
|
||||
'Write tags {} have no integer tag_id or a data_type the OPC Gateway does not accept',
|
||||
NotificationLevel.ERROR,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _merge_issues(
|
||||
issues: dict[str, set[str]], pipeline_issues: dict[str, set[str]], schedule_name: str
|
||||
) -> None:
|
||||
"""Add one pipeline's issues to the run's, each value prefixed with schedule_name/."""
|
||||
for kind, values in pipeline_issues.items():
|
||||
issues.setdefault(kind, set()).update(f'{schedule_name}/{value}' for value in values)
|
||||
|
||||
|
||||
class ScheduleType(TypedDict):
|
||||
"""
|
||||
@@ -103,6 +178,20 @@ Each entry specifies:
|
||||
"""
|
||||
|
||||
|
||||
def schedule_config_hash(schedule: dict[str, Any]) -> str:
|
||||
"""
|
||||
Fingerprint of a schedule as process_schedules builds it, without updated_at.
|
||||
|
||||
Stored in orchestrated_schedules when the schedule is deployed: any change to what
|
||||
reaches Temporal (a pipeline edited without bumping updated_at, a tag renamed, moved,
|
||||
retyped, deactivated or deleted, a connector changed, or a new Orchestrator version
|
||||
building it differently) changes the hash and redeploys the schedule.
|
||||
"""
|
||||
content = {k: v for k, v in schedule.items() if k not in CONFIG_HASH_EXCLUDED_FIELDS}
|
||||
serialized = json.dumps(content, sort_keys=True, separators=(',', ':'), default=str)
|
||||
return hashlib.sha256(serialized.encode()).hexdigest()
|
||||
|
||||
|
||||
class Formatters(SientiaMonitoring):
|
||||
"""
|
||||
Schedule and slot configuration formatting and notification filtering activity.
|
||||
@@ -175,6 +264,16 @@ class Formatters(SientiaMonitoring):
|
||||
- input_data (dict[str, Any]): The input data containing
|
||||
the schedules to process.
|
||||
- pipelines (list[dict[str, Any]]): The schedules to process.
|
||||
- connectors (list[dict[str, Any]], optional): OPC UA connectors of the
|
||||
tag database, for the write tags.
|
||||
- connector_tags (list[dict[str, Any]], optional): The connector_tags the
|
||||
pipelines reference, of every category and status.
|
||||
|
||||
Scouter read tags that carry only a tag_id are named from the tag database (the
|
||||
same names as the slot). A predictions_batch gets its opc_output_config from its
|
||||
write_tags ([{tag_id}]) through the tag database; what cannot be resolved is left
|
||||
out and notified once per run. A change to a tag reaches the schedule through
|
||||
create_schedule_config, which compares the resolved schedule (config_hash).
|
||||
|
||||
Returns:
|
||||
- dict[str, Any]: The schedule configuration dictionary keyed by namespace
|
||||
@@ -185,6 +284,11 @@ class Formatters(SientiaMonitoring):
|
||||
self.info('Processing schedules...', metadata=metadata)
|
||||
|
||||
pipelines = input_data['pipelines']
|
||||
connector_tags = input_data.get('connector_tags', [])
|
||||
read_tags_by_id = index_connector_tags(connector_tags)
|
||||
write_tags_by_id = index_connector_tags(connector_tags, WRITE_CATEGORIES)
|
||||
connectors = index_connectors(input_data.get('connectors', []))
|
||||
write_issues: dict[str, set[str]] = {}
|
||||
|
||||
schedule_config: dict[str, dict[str, Any]] = {
|
||||
self.scouter_namespace: {},
|
||||
@@ -201,14 +305,31 @@ class Formatters(SientiaMonitoring):
|
||||
schedule_type = schedule_types[workflow_type]
|
||||
namespace = schedule_type['namespace']
|
||||
function = schedule_type['function']
|
||||
schedule_name = pipeline['schedule_name']
|
||||
|
||||
schedule_config[namespace][pipeline['schedule_name']] = {
|
||||
if workflow_type == 'scouter' and 'read_tags' in pipeline:
|
||||
# Same names as the slot's names[] (process_slots): SIENTIAPDE-1974 SC1
|
||||
pipeline = {
|
||||
**pipeline,
|
||||
'read_tags': resolve_tag_names(pipeline['read_tags'], read_tags_by_id),
|
||||
}
|
||||
|
||||
if workflow_type in WRITE_WORKFLOW_TYPES:
|
||||
opc_output_config, issues = resolve_write_tags(
|
||||
pipeline.get('write_tags') or [], write_tags_by_id, connectors
|
||||
)
|
||||
_merge_issues(write_issues, issues, schedule_name)
|
||||
pipeline = {**pipeline, 'opc_output_config': opc_output_config}
|
||||
|
||||
schedule_config[namespace][schedule_name] = {
|
||||
**function(pipeline),
|
||||
'updated_at': pipeline.get(
|
||||
'updated_at', now().strftime(DATETIME_FORMAT_MS_WITH_TZ)
|
||||
),
|
||||
# A pipeline without updated_at is redeployed on every run, as before
|
||||
'updated_at': pipeline.get('updated_at')
|
||||
or now().strftime(DATETIME_FORMAT_MS_WITH_TZ),
|
||||
}
|
||||
|
||||
self._notify_issues(write_issues, WRITE_TAG_ISSUES, metadata)
|
||||
|
||||
self.info('Processed schedules', metadata=metadata)
|
||||
self.debug(json.dumps(schedule_config, indent=4, sort_keys=True), metadata=metadata)
|
||||
|
||||
@@ -218,14 +339,25 @@ class Formatters(SientiaMonitoring):
|
||||
def process_slots(self, input_data: dict[str, Any]) -> dict[str, Any]:
|
||||
"""
|
||||
Extracts all read tags from input pipelines, divides them into slots and
|
||||
returns a slot config dictionary. If no ingestor is available, only one slot
|
||||
is created.
|
||||
returns a slot config dictionary in the slot v2 format (contract C1). If no
|
||||
ingestor is available, only one slot is created.
|
||||
|
||||
The division is by address: all the names of an address stay in the same slot
|
||||
(SIENTIAPDE-1974, O5). Read tags that cannot be resolved are left out and
|
||||
notified once per run: a read tag without tag_id or a usable frequency, a tag_id
|
||||
missing from connector_tags, or a connector missing or inactive. A connector whose
|
||||
client_certificate_id has no active client certificate goes into the slot without
|
||||
a certificate, with a warning (D5).
|
||||
|
||||
Args:
|
||||
- input_data (dict[str, Any]): The input data containing
|
||||
the schedules to process.
|
||||
- pipelines (list[dict[str, Any]]): The schedules to process.
|
||||
- opc_servers (list[str]): The OPC servers to create ingestor config.
|
||||
- connectors (list[dict[str, Any]], optional): Active OPC UA connectors of
|
||||
the tag database, the only source of the OPC servers, each with the
|
||||
certificates of its client_certificate_id (connectors_query).
|
||||
- connector_tags (list[dict[str, Any]], optional): The connector_tags the
|
||||
pipelines reference; only the active Read tags are used.
|
||||
- active_ingestors (list[str]): The active ingestors to divide into slots.
|
||||
|
||||
Returns:
|
||||
@@ -237,55 +369,52 @@ class Formatters(SientiaMonitoring):
|
||||
self.info('Processing slots...', metadata=metadata)
|
||||
|
||||
pipelines = input_data['pipelines']
|
||||
opc_servers_list = input_data['opc_servers']
|
||||
active_ingestors = input_data['active_ingestors']
|
||||
connectors = index_connectors(input_data.get('connectors', []))
|
||||
tags_by_id = index_connector_tags(input_data.get('connector_tags', []))
|
||||
|
||||
opc_servers = {}
|
||||
for server in opc_servers_list:
|
||||
opc_servers[server['id']] = {
|
||||
**server,
|
||||
}
|
||||
entries, issues = gather_read_tags(pipelines, connectors, tags_by_id)
|
||||
self._notify_issues(issues, SLOT_ISSUES, metadata)
|
||||
|
||||
tags = list(gather_read_tags(pipelines).values())
|
||||
|
||||
number_of_tags = len(tags)
|
||||
number_of_slots = len(active_ingestors) if active_ingestors else 1
|
||||
tags_per_slot = ceil(number_of_tags / number_of_slots)
|
||||
entries_per_slot = ceil(len(entries) / number_of_slots)
|
||||
|
||||
slot_config: dict[str, Any] = {}
|
||||
last_index = 0
|
||||
|
||||
for i in range(1, number_of_slots):
|
||||
slot_tags = tags[last_index : last_index + tags_per_slot]
|
||||
slot_config[f'{i}'], notifications = build_tag_config(slot_tags, opc_servers)
|
||||
|
||||
if notifications:
|
||||
self.send_notification(
|
||||
metadata=metadata,
|
||||
notification_id='ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION',
|
||||
message=f'Servers {", ".join(notifications)} not found in opc_servers',
|
||||
block='orchestrator',
|
||||
level=NotificationLevel.ERROR,
|
||||
)
|
||||
|
||||
last_index += tags_per_slot
|
||||
|
||||
slot_tags = tags[last_index:]
|
||||
slot_config[f'{number_of_slots}'], notifications = build_tag_config(slot_tags, opc_servers)
|
||||
if notifications:
|
||||
self.send_notification(
|
||||
metadata=metadata,
|
||||
notification_id='ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION',
|
||||
message=f'Servers {", ".join(notifications)} not found in opc_servers',
|
||||
block='orchestrator',
|
||||
level=NotificationLevel.ERROR,
|
||||
)
|
||||
for i in range(number_of_slots):
|
||||
slot_entries = entries[i * entries_per_slot : (i + 1) * entries_per_slot]
|
||||
slot_config[f'{i + 1}'] = build_tag_config(slot_entries)
|
||||
|
||||
self.info('Processed slots', metadata=metadata)
|
||||
self.debug(json.dumps(slot_config, indent=4, sort_keys=True), metadata=metadata)
|
||||
|
||||
return slot_config
|
||||
|
||||
def _notify_issues(
|
||||
self,
|
||||
issues: dict[str, set[str]],
|
||||
kinds: tuple[tuple[str, str, str, str], ...],
|
||||
metadata: dict[str, Any],
|
||||
) -> None:
|
||||
"""
|
||||
Notify what was left out of the slots or schedules, one notification per kind.
|
||||
|
||||
Args:
|
||||
issues (dict[str, set[str]]): What was left out, by kind
|
||||
kinds (tuple[tuple[str, str, str, str], ...]): (kind, notification_id, message
|
||||
with one {} for the joined values, level), e.g. SLOT_ISSUES or
|
||||
WRITE_TAG_ISSUES
|
||||
metadata (dict[str, Any]): Metadata for the notifications
|
||||
"""
|
||||
for kind, notification_id, message, level in kinds:
|
||||
if issues.get(kind):
|
||||
self.send_notification(
|
||||
metadata=metadata,
|
||||
notification_id=notification_id,
|
||||
message=message.format(', '.join(sorted(issues[kind]))),
|
||||
block='orchestrator',
|
||||
level=level,
|
||||
)
|
||||
|
||||
@activity.defn(name='format_schedule_config')
|
||||
def format_schedule_config(self, input_data: dict[str, Any]) -> dict[str, Any]:
|
||||
"""
|
||||
@@ -293,6 +422,8 @@ class Formatters(SientiaMonitoring):
|
||||
|
||||
Transforms a list of schedule configurations into a nested dictionary structure
|
||||
organized by namespace and schedule name for efficient lookup and comparison.
|
||||
Each schedule keeps when it was last deployed (updated_at) and the config_hash
|
||||
it was deployed with (None for schedules deployed before config_hash existed).
|
||||
|
||||
Args:
|
||||
input_data (dict[str, Any]): The input data containing:
|
||||
@@ -300,7 +431,7 @@ class Formatters(SientiaMonitoring):
|
||||
- metadata (dict): Metadata for logging purposes
|
||||
|
||||
Returns:
|
||||
dict[str, Any]: The formatted schedule config organized by namespace and schedule name
|
||||
dict[str, Any]: {namespace: {schedule_name: {'updated_at', 'config_hash'}}}
|
||||
"""
|
||||
metadata = input_data['metadata']
|
||||
|
||||
@@ -308,16 +439,18 @@ class Formatters(SientiaMonitoring):
|
||||
|
||||
schedule_config = input_data['schedule_config']
|
||||
|
||||
config: dict[str, dict[str, str]] = {}
|
||||
config: dict[str, dict[str, dict[str, Any]]] = {}
|
||||
for schedule in schedule_config:
|
||||
namespace = schedule['namespace']
|
||||
schedule_name = schedule['schedule_name']
|
||||
updated_at = schedule['updated_at']
|
||||
|
||||
if namespace not in config:
|
||||
config[namespace] = {}
|
||||
|
||||
config[namespace][schedule_name] = updated_at
|
||||
config[namespace][schedule_name] = {
|
||||
'updated_at': schedule['updated_at'],
|
||||
'config_hash': schedule.get('config_hash'),
|
||||
}
|
||||
|
||||
self.info('Formatted schedule config', metadata=metadata)
|
||||
self.debug(json.dumps(config, indent=4, sort_keys=True), metadata=metadata)
|
||||
@@ -336,32 +469,42 @@ class Formatters(SientiaMonitoring):
|
||||
"""
|
||||
Compare new and current schedules to determine which should be updated or created.
|
||||
|
||||
Compares schedule timestamps to identify schedules that need updating (newer timestamp)
|
||||
or creating (schedule doesn't exist). Results are accumulated in the provided dictionaries.
|
||||
A deployed schedule is updated when its updated_at is newer than its last deploy
|
||||
(bumping updated_at forces a redeploy) or when its config_hash differs from the one
|
||||
it was deployed with (any change to the resolved schedule, including tag database
|
||||
changes and deleted tags). A schedule not deployed yet is created. Each schedule
|
||||
sent to update or create carries its config_hash, stored once it is deployed.
|
||||
Results are accumulated in the provided dictionaries.
|
||||
|
||||
Args:
|
||||
schedules (dict[str, Any]): New schedules to compare
|
||||
current_schedules (dict[str, Any]): Existing schedules to compare against
|
||||
current_schedules (dict[str, Any]): Existing schedules to compare against, as
|
||||
format_schedule_config gives them
|
||||
to_update (dict[str, Any]): Output accumulator for schedules that need updating
|
||||
to_create (dict[str, Any]): Output accumulator for schedules that need creating
|
||||
namespace (str): Namespace for the schedules being compared
|
||||
metadata (dict[str, Any]): Metadata for logging purposes
|
||||
"""
|
||||
for schedule_name, schedule in schedules.items():
|
||||
config_hash = schedule_config_hash(schedule)
|
||||
deployed = {**schedule, 'config_hash': config_hash}
|
||||
|
||||
if schedule_name in current_schedules:
|
||||
update_timestamp = schedule.get('updated_at', now())
|
||||
|
||||
old_timestamp = current_schedules[schedule_name]
|
||||
current = current_schedules[schedule_name]
|
||||
old_timestamp = current['updated_at']
|
||||
|
||||
self.debug(
|
||||
f'Comparing schedule {schedule_name}:{update_timestamp} vs {old_timestamp}',
|
||||
f'Comparing schedule {schedule_name}:{update_timestamp} vs {old_timestamp}'
|
||||
f' (config_hash {config_hash} vs {current.get("config_hash")})',
|
||||
metadata=metadata,
|
||||
)
|
||||
|
||||
if update_timestamp > old_timestamp:
|
||||
to_update[namespace][schedule_name] = schedule
|
||||
if update_timestamp > old_timestamp or config_hash != current.get('config_hash'):
|
||||
to_update[namespace][schedule_name] = deployed
|
||||
else:
|
||||
to_create[namespace][schedule_name] = schedule
|
||||
to_create[namespace][schedule_name] = deployed
|
||||
|
||||
@activity.defn(name='create_schedule_config')
|
||||
def create_schedule_config(self, input_data: dict[str, Any]) -> dict[str, Any]:
|
||||
@@ -370,7 +513,8 @@ class Formatters(SientiaMonitoring):
|
||||
|
||||
Compares the current schedule configuration in Temporal with the new schedule
|
||||
configuration to determine which schedules need to be created, updated, or deleted.
|
||||
Uses timestamp comparison to identify schedules that have changed.
|
||||
Uses the updated_at timestamps and the config_hash (compare_config_timestamps) to
|
||||
identify schedules that have changed.
|
||||
Also sends a SIMPLE_METRICS_FREQUENCY_ALERT_BELOW_FLOOR WARNING for laborious schedules
|
||||
whose frequency_alert is below 1m (the value is kept) or unparseable (it is ignored),
|
||||
and a DRIFT_CHUNK_PERIOD_INVALID WARNING for drift schedules whose configured
|
||||
@@ -415,8 +559,8 @@ class Formatters(SientiaMonitoring):
|
||||
schedules, current_schedules, to_update, to_create, namespace, metadata
|
||||
)
|
||||
|
||||
# Fires whenever the schedule is created or updated (each `updated_at` change,
|
||||
# including escalation flips).
|
||||
# Fires whenever the schedule is created or updated (each `updated_at` or
|
||||
# config_hash change, including escalation flips).
|
||||
changed = {**to_create[self.laborious_namespace], **to_update[self.laborious_namespace]}
|
||||
for schedule_name, schedule in changed.items():
|
||||
warning = frequency_alert_warning(schedule.get('frequency_alert'))
|
||||
@@ -469,15 +613,21 @@ class Formatters(SientiaMonitoring):
|
||||
to determine which slots need to be inserted or deleted. Slots are identified
|
||||
by numeric IDs, and excess slots are marked for deletion.
|
||||
|
||||
Only slots whose content changed are inserted, so an unchanged slot is not
|
||||
rewritten (nor reported) on every run. gather_read_tags builds the slot in a
|
||||
stable order, so the same pipelines always give the same slot (SIENTIAPDE-1974, O6).
|
||||
A slot in the v1 format differs from its v2 rewrite and is replaced once.
|
||||
|
||||
Args:
|
||||
input_data (dict[str, Any]): The input data containing:
|
||||
- current_slot_config (dict[str, Any]): The current slot config in Redis
|
||||
- current_slot_config (dict[str, Any]): The current slot config in Redis,
|
||||
keyed by Redis key (slot:opc_tags:{id}) or by slot id
|
||||
- slot_config (dict[str, Any]): The slot config to process
|
||||
- metadata (dict): Metadata for logging purposes
|
||||
|
||||
Returns:
|
||||
dict[str, Any]: A dictionary containing:
|
||||
- to_insert (dict): Slots that need to be inserted/updated
|
||||
- to_insert (dict): Slots that are new or changed
|
||||
- to_delete (list[str]): Slot IDs that need to be deleted
|
||||
"""
|
||||
|
||||
@@ -495,7 +645,16 @@ class Formatters(SientiaMonitoring):
|
||||
if number_of_current_slots > number_of_slots:
|
||||
to_delete = [str(i) for i in range(number_of_slots + 1, number_of_current_slots + 1)]
|
||||
|
||||
output = {'to_delete': to_delete, 'to_insert': slot_config}
|
||||
current_by_id = {
|
||||
str(key).rsplit(':', 1)[-1]: config for key, config in current_slot_config.items()
|
||||
}
|
||||
to_insert = {
|
||||
slot: config
|
||||
for slot, config in slot_config.items()
|
||||
if slot not in current_by_id or current_by_id[slot] != config
|
||||
}
|
||||
|
||||
output = {'to_delete': to_delete, 'to_insert': to_insert}
|
||||
|
||||
self.info('Created slot config', metadata=metadata)
|
||||
self.debug(json.dumps(output, indent=4, sort_keys=True), metadata=metadata)
|
||||
|
||||
@@ -193,7 +193,8 @@ class MongoDB(SientiaMonitoring):
|
||||
|
||||
for document in aggregated_documents:
|
||||
for timestamp_field in timestamp_fields:
|
||||
if timestamp_field in document:
|
||||
# Only BSON dates: a field already stored as text is left as it is
|
||||
if isinstance(document.get(timestamp_field), datetime):
|
||||
document[timestamp_field] = (
|
||||
document[timestamp_field]
|
||||
.replace(tzinfo=UTC)
|
||||
@@ -223,12 +224,14 @@ class MongoDB(SientiaMonitoring):
|
||||
"""
|
||||
Update `updated_at` timestamps for successfully updated pipelines.
|
||||
|
||||
Updates the `updated_at` field in the `orchestrated_schedules` collection
|
||||
for all pipelines that were successfully updated in Temporal.
|
||||
Updates the `updated_at` and `config_hash` fields in the `orchestrated_schedules`
|
||||
collection for all pipelines that were successfully updated in Temporal, so the
|
||||
next run compares against what was deployed.
|
||||
|
||||
Args:
|
||||
input_data (dict[str, Any]): Input data containing:
|
||||
- updated_pipelines (list[dict]): Pipelines with success flags to consider
|
||||
- updated_pipelines (list[dict]): Pipelines with success flags (and
|
||||
config_hash) to consider
|
||||
- metadata (dict): Metadata for logging purposes
|
||||
"""
|
||||
updated_pipelines = input_data.get('updated_pipelines', [])
|
||||
@@ -237,34 +240,53 @@ class MongoDB(SientiaMonitoring):
|
||||
|
||||
self.info('Updating pipelines timestamps...', metadata=metadata)
|
||||
|
||||
argument = [
|
||||
{'schedule_name': pipeline['schedule_name'], 'namespace': pipeline['namespace']}
|
||||
for pipeline in updated_pipelines
|
||||
if pipeline['success']
|
||||
]
|
||||
if not argument:
|
||||
succeeded = [pipeline for pipeline in updated_pipelines if pipeline['success']]
|
||||
if not succeeded:
|
||||
# Every update failed: stamp nothing so they are retried next tick (same guard as
|
||||
# create_pipelines_timestamps). An empty filter would match every tracked schedule.
|
||||
# create_pipelines_timestamps).
|
||||
self.info('No successful pipeline updates to timestamp', metadata=metadata)
|
||||
return
|
||||
data_filter = {'$or': argument}
|
||||
|
||||
try:
|
||||
self.mongo_db_repository.update_many(
|
||||
'orchestrated_schedules', data_filter, {'$set': {'updated_at': date_now}}, metadata
|
||||
)
|
||||
except Exception as e:
|
||||
trace = traceback.format_exc()
|
||||
# One update per schedule (each has its own config_hash). A failure does not stop
|
||||
# the others: they are all tried, and the failures are notified together.
|
||||
failures: dict[str, str] = {}
|
||||
traces: list[str] = []
|
||||
for pipeline in succeeded:
|
||||
schedule = f'{pipeline["namespace"]}/{pipeline["schedule_name"]}'
|
||||
try:
|
||||
self.mongo_db_repository.update_one(
|
||||
'orchestrated_schedules',
|
||||
{
|
||||
'schedule_name': pipeline['schedule_name'],
|
||||
'namespace': pipeline['namespace'],
|
||||
},
|
||||
{
|
||||
'$set': {
|
||||
'updated_at': date_now,
|
||||
'config_hash': pipeline.get('config_hash'),
|
||||
}
|
||||
},
|
||||
metadata,
|
||||
)
|
||||
except Exception as e:
|
||||
failures[schedule] = str(e)
|
||||
traces.append(f'{schedule}\n{traceback.format_exc()}')
|
||||
|
||||
if failures:
|
||||
trace = '\n'.join(traces)
|
||||
details = '; '.join(f'{schedule}: {error}' for schedule, error in failures.items())
|
||||
self.send_notification(
|
||||
metadata=metadata,
|
||||
notification_id='MONGODB_UPDATE_PIPELINES_ERROR',
|
||||
message=f'Failed to update pipelines timestamps: {e}',
|
||||
message=f'Failed to update pipelines timestamps: {details}',
|
||||
block='update_pipelines_timestamps',
|
||||
level=NotificationLevel.ERROR,
|
||||
attachment_content=trace,
|
||||
)
|
||||
self.error(trace, metadata=metadata)
|
||||
raise e
|
||||
raise RuntimeError(
|
||||
f'Failed to update {len(failures)} of {len(succeeded)} pipelines timestamps'
|
||||
)
|
||||
|
||||
self.info(
|
||||
f'Updated {len(updated_pipelines)} pipelines timestamps',
|
||||
@@ -277,11 +299,13 @@ class MongoDB(SientiaMonitoring):
|
||||
Insert `updated_at` timestamps for newly created pipelines.
|
||||
|
||||
Inserts new documents into the `orchestrated_schedules` collection
|
||||
for all pipelines that were successfully created in Temporal.
|
||||
for all pipelines that were successfully created in Temporal, with the
|
||||
`config_hash` they were created with.
|
||||
|
||||
Args:
|
||||
input_data (dict[str, Any]): Input data containing:
|
||||
- created_pipelines (list[dict]): Pipelines with success flags to consider
|
||||
- created_pipelines (list[dict]): Pipelines with success flags (and
|
||||
config_hash) to consider
|
||||
- metadata (dict): Metadata for logging purposes
|
||||
"""
|
||||
created_pipelines = input_data.get('created_pipelines', [])
|
||||
@@ -296,6 +320,7 @@ class MongoDB(SientiaMonitoring):
|
||||
'schedule_name': pipeline['schedule_name'],
|
||||
'namespace': pipeline['namespace'],
|
||||
'updated_at': date_now,
|
||||
'config_hash': pipeline.get('config_hash'),
|
||||
}
|
||||
for pipeline in created_pipelines
|
||||
if pipeline['success']
|
||||
|
||||
@@ -177,7 +177,9 @@ class TemporalManager(SientiaMonitoring):
|
||||
- schedules (dict[str, Any]): The schedules to create.
|
||||
|
||||
Returns:
|
||||
- list[dict[str, Any]]: Report entries for each attempted schedule creation.
|
||||
- list[dict[str, Any]]: Report entries for each attempted schedule creation,
|
||||
with the schedule's config_hash (create_schedule_config) for
|
||||
create_pipelines_timestamps.
|
||||
"""
|
||||
|
||||
schedules_to_create = input_data['schedules']
|
||||
@@ -198,6 +200,8 @@ class TemporalManager(SientiaMonitoring):
|
||||
)
|
||||
|
||||
for schedule_name, schedule in schedules.items():
|
||||
# Stored in orchestrated_schedules, not sent to the workflow
|
||||
config_hash = schedule.pop('config_hash', None)
|
||||
search_attributes = TypedSearchAttributes(
|
||||
[
|
||||
SearchAttributePair(key=self.model_id_id_key, value=schedule['model_id']),
|
||||
@@ -258,6 +262,7 @@ class TemporalManager(SientiaMonitoring):
|
||||
'schedule_name': schedule_name,
|
||||
'success': True,
|
||||
'message': 'Schedule created successfully',
|
||||
'config_hash': config_hash,
|
||||
}
|
||||
)
|
||||
|
||||
@@ -272,6 +277,7 @@ class TemporalManager(SientiaMonitoring):
|
||||
'schedule_name': schedule_name,
|
||||
'success': False,
|
||||
'message': str(e),
|
||||
'config_hash': config_hash,
|
||||
}
|
||||
)
|
||||
|
||||
@@ -360,7 +366,9 @@ class TemporalManager(SientiaMonitoring):
|
||||
- schedules (dict[str, Any]): The schedules to update.
|
||||
|
||||
Returns:
|
||||
- list[dict[str, Any]]: Report entries for each attempted schedule update.
|
||||
- list[dict[str, Any]]: Report entries for each attempted schedule update,
|
||||
with the schedule's config_hash (create_schedule_config) for
|
||||
update_pipelines_timestamps.
|
||||
"""
|
||||
|
||||
schedules_to_update = input_data['schedules']
|
||||
@@ -380,6 +388,8 @@ class TemporalManager(SientiaMonitoring):
|
||||
)
|
||||
|
||||
for schedule_name, schedule in schedules.items():
|
||||
# Stored in orchestrated_schedules, not sent to the workflow
|
||||
config_hash = schedule.pop('config_hash', None)
|
||||
try:
|
||||
await self._update_single_schedule(client, schedule_name, schedule, metadata)
|
||||
|
||||
@@ -389,6 +399,7 @@ class TemporalManager(SientiaMonitoring):
|
||||
'schedule_name': schedule_name,
|
||||
'success': True,
|
||||
'message': 'Schedule updated successfully',
|
||||
'config_hash': config_hash,
|
||||
}
|
||||
)
|
||||
|
||||
@@ -403,6 +414,7 @@ class TemporalManager(SientiaMonitoring):
|
||||
'schedule_name': schedule_name,
|
||||
'success': False,
|
||||
'message': str(e),
|
||||
'config_hash': config_hash,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
301
orchestrator/utils/opc_slots.py
Normal file
301
orchestrator/utils/opc_slots.py
Normal file
@@ -0,0 +1,301 @@
|
||||
"""OPC slot v2 (contract C1): read tags grouped by server and address for the gateway."""
|
||||
|
||||
from typing import Any, cast
|
||||
|
||||
from orchestrator.utils.tag_database import (
|
||||
client_certificate_id,
|
||||
is_complete,
|
||||
resolve_certificate,
|
||||
resolve_tag_names,
|
||||
tag_id_of,
|
||||
)
|
||||
|
||||
|
||||
def _connector_server(connector: dict[str, Any], issues: dict[str, set[str]]) -> dict[str, Any]:
|
||||
"""
|
||||
Slot server fields of a tag database connector (connectors collection).
|
||||
|
||||
A connector whose client_certificate_id has no active client certificate still goes
|
||||
into the slot, without a certificate, and is recorded in issues['certificates'] (D5).
|
||||
"""
|
||||
certificate = resolve_certificate(connector)
|
||||
if certificate is None and client_certificate_id(connector) is not None:
|
||||
issues['certificates'].add(str(connector['connector_id']))
|
||||
return {
|
||||
'server_id': str(connector['connector_id']),
|
||||
'name': connector['name'],
|
||||
'url': connector['url'],
|
||||
'subscription_period_ms': connector.get('subscription_period_ms'),
|
||||
'security': {
|
||||
'mode': connector['security']['mode'],
|
||||
'policy': connector['security']['policy'],
|
||||
},
|
||||
'certificate': certificate,
|
||||
}
|
||||
|
||||
|
||||
def _frequency(tag: dict[str, Any]) -> int | None:
|
||||
"""The read tag's frequency as int, or None when it is missing or not an integer."""
|
||||
try:
|
||||
return int(tag['frequency'])
|
||||
except (KeyError, TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _locate_read_tag(
|
||||
tag_id: int,
|
||||
connectors: dict[str, dict[str, Any]],
|
||||
tags_by_id: dict[int, dict[str, Any]],
|
||||
issues: dict[str, set[str]],
|
||||
) -> tuple[dict[str, Any], str] | None:
|
||||
"""
|
||||
Server and address of a read tag, or None when they cannot be resolved.
|
||||
|
||||
The address and connector come from connector_tags and the server from connectors
|
||||
(RF3). What cannot be resolved is recorded in issues ('tags' or 'connectors'), and a
|
||||
server without its registered certificate in issues['certificates'].
|
||||
"""
|
||||
connector_tag = tags_by_id.get(tag_id)
|
||||
if connector_tag is None:
|
||||
issues['tags'].add(str(tag_id))
|
||||
return None
|
||||
connector_id = str(connector_tag['connector_id'])
|
||||
connector = connectors.get(connector_id)
|
||||
if connector is None:
|
||||
issues['connectors'].add(connector_id)
|
||||
return None
|
||||
return _connector_server(connector, issues), connector_tag['address']
|
||||
|
||||
|
||||
def _cycle_multiple(active_read: Any) -> int | None:
|
||||
"""The cycle_multiple of an enabled active_read as int (default 1), or None when invalid."""
|
||||
try:
|
||||
return int(active_read.get('cycle_multiple', 1))
|
||||
except (AttributeError, TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _invalid_read_tag(tag: dict[str, Any]) -> bool:
|
||||
"""
|
||||
A read tag the slot cannot use: no integer frequency, or an enabled active_read
|
||||
without an integer cycle_multiple.
|
||||
"""
|
||||
if _frequency(tag) is None:
|
||||
return True
|
||||
active_read = tag.get('active_read')
|
||||
return (
|
||||
isinstance(active_read, dict)
|
||||
and bool(active_read.get('enabled'))
|
||||
and _cycle_multiple(active_read) is None
|
||||
)
|
||||
|
||||
|
||||
def _read_tag_ref(tag: dict[str, Any]) -> Any:
|
||||
"""How an invalid read tag is reported: its tag_id, or its tag_name without one."""
|
||||
return tag.get('tag_id') if tag.get('tag_id') not in (None, '') else tag.get('tag_name')
|
||||
|
||||
|
||||
def _is_slot_pipeline(pipeline: dict[str, Any]) -> bool:
|
||||
"""A scouter pipeline whose read tags can be named in the slot (schedule, model id and name)."""
|
||||
model = pipeline.get('model')
|
||||
return (
|
||||
is_complete(pipeline, ('schedule_name', 'model_id'))
|
||||
and isinstance(model, dict)
|
||||
and is_complete(model, ('name',))
|
||||
)
|
||||
|
||||
|
||||
def _merge_active_read(current: dict[str, Any], requested: dict[str, Any] | None) -> dict[str, Any]:
|
||||
"""Active read of an address: on if any pipeline asks, with the lowest cycle_multiple (D3)."""
|
||||
if not isinstance(requested, dict) or not requested.get('enabled'):
|
||||
return current
|
||||
cycle_multiple = cast(int, _cycle_multiple(requested))
|
||||
if current.get('enabled'):
|
||||
cycle_multiple = min(cycle_multiple, current['cycle_multiple'])
|
||||
return {'enabled': True, 'cycle_multiple': cycle_multiple}
|
||||
|
||||
|
||||
def _assign_server_keys(servers: dict[str, dict[str, Any]]) -> None:
|
||||
"""
|
||||
Give each server a unique slot key: its name, or name:server_id (extended until
|
||||
free) when another server already took it. Assigned in server_id order, so the keys
|
||||
are the same on every run and in every slot.
|
||||
"""
|
||||
taken: set[str] = set()
|
||||
for server_id in sorted(servers):
|
||||
server = servers[server_id]
|
||||
key = server['name']
|
||||
while key in taken:
|
||||
key = f'{key}:{server_id}'
|
||||
server['key'] = key
|
||||
taken.add(key)
|
||||
|
||||
|
||||
def _add_read_tag(
|
||||
entries: dict[tuple[str, str], dict[str, Any]],
|
||||
servers: dict[str, dict[str, Any]],
|
||||
pipeline: dict[str, Any],
|
||||
tag: dict[str, Any],
|
||||
server: dict[str, Any],
|
||||
address: str,
|
||||
) -> None:
|
||||
"""Merge one resolved read tag into its (server, address) entry and its server."""
|
||||
server_id = server['server_id']
|
||||
servers[server_id] = server
|
||||
|
||||
# int, so the slot does not depend on how each pipeline stored the number
|
||||
frequency = cast(int, _frequency(tag))
|
||||
entry = entries.setdefault(
|
||||
(server_id, address),
|
||||
{
|
||||
'address': address,
|
||||
'frequency': frequency,
|
||||
'active_read': {'enabled': False},
|
||||
'names': [],
|
||||
},
|
||||
)
|
||||
entry['frequency'] = min(entry['frequency'], frequency)
|
||||
entry['active_read'] = _merge_active_read(entry['active_read'], tag.get('active_read'))
|
||||
|
||||
name = {
|
||||
'collection': f'raw_{pipeline["schedule_name"]}',
|
||||
'tag_name': tag['tag_name'],
|
||||
'model_id': pipeline['model_id'],
|
||||
'model_name': pipeline['model']['name'],
|
||||
}
|
||||
if name not in entry['names']:
|
||||
entry['names'].append(name)
|
||||
|
||||
|
||||
def _add_pipeline_read_tags(
|
||||
entries: dict[tuple[str, str], dict[str, Any]],
|
||||
servers: dict[str, dict[str, Any]],
|
||||
issues: dict[str, set[str]],
|
||||
pipeline: dict[str, Any],
|
||||
connectors: dict[str, dict[str, Any]],
|
||||
tags_by_id: dict[int, dict[str, Any]],
|
||||
) -> None:
|
||||
"""Add the read tags of one slot pipeline to entries, reporting the ones left out in issues."""
|
||||
for tag in resolve_tag_names(pipeline.get('read_tags') or [], tags_by_id):
|
||||
tag_id = tag_id_of(tag)
|
||||
if tag_id is None or _invalid_read_tag(tag):
|
||||
issues['invalid'].add(f'{pipeline["schedule_name"]}/{_read_tag_ref(tag)}')
|
||||
continue
|
||||
|
||||
located = _locate_read_tag(tag_id, connectors, tags_by_id, issues)
|
||||
if located is None:
|
||||
continue
|
||||
_add_read_tag(entries, servers, pipeline, tag, *located)
|
||||
|
||||
|
||||
def gather_read_tags(
|
||||
pipelines: list[dict[str, Any]],
|
||||
connectors: dict[str, dict[str, Any]],
|
||||
tags_by_id: dict[int, dict[str, Any]],
|
||||
) -> tuple[list[dict[str, Any]], dict[str, set[str]]]:
|
||||
"""
|
||||
Group the read tags of all scouter pipelines by (server, address) for the slot v2.
|
||||
|
||||
An address read by N pipelines becomes ONE entry with N items in names[]: the
|
||||
gateway reads it once and writes it to N collections (SIENTIAPDE-1974, RF2). Each
|
||||
read tag is resolved through the tag database by its tag_id (RF3): connectors is the
|
||||
only source of the OPC servers.
|
||||
|
||||
Args:
|
||||
pipelines (list[dict[str, Any]]): Pipeline configurations. Only workflow_type
|
||||
'scouter' is read; each needs schedule_name, model_id, model.name and
|
||||
read_tags (with tag_id and frequency, and optionally active_read)
|
||||
connectors (dict[str, dict[str, Any]]): Active OPC UA connectors indexed by
|
||||
index_connectors
|
||||
tags_by_id (dict[int, dict[str, Any]]): Active Read connector_tags indexed by
|
||||
index_connector_tags
|
||||
|
||||
Returns:
|
||||
tuple[list[dict[str, Any]], dict[str, set[str]]]:
|
||||
- One entry per (server, address), sorted by server_id then address, with:
|
||||
- server (dict): server_id, name, url, subscription_period_ms (None =
|
||||
derive from the frequencies), security ({mode, policy}), certificate
|
||||
(resolve_certificate; None = connect without security) and key (the
|
||||
server's unique slot key)
|
||||
- address (str): The OPC node address
|
||||
- frequency (int): The lowest frequency among the pipelines (RF6)
|
||||
- active_read (dict): {'enabled': False} unless a pipeline enables it (D3)
|
||||
- names (list[dict]): collection ('raw_{schedule_name}'), tag_name,
|
||||
model_id and model_name, one per pipeline name, sorted
|
||||
- What was left out: 'tags' (tag_id), 'connectors' (connector_id) and
|
||||
'invalid' (schedule_name/read tag without an int tag_id, a usable frequency or
|
||||
cycle_multiple, or the schedule_name of a pipeline without schedule_name,
|
||||
model_id or model.name); and 'certificates': connector_ids that went into the
|
||||
slot without the certificate their client_certificate_id points to
|
||||
"""
|
||||
issues: dict[str, set[str]] = {
|
||||
'tags': set(),
|
||||
'connectors': set(),
|
||||
'invalid': set(),
|
||||
'certificates': set(),
|
||||
}
|
||||
servers: dict[str, dict[str, Any]] = {}
|
||||
entries: dict[tuple[str, str], dict[str, Any]] = {}
|
||||
|
||||
for pipeline in pipelines:
|
||||
if pipeline.get('workflow_type') != 'scouter':
|
||||
continue
|
||||
if not _is_slot_pipeline(pipeline):
|
||||
issues['invalid'].add(str(pipeline.get('schedule_name')))
|
||||
continue
|
||||
_add_pipeline_read_tags(entries, servers, issues, pipeline, connectors, tags_by_id)
|
||||
|
||||
_assign_server_keys(servers)
|
||||
|
||||
gathered = []
|
||||
for (server_id, _address), entry in sorted(entries.items()):
|
||||
# A stable order keeps the slot identical between runs, so the gateway does
|
||||
# not re-subscribe an unchanged slot (O6)
|
||||
entry['names'].sort(key=lambda n: (n['collection'], n['tag_name'], str(n['model_id'])))
|
||||
gathered.append({'server': servers[server_id], **entry})
|
||||
|
||||
return gathered, issues
|
||||
|
||||
|
||||
def build_tag_config(entries: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
"""
|
||||
Build one slot in the slot v2 format (contract C1) from address entries. Each server
|
||||
carries its security and the MinIO objects of its client certificate, which the
|
||||
gateway downloads (SIENTIAPDE-1974); certificate None means no security (D5).
|
||||
|
||||
Args:
|
||||
entries (list[dict[str, Any]]): Address entries from gather_read_tags for this slot
|
||||
|
||||
Returns:
|
||||
dict[str, Any]: The slot keyed by server key, each server with server_id, name,
|
||||
url, subscription_period_ms, security, certificate and tags. Each tag (keyed
|
||||
by address) has frequency, active_read and names. A connector without a
|
||||
subscription_period_ms gets half its lowest tag frequency in this slot.
|
||||
"""
|
||||
slot_config: dict[str, Any] = {}
|
||||
|
||||
for entry in entries:
|
||||
server = entry['server']
|
||||
if server['key'] not in slot_config:
|
||||
slot_config[server['key']] = {
|
||||
'server_id': server['server_id'],
|
||||
'name': server['name'],
|
||||
'url': server['url'],
|
||||
'subscription_period_ms': server['subscription_period_ms'],
|
||||
'security': server['security'],
|
||||
'certificate': server['certificate'],
|
||||
'tags': {},
|
||||
}
|
||||
|
||||
slot_config[server['key']]['tags'][entry['address']] = {
|
||||
'frequency': entry['frequency'],
|
||||
'active_read': entry['active_read'],
|
||||
'names': entry['names'],
|
||||
}
|
||||
|
||||
for server_config in slot_config.values():
|
||||
if server_config['subscription_period_ms'] is None:
|
||||
frequencies = [tag['frequency'] for tag in server_config['tags'].values()]
|
||||
server_config['subscription_period_ms'] = min(frequencies) / 2
|
||||
|
||||
return slot_config
|
||||
@@ -1,3 +1,5 @@
|
||||
"""Pipeline configuration builders: one Temporal schedule input per pipeline type."""
|
||||
|
||||
from math import ceil
|
||||
from typing import Any
|
||||
|
||||
@@ -264,8 +266,10 @@ def scouter(config: dict[str, Any]):
|
||||
Args:
|
||||
config (dict[str, Any]): Pipeline configuration containing:
|
||||
- schedule_name (str): Name of the schedule (used for topic generation)
|
||||
- read_tags (list[dict]): List of tag configurations with:
|
||||
- tag_name (str): Name of the tag to read
|
||||
- read_tags (list[dict]): List of tag configurations, already passed through
|
||||
resolve_tag_names (process_schedules does it), with:
|
||||
- tag_name (str): Name of the tag to read. A tag without one (a tag_id the
|
||||
tag database cannot resolve) is left out; process_slots notifies it
|
||||
- aggr_func (str, optional): Aggregation function for data collection (default: 'lts')
|
||||
Common values: 'lts' (last), 'avg' (average), 'min', 'max', 'sum'
|
||||
- data_range (list[int], optional): Valid data range [min, max] (default: [-100, 100])
|
||||
@@ -278,6 +282,8 @@ def scouter(config: dict[str, Any]):
|
||||
|
||||
tags = {}
|
||||
for tag in config['read_tags']:
|
||||
if not tag.get('tag_name'):
|
||||
continue
|
||||
tags[tag['tag_name']] = {
|
||||
'aggr_func': tag.get('aggr_func', 'lts'),
|
||||
'data_range': tag.get('data_range', [-100, 100]),
|
||||
@@ -415,11 +421,10 @@ def predictions_batch(config: dict[str, Any]):
|
||||
Args:
|
||||
config (dict[str, Any]): Pipeline configuration containing:
|
||||
- query (str): SQL query to retrieve input data for predictions
|
||||
- write_tags (list[dict]): List of OPC tag configurations for write-back with:
|
||||
- server_id (str): ID of the target OPC server
|
||||
- type (str): Tag type - 'prediction' (model output) or 'confidence' (prediction confidence)
|
||||
- addr (str): OPC tag address/path
|
||||
- data_type (str, optional): OPC data type (default: 'float')
|
||||
- opc_output_config (dict, optional): The OPC write-back configuration, built by
|
||||
process_schedules from the pipeline's write_tags ([{tag_id}]) through the tag
|
||||
database (resolve_write_tags): {connector_id: {prediction_tags |
|
||||
confidence_tags: {address: {data_type}}}} (default: {})
|
||||
- datetime_columns (list[str], optional): Column names to parse as datetime (default: [])
|
||||
- path_priority (list[str], optional): Filter policy execution order (default: ["STOP", "CONTINUE", "REPEAT"])
|
||||
- input_filters (list[dict], optional): Input data validation filters
|
||||
@@ -435,23 +440,6 @@ def predictions_batch(config: dict[str, Any]):
|
||||
dict[str, Any]: Complete predictions batch configuration with OPC output mappings,
|
||||
PI Web API output configuration, multi-stage filters, SQL query, and retention policies
|
||||
"""
|
||||
tags: dict[str, Any] = {}
|
||||
for tag in config.get('write_tags', []):
|
||||
if tag['server_id'] not in tags:
|
||||
tags[tag['server_id']] = {}
|
||||
|
||||
tag_type = tag['type']
|
||||
|
||||
if tag_type == 'prediction' or tag_type == 'confidence':
|
||||
tag_type_str = f'{tag_type}_tags'
|
||||
|
||||
if tag_type_str not in tags[tag['server_id']]:
|
||||
tags[tag['server_id']][tag_type_str] = {}
|
||||
|
||||
tags[tag['server_id']][tag_type_str][tag['addr']] = {
|
||||
'data_type': tag.get('data_type', 'float'),
|
||||
}
|
||||
|
||||
path_priority = process_path_priority(
|
||||
config.get('path_priority', ['STOP', 'CONTINUE', 'REPEAT'])
|
||||
)
|
||||
@@ -465,7 +453,7 @@ def predictions_batch(config: dict[str, Any]):
|
||||
'save_transform': config.get('save_transform', True),
|
||||
'transform_table_name': 'transformed_data',
|
||||
'retention_time': config.get('model_retention_minutes', 60) * 60,
|
||||
'opc_output_config': tags,
|
||||
'opc_output_config': config.get('opc_output_config', {}),
|
||||
'pi_web_api_output_config': config.get('pi_web_api_output_config', {}),
|
||||
'input_filters': overlap_filter_config(
|
||||
{'EMPTY_DATA': {'policy': 'STOP', 'config': {}}}, config.get('input_filters', [])
|
||||
@@ -484,112 +472,3 @@ def predictions_batch(config: dict[str, Any]):
|
||||
'path_priority': path_priority,
|
||||
'predictions_storage_policy': config.get('predictions_storage_policy', 'lts:1'),
|
||||
}
|
||||
|
||||
|
||||
def gather_read_tags(pipelines: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
"""
|
||||
Gather all read tags from scouter pipeline configurations.
|
||||
|
||||
Collects all read tags from scouter-type pipelines and organizes them by
|
||||
server_id and tag_address, tracking which Kafka topics each tag is associated with.
|
||||
This function is used during slot configuration to aggregate tags across multiple
|
||||
scouter pipelines for efficient OPC server slot allocation.
|
||||
|
||||
Args:
|
||||
pipelines (list[dict[str, Any]]): List of pipeline configurations to process.
|
||||
Only pipelines with workflow_type 'scouter' are processed. Each scouter
|
||||
pipeline should contain a 'read_tags' list with tag configurations.
|
||||
|
||||
Returns:
|
||||
dict[str, Any]: Dictionary of read tags keyed by "server_id:tag_address",
|
||||
where each entry contains:
|
||||
- All original tag configuration fields
|
||||
- topics (list[str]): List of Kafka topic names associated with this tag
|
||||
(format: 'raw_{schedule_name}')
|
||||
"""
|
||||
|
||||
tags = {}
|
||||
|
||||
# Get all read tags from pipelines
|
||||
for pipeline in pipelines:
|
||||
if pipeline['workflow_type'] != 'scouter':
|
||||
continue
|
||||
|
||||
for tag in pipeline.get('read_tags', []):
|
||||
tag_string = f'{tag["server_id"]}:{tag["tag_address"]}'
|
||||
if tag_string not in tags:
|
||||
tags[tag_string] = {**tag, 'topics': []}
|
||||
|
||||
tags[tag_string]['topics'].append(f'raw_{pipeline["schedule_name"]}')
|
||||
|
||||
return tags
|
||||
|
||||
|
||||
def build_tag_config(
|
||||
tags: list[dict[str, Any]], opc_servers: dict[str, Any]
|
||||
) -> tuple[dict[str, Any], list]:
|
||||
"""
|
||||
Build tag configuration for a specific slot and OPC server.
|
||||
|
||||
Organizes tags by OPC server name and calculates the minimum subscription period
|
||||
based on tag frequencies. Validates that all server IDs exist in the OPC
|
||||
servers configuration. The subscription period is set to half of the minimum
|
||||
tag frequency to ensure efficient data collection.
|
||||
|
||||
Args:
|
||||
tags (list[dict[str, Any]]): List of tag configurations containing:
|
||||
- server_id (str): ID of the OPC server
|
||||
- tag_address (str): Address/path of the OPC tag
|
||||
- frequency (int): Tag read frequency in milliseconds
|
||||
- Additional tag-specific configuration fields
|
||||
opc_servers (dict[str, Any]): Dictionary of OPC server configurations keyed by server_id.
|
||||
Each server configuration should contain:
|
||||
- server_name (str): Human-readable server name
|
||||
- url (str): OPC server URL
|
||||
- uri (str): OPC server URI
|
||||
- cert_path (str, optional): Certificate file path
|
||||
- private_key_path (str, optional): Private key file path
|
||||
- server_cert_path (str, optional): Server certificate file path
|
||||
|
||||
Returns:
|
||||
tuple[dict[str, Any], list]: A tuple containing:
|
||||
- Slot configuration dictionary organized by server_name, where each server
|
||||
contains connection details, tags dictionary, and subscription_period_ms
|
||||
- List of server IDs (str) that were not found in opc_servers configuration
|
||||
"""
|
||||
|
||||
slot_config = {}
|
||||
notifications = []
|
||||
|
||||
for tag in tags:
|
||||
server_id = tag['server_id']
|
||||
|
||||
if server_id not in opc_servers:
|
||||
notifications.append(server_id)
|
||||
continue
|
||||
|
||||
server_name = opc_servers[server_id]['server_name']
|
||||
if server_name not in slot_config:
|
||||
slot_config[server_name] = {
|
||||
'server_id': server_id,
|
||||
'name': server_name,
|
||||
'url': opc_servers[server_id]['url'],
|
||||
'server_uri': opc_servers[server_id]['uri'],
|
||||
'cert_path': opc_servers[server_id].get('cert_path', None),
|
||||
'private_key_path': opc_servers[server_id].get('private_key_path', None),
|
||||
'server_cert_path': opc_servers[server_id].get('server_cert_path', None),
|
||||
'tags': {},
|
||||
}
|
||||
|
||||
slot_config[server_name]['tags'][tag['tag_address']] = {
|
||||
**tag,
|
||||
}
|
||||
|
||||
for server_name in slot_config:
|
||||
frequencies = [int(x['frequency']) for x in slot_config[server_name]['tags'].values()]
|
||||
|
||||
min_frequency = min(frequencies) if frequencies else 1000
|
||||
|
||||
slot_config[server_name]['subscription_period_ms'] = min_frequency / 2
|
||||
|
||||
return slot_config, notifications
|
||||
310
orchestrator/utils/tag_database.py
Normal file
310
orchestrator/utils/tag_database.py
Normal file
@@ -0,0 +1,310 @@
|
||||
"""
|
||||
Tag database (connectors + connector_tags): which tags the pipelines reference, the
|
||||
indexes of what was loaded, and the resolution of the OPC UA read and write tags.
|
||||
"""
|
||||
|
||||
from typing import Any
|
||||
|
||||
# Fields a tag database document needs to take part in a slot or a write config.
|
||||
# Documents missing any of them are left out of the index, so their tags are notified
|
||||
# as not found instead of failing the whole run.
|
||||
CONNECTOR_TAG_FIELDS = ('tag_id', 'connector_id', 'address', 'tag_name')
|
||||
|
||||
|
||||
CONNECTOR_FIELDS = ('connector_id', 'name', 'url')
|
||||
|
||||
|
||||
# security fields of an OPC UA connector: the source of its OPC security (D4)
|
||||
SECURITY_FIELDS = ('mode', 'policy')
|
||||
|
||||
|
||||
# Fields of a client certificate the slot carries, and of each object it names in MinIO
|
||||
CERTIFICATE_FIELDS = ('certificate_id', 'application_uri')
|
||||
CERTIFICATE_OBJECTS = ('cert', 'private_key')
|
||||
STORED_OBJECT_FIELDS = ('bucket', 'object_name')
|
||||
|
||||
|
||||
# connector_tags categories each use accepts
|
||||
READ_CATEGORIES = ('Read',)
|
||||
WRITE_CATEGORIES = ('Write', 'Prediction Confidence')
|
||||
|
||||
|
||||
# opc_output_config block each write category goes to
|
||||
WRITE_TAG_GROUPS = {'Write': 'prediction_tags', 'Prediction Confidence': 'confidence_tags'}
|
||||
|
||||
|
||||
# data_type values the OPC Gateway write engine accepts (contract C2)
|
||||
OPC_DATA_TYPES = ('float', 'double', 'real', 'int', 'integer', 'bool', 'boolean', 'str', 'string')
|
||||
DEFAULT_DATA_TYPE = 'float'
|
||||
|
||||
|
||||
# Workflow types whose write_tags reference the tag database
|
||||
WRITE_WORKFLOW_TYPES = ('predictions_batch', 'xgboost_predictions_batch')
|
||||
|
||||
|
||||
def is_complete(document: dict[str, Any], fields: tuple[str, ...]) -> bool:
|
||||
return all(document.get(field) not in (None, '') for field in fields)
|
||||
|
||||
|
||||
def tag_id_of(tag: Any) -> int | None:
|
||||
"""
|
||||
The tag's tag_id, or None when it is not an integer.
|
||||
|
||||
The connector_tags validator stores tag_id as int, so a pipeline references a tag
|
||||
by the same int: text ('7'), bool, null or a missing tag_id is no reference.
|
||||
"""
|
||||
tag_id = tag.get('tag_id') if isinstance(tag, dict) else None
|
||||
return tag_id if isinstance(tag_id, int) and not isinstance(tag_id, bool) else None
|
||||
|
||||
|
||||
def _is_active(document: dict[str, Any]) -> bool:
|
||||
"""A document without status (an override that does not project it) counts as active."""
|
||||
return document.get('status') in (None, 'Active')
|
||||
|
||||
|
||||
def pipeline_tag_refs(pipeline: dict[str, Any]) -> list[Any]:
|
||||
"""
|
||||
The tag references of a pipeline: the read_tags of a scouter and the write_tags of a
|
||||
predictions_batch. Entries are returned as they are (they may lack a tag_id).
|
||||
"""
|
||||
workflow_type = pipeline.get('workflow_type')
|
||||
if workflow_type == 'scouter':
|
||||
return list(pipeline.get('read_tags') or [])
|
||||
if workflow_type in WRITE_WORKFLOW_TYPES:
|
||||
return list(pipeline.get('write_tags') or [])
|
||||
return []
|
||||
|
||||
|
||||
def referenced_tag_ids(pipelines: list[dict[str, Any]]) -> list[int]:
|
||||
"""
|
||||
The tag_ids the pipelines reference (scouter read_tags and predictions_batch
|
||||
write_tags), for the connector_tags query.
|
||||
|
||||
Sorted, so the query is the same on every run. Runs in the workflow, so it never
|
||||
raises on malformed references.
|
||||
|
||||
Args:
|
||||
pipelines (list[dict[str, Any]]): Pipeline configurations
|
||||
|
||||
Returns:
|
||||
list[int]: The referenced tag_ids (empty when no tag carries a tag_id)
|
||||
"""
|
||||
ids = {tag_id_of(tag) for pipeline in pipelines for tag in pipeline_tag_refs(pipeline)}
|
||||
return sorted(tag_id for tag_id in ids if tag_id is not None)
|
||||
|
||||
|
||||
def filter_by_tag_ids(query: dict[str, Any], tag_ids: list[int]) -> dict[str, Any]:
|
||||
"""
|
||||
Restrict a connector_tags aggregation to the referenced tag_ids.
|
||||
|
||||
Args:
|
||||
query (dict[str, Any]): The connector_tags query ({'collection', 'aggregation'})
|
||||
tag_ids (list[int]): The ids from referenced_tag_ids
|
||||
|
||||
Returns:
|
||||
dict[str, Any]: A new query whose aggregation starts with a tag_id $match. The
|
||||
aggregation list is new too: aggregate_documents_in_mongodb appends a stage
|
||||
to the list it gets.
|
||||
"""
|
||||
return {
|
||||
**query,
|
||||
'aggregation': [{'$match': {'tag_id': {'$in': tag_ids}}}, *query['aggregation']],
|
||||
}
|
||||
|
||||
|
||||
def index_tag_documents(connector_tags: list[dict[str, Any]]) -> dict[int, dict[str, Any]]:
|
||||
"""
|
||||
Every loaded connector_tags document keyed by tag_id, whatever its status, category
|
||||
or completeness.
|
||||
"""
|
||||
indexed = {}
|
||||
for tag in connector_tags:
|
||||
tag_id = tag_id_of(tag)
|
||||
if tag_id is not None:
|
||||
indexed[tag_id] = tag
|
||||
return indexed
|
||||
|
||||
|
||||
def index_connector_tags(
|
||||
connector_tags: list[dict[str, Any]], categories: tuple[str, ...] = READ_CATEGORIES
|
||||
) -> dict[int, dict[str, Any]]:
|
||||
"""
|
||||
Index the tag database's usable connector_tags by tag_id.
|
||||
|
||||
Documents that are not active, have another category, or lack an int tag_id,
|
||||
connector_id, address or tag_name are left out.
|
||||
|
||||
Args:
|
||||
connector_tags (list[dict[str, Any]]): connector_tags documents (SIENTIAPDE-1569)
|
||||
categories (tuple[str, ...]): Accepted categories (READ_CATEGORIES for the slot
|
||||
and the scouter names, WRITE_CATEGORIES for the write tags). A document
|
||||
without category is accepted.
|
||||
|
||||
Returns:
|
||||
dict[int, dict[str, Any]]: connector_tags documents keyed by tag_id
|
||||
"""
|
||||
return {
|
||||
tag_id: tag
|
||||
for tag_id, tag in index_tag_documents(connector_tags).items()
|
||||
if is_complete(tag, CONNECTOR_TAG_FIELDS)
|
||||
and _is_active(tag)
|
||||
and tag.get('category') in (None, *categories)
|
||||
}
|
||||
|
||||
|
||||
def index_connectors(connectors: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
|
||||
"""
|
||||
Index the tag database's connectors by connector_id (as text).
|
||||
|
||||
Documents that are not active or lack connector_id, name, url, security.mode or
|
||||
security.policy are left out.
|
||||
|
||||
Args:
|
||||
connectors (list[dict[str, Any]]): connectors documents (SIENTIAPDE-1569)
|
||||
|
||||
Returns:
|
||||
dict[str, dict[str, Any]]: connectors documents keyed by str(connector_id)
|
||||
"""
|
||||
return {
|
||||
str(connector['connector_id']): connector
|
||||
for connector in connectors
|
||||
if is_complete(connector, CONNECTOR_FIELDS)
|
||||
and _has_security(connector)
|
||||
and _is_active(connector)
|
||||
}
|
||||
|
||||
|
||||
def _has_security(connector: dict[str, Any]) -> bool:
|
||||
security = connector.get('security')
|
||||
return isinstance(security, dict) and is_complete(security, SECURITY_FIELDS)
|
||||
|
||||
|
||||
def client_certificate_id(connector: dict[str, Any]) -> Any:
|
||||
"""The connector's security.client_certificate_id (None when it has none)."""
|
||||
security = connector.get('security')
|
||||
return security.get('client_certificate_id') if isinstance(security, dict) else None
|
||||
|
||||
|
||||
def _is_usable_certificate(certificate: Any) -> bool:
|
||||
return (
|
||||
isinstance(certificate, dict)
|
||||
and is_complete(certificate, CERTIFICATE_FIELDS)
|
||||
and all(
|
||||
isinstance(certificate.get(name), dict)
|
||||
and is_complete(certificate[name], STORED_OBJECT_FIELDS)
|
||||
for name in CERTIFICATE_OBJECTS
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def resolve_certificate(connector: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""
|
||||
The client certificate a connector goes into the slot with, or None (D5).
|
||||
|
||||
connectors_query's $lookup puts in `certificates` the opc_certificates document the
|
||||
connector's security.client_certificate_id points to, only when it is active and of
|
||||
kind client. None means no certificate: the gateway connects without security. A
|
||||
connector with a client_certificate_id and no certificate is most likely a
|
||||
registration error, reported by the caller (see client_certificate_id).
|
||||
|
||||
Args:
|
||||
connector (dict[str, Any]): A connectors document from connectors_query
|
||||
|
||||
Returns:
|
||||
dict[str, Any] | None: certificate_id, application_uri, and cert and private_key
|
||||
as {bucket, object_name}; None without a complete certificate
|
||||
"""
|
||||
for certificate in connector.get('certificates') or []:
|
||||
if _is_usable_certificate(certificate):
|
||||
return {
|
||||
**{field: certificate[field] for field in CERTIFICATE_FIELDS},
|
||||
**{
|
||||
name: {field: certificate[name][field] for field in STORED_OBJECT_FIELDS}
|
||||
for name in CERTIFICATE_OBJECTS
|
||||
},
|
||||
}
|
||||
return None
|
||||
|
||||
|
||||
def resolve_tag_names(
|
||||
read_tags: list[dict[str, Any]], tags_by_id: dict[int, dict[str, Any]]
|
||||
) -> list[dict[str, Any]]:
|
||||
"""
|
||||
Fill in the tag_name of each read tag of a scouter pipeline.
|
||||
|
||||
The name is the pipeline's own tag_name when it has one, else the connector_tags
|
||||
name of its tag_id. The scouter's model_tags (scouter()) and the slot's names[]
|
||||
(gather_read_tags()) both take the name from here, so the name the gateway writes
|
||||
is always the one the scouter looks for (SIENTIAPDE-1974, SC1).
|
||||
|
||||
Args:
|
||||
read_tags (list[dict[str, Any]]): The pipeline's read_tags
|
||||
tags_by_id (dict[int, dict[str, Any]]): connector_tags indexed by index_connector_tags
|
||||
|
||||
Returns:
|
||||
list[dict[str, Any]]: The read tags, each with tag_name set when it could be
|
||||
resolved. A tag_id missing from tags_by_id leaves the tag without a name.
|
||||
"""
|
||||
resolved = []
|
||||
for tag in read_tags:
|
||||
name = tag.get('tag_name')
|
||||
tag_id = tag_id_of(tag)
|
||||
if not name and tag_id is not None:
|
||||
name = tags_by_id.get(tag_id, {}).get('tag_name')
|
||||
resolved.append({**tag, 'tag_name': name} if name else tag)
|
||||
return resolved
|
||||
|
||||
|
||||
def resolve_write_tags(
|
||||
write_tags: list[Any],
|
||||
tags_by_id: dict[int, dict[str, Any]],
|
||||
connectors: dict[str, dict[str, Any]],
|
||||
) -> tuple[dict[str, Any], dict[str, set[str]]]:
|
||||
"""
|
||||
Build the opc_output_config of a predictions_batch from its write_tags.
|
||||
|
||||
Each write tag is only a tag_id: the tag database gives the rest. The category says
|
||||
where the value goes ('Write' → prediction_tags, 'Prediction Confidence' →
|
||||
confidence_tags), the connector_id is the server the OPC Gateway writes to (it
|
||||
resolves it in connectors), and the address and data_type come from connector_tags
|
||||
(data_type defaults to 'float').
|
||||
|
||||
Args:
|
||||
write_tags (list[Any]): The pipeline's write_tags ([{tag_id}])
|
||||
tags_by_id (dict[int, dict[str, Any]]): Write connector_tags indexed by
|
||||
index_connector_tags(..., WRITE_CATEGORIES)
|
||||
connectors (dict[str, dict[str, Any]]): Active OPC UA connectors indexed by
|
||||
index_connectors
|
||||
|
||||
Returns:
|
||||
tuple: the opc_output_config ({connector_id: {prediction_tags | confidence_tags:
|
||||
{address: {data_type}}}}) and what was left out, by kind: 'tags' (tag_id not
|
||||
an active, complete Write / Prediction Confidence tag), 'connectors'
|
||||
(connector_id) and 'invalid' (an entry without an int tag_id, or a tag whose
|
||||
data_type the gateway does not accept)
|
||||
"""
|
||||
issues: dict[str, set[str]] = {'tags': set(), 'connectors': set(), 'invalid': set()}
|
||||
output_config: dict[str, Any] = {}
|
||||
for entry in write_tags:
|
||||
tag_id = tag_id_of(entry)
|
||||
if tag_id is None:
|
||||
issues['invalid'].add(str(entry))
|
||||
continue
|
||||
connector_tag = tags_by_id.get(tag_id)
|
||||
# The category is what tells a prediction from a confidence: it cannot be missing
|
||||
if connector_tag is None or connector_tag.get('category') not in WRITE_TAG_GROUPS:
|
||||
issues['tags'].add(str(tag_id))
|
||||
continue
|
||||
connector_id = str(connector_tag['connector_id'])
|
||||
if connector_id not in connectors:
|
||||
issues['connectors'].add(connector_id)
|
||||
continue
|
||||
data_type = connector_tag.get('data_type') or DEFAULT_DATA_TYPE
|
||||
if data_type not in OPC_DATA_TYPES:
|
||||
issues['invalid'].add(str(tag_id))
|
||||
continue
|
||||
group = WRITE_TAG_GROUPS[connector_tag['category']]
|
||||
output_config.setdefault(connector_id, {}).setdefault(group, {})[
|
||||
connector_tag['address']
|
||||
] = {'data_type': data_type}
|
||||
return output_config, issues
|
||||
@@ -5,6 +5,7 @@ with workflow.unsafe.imports_passed_through():
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
from prometheus_client import start_http_server
|
||||
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
|
||||
@@ -192,13 +193,12 @@ async def main():
|
||||
try:
|
||||
await asyncio.gather(*handlers)
|
||||
except BaseException as e:
|
||||
logger.error(f'An unhandled exception occurred: {e}', exc_info=True)
|
||||
# sientia_do's Logger.error takes only a message: the traceback goes in it
|
||||
logger.error(f'An unhandled exception occurred: {e}\n{traceback.format_exc()}')
|
||||
exit_code = 1
|
||||
finally:
|
||||
if notification_handler:
|
||||
notification_handler.shutdown()
|
||||
if activities:
|
||||
activities.shutdown()
|
||||
notification_handler.shutdown()
|
||||
activities.shutdown()
|
||||
metrics.APP_UP.labels(pod_id=POD_ID).set(0)
|
||||
sys.exit(exit_code)
|
||||
|
||||
|
||||
@@ -7,6 +7,90 @@ with workflow.unsafe.imports_passed_through():
|
||||
from sientia_do.temporal.policies import retry_policy
|
||||
|
||||
from orchestrator.activities.activities import Activities
|
||||
from orchestrator.utils.tag_database import (
|
||||
filter_by_tag_ids,
|
||||
referenced_tag_ids,
|
||||
)
|
||||
|
||||
# MongoDB aggregation stages
|
||||
MATCH = '$match'
|
||||
PROJECT = '$project'
|
||||
LOOKUP = '$lookup'
|
||||
|
||||
# Tag database (SIENTIAPDE-1569) read by the slot v2 and the schedules (SIENTIAPDE-1974).
|
||||
# The workflow input may override these aggregations (connectors_query /
|
||||
# connector_tags_query); the workflow puts a tag_id $match in front of
|
||||
# connector_tags_query. Every category and status of the referenced tags is loaded: the
|
||||
# slot uses the active Read tags, predictions_batch the active Write / Prediction
|
||||
# Confidence ones (status and category are filtered in the code). Only the fields used
|
||||
# are projected: other attributes may hold BSON types a Temporal payload cannot carry
|
||||
# (so the certificate dates are left out).
|
||||
# Each connector brings, in `certificates`, the opc_certificates document its
|
||||
# security.client_certificate_id points to, only when it is active and of kind client
|
||||
# (D5); resolve_certificate turns it into the slot's certificate.
|
||||
DEFAULT_CONNECTORS_QUERY = {
|
||||
'collection': 'connectors',
|
||||
'aggregation': [
|
||||
{MATCH: {'type': 'opc_ua', 'status': 'Active'}},
|
||||
{
|
||||
LOOKUP: {
|
||||
'from': 'opc_certificates',
|
||||
'let': {'certificate_id': '$security.client_certificate_id'},
|
||||
'pipeline': [
|
||||
{
|
||||
MATCH: {
|
||||
'$expr': {'$eq': ['$certificate_id', '$$certificate_id']},
|
||||
'status': 'active',
|
||||
'kind': 'client',
|
||||
}
|
||||
},
|
||||
{
|
||||
PROJECT: {
|
||||
'_id': 0,
|
||||
'certificate_id': 1,
|
||||
'application_uri': 1,
|
||||
'cert.bucket': 1,
|
||||
'cert.object_name': 1,
|
||||
'private_key.bucket': 1,
|
||||
'private_key.object_name': 1,
|
||||
}
|
||||
},
|
||||
],
|
||||
'as': 'certificates',
|
||||
}
|
||||
},
|
||||
{
|
||||
PROJECT: {
|
||||
'_id': 0,
|
||||
'connector_id': 1,
|
||||
'name': 1,
|
||||
'url': 1,
|
||||
'subscription_period_ms': 1,
|
||||
'security.mode': 1,
|
||||
'security.policy': 1,
|
||||
'security.client_certificate_id': 1,
|
||||
'certificates': 1,
|
||||
}
|
||||
},
|
||||
],
|
||||
}
|
||||
DEFAULT_CONNECTOR_TAGS_QUERY = {
|
||||
'collection': 'connector_tags',
|
||||
'aggregation': [
|
||||
{
|
||||
PROJECT: {
|
||||
'_id': 0,
|
||||
'tag_id': 1,
|
||||
'connector_id': 1,
|
||||
'address': 1,
|
||||
'tag_name': 1,
|
||||
'category': 1,
|
||||
'data_type': 1,
|
||||
'status': 1,
|
||||
}
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@workflow.defn(name='orchestrator')
|
||||
@@ -20,14 +104,79 @@ class Orchestrator:
|
||||
and Redis infrastructure.
|
||||
"""
|
||||
|
||||
async def _load_tag_database(
|
||||
self,
|
||||
input_data: dict[str, Any],
|
||||
metadata: dict[str, Any],
|
||||
pipelines: list[dict[str, Any]],
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
"""
|
||||
Load the tag database (SIENTIAPDE-1974): the only source of the OPC servers and
|
||||
tags of the read and write tags.
|
||||
|
||||
Only the referenced tags are loaded, with the fields used, so the payload stays
|
||||
bounded by the pipelines. Nothing is loaded when no scouter read tag nor
|
||||
predictions_batch write tag has a tag_id.
|
||||
|
||||
Args:
|
||||
input_data (dict[str, Any]): Workflow input (optional connectors_query and
|
||||
connector_tags_query)
|
||||
metadata (dict[str, Any]): Activity metadata ({'metadata': {...}})
|
||||
pipelines (list[dict[str, Any]]): The loaded pipeline configurations
|
||||
|
||||
Returns:
|
||||
tuple[list[dict[str, Any]], list[dict[str, Any]]]: connectors and connector_tags
|
||||
"""
|
||||
tag_ids = referenced_tag_ids(pipelines)
|
||||
if not tag_ids:
|
||||
return [], []
|
||||
|
||||
return await self._load_connectors(
|
||||
metadata,
|
||||
input_data.get('connectors_query', DEFAULT_CONNECTORS_QUERY),
|
||||
input_data.get('connector_tags_query', DEFAULT_CONNECTOR_TAGS_QUERY),
|
||||
tag_ids,
|
||||
)
|
||||
|
||||
async def _load_connectors(
|
||||
self,
|
||||
metadata: dict[str, Any],
|
||||
connectors_query: dict[str, Any],
|
||||
connector_tags_query: dict[str, Any],
|
||||
tag_ids: list[Any],
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
"""Run a connectors query and a connector_tags query restricted to tag_ids, in parallel."""
|
||||
connectors_handler = workflow.start_local_activity_method(
|
||||
Activities.aggregate_documents_in_mongodb,
|
||||
{
|
||||
**metadata,
|
||||
# A fresh aggregation list: the activity appends a stage to the one it gets
|
||||
'query': {**connectors_query, 'aggregation': list(connectors_query['aggregation'])},
|
||||
},
|
||||
retry_policy=retry_policy,
|
||||
start_to_close_timeout=timedelta(seconds=60),
|
||||
)
|
||||
|
||||
connector_tags_handler = workflow.start_local_activity_method(
|
||||
Activities.aggregate_documents_in_mongodb,
|
||||
{
|
||||
**metadata,
|
||||
'query': filter_by_tag_ids(connector_tags_query, tag_ids),
|
||||
},
|
||||
retry_policy=retry_policy,
|
||||
start_to_close_timeout=timedelta(seconds=60),
|
||||
)
|
||||
|
||||
return await connectors_handler, await connector_tags_handler
|
||||
|
||||
@workflow.run
|
||||
async def run(self, input_data: dict[str, Any]):
|
||||
"""
|
||||
Execute the orchestration workflow for pipeline and slot management.
|
||||
|
||||
This workflow retrieves pipeline configurations and OPC server data,
|
||||
processes schedules and slot configurations, and deploys them to
|
||||
the appropriate services. It handles creation, updates, and deletion
|
||||
This workflow retrieves pipeline configurations and the tag database
|
||||
(connectors and connector_tags), processes schedules and slot configurations,
|
||||
and deploys them to the appropriate services. It handles creation, updates, and deletion
|
||||
of schedules and slots based on current system state.
|
||||
|
||||
Args:
|
||||
@@ -35,7 +184,11 @@ class Orchestrator:
|
||||
Required fields:
|
||||
- schedule_name (str): Name of the orchestration schedule
|
||||
- pipelines_query (dict[str, Any]): MongoDB query for pipeline configurations
|
||||
- opc_servers_query (dict[str, Any]): MongoDB query for OPC server data
|
||||
Optional fields:
|
||||
- connectors_query (dict[str, Any]): MongoDB aggregation for the tag
|
||||
database connectors (default: DEFAULT_CONNECTORS_QUERY)
|
||||
- connector_tags_query (dict[str, Any]): MongoDB aggregation for the tag
|
||||
database tags (default: DEFAULT_CONNECTOR_TAGS_QUERY)
|
||||
|
||||
Returns:
|
||||
None: Workflow completes without return value
|
||||
@@ -66,13 +219,6 @@ class Orchestrator:
|
||||
start_to_close_timeout=timedelta(seconds=60),
|
||||
)
|
||||
|
||||
opc_servers_handler = workflow.start_local_activity_method(
|
||||
Activities.find_documents_in_mongodb,
|
||||
{**metadata, 'query': input_data['opc_servers_query']},
|
||||
retry_policy=retry_policy,
|
||||
start_to_close_timeout=timedelta(seconds=60),
|
||||
)
|
||||
|
||||
orchestrated_schedules_handler = workflow.start_local_activity_method(
|
||||
Activities.find_documents_in_mongodb,
|
||||
{
|
||||
@@ -103,9 +249,13 @@ class Orchestrator:
|
||||
)
|
||||
|
||||
pipeline_config = await pipeline_config_handler
|
||||
|
||||
connectors, connector_tags = await self._load_tag_database(
|
||||
input_data, metadata, pipeline_config
|
||||
)
|
||||
|
||||
orchestrated_schedules = await orchestrated_schedules_handler
|
||||
current_slot_config = await current_slot_config_handler
|
||||
opc_servers = await opc_servers_handler
|
||||
active_ingestors = await active_ingestors_handler
|
||||
|
||||
formatted_orchestrated_schedules_handler = workflow.start_local_activity_method(
|
||||
@@ -117,7 +267,12 @@ class Orchestrator:
|
||||
|
||||
schedules_config_handler = workflow.start_local_activity_method(
|
||||
Activities.process_schedules,
|
||||
{**metadata, 'pipelines': pipeline_config},
|
||||
{
|
||||
**metadata,
|
||||
'pipelines': pipeline_config,
|
||||
'connectors': connectors,
|
||||
'connector_tags': connector_tags,
|
||||
},
|
||||
retry_policy=retry_policy,
|
||||
start_to_close_timeout=timedelta(seconds=60),
|
||||
)
|
||||
@@ -126,7 +281,8 @@ class Orchestrator:
|
||||
Activities.process_slots,
|
||||
{
|
||||
**metadata,
|
||||
'opc_servers': opc_servers,
|
||||
'connectors': connectors,
|
||||
'connector_tags': connector_tags,
|
||||
'active_ingestors': active_ingestors,
|
||||
'pipelines': pipeline_config,
|
||||
},
|
||||
|
||||
@@ -16,25 +16,22 @@
|
||||
],
|
||||
"read_tags": [
|
||||
{
|
||||
"server_id": "1",
|
||||
"tag_id": 1,
|
||||
"tag_name": "Counter",
|
||||
"tag_address": "ns=2;s=GatewayProSYS.ProSYS.Simulation.Counter",
|
||||
"aggr_func": "lts",
|
||||
"data_range": [1, 30],
|
||||
"frequency": 30000
|
||||
},
|
||||
{
|
||||
"server_id": "1",
|
||||
"tag_id": 2,
|
||||
"tag_name": "Square",
|
||||
"tag_address": "ns=2;s=GatewayProSYS.ProSYS.Simulation.Square",
|
||||
"aggr_func": "lts",
|
||||
"data_range": [-2, 2],
|
||||
"frequency": 30000
|
||||
},
|
||||
{
|
||||
"server_id": "1",
|
||||
"tag_id": 3,
|
||||
"tag_name": "Rollout",
|
||||
"tag_address": "ns=2;s=GatewayProSYS.ProSYS.Simulation.Rollout",
|
||||
"aggr_func": "lts",
|
||||
"data_range": [-2, 2],
|
||||
"frequency": 30000
|
||||
|
||||
@@ -85,7 +85,9 @@ def test_close(sientia_monitoring_mock, email):
|
||||
email.close()
|
||||
|
||||
email.server.quit.assert_called_once()
|
||||
sientia_monitoring_mock.shutdown.assert_called_once()
|
||||
# Count only this instance: an Email left by another test may be garbage-collected
|
||||
# (Email.__del__ -> close) while the patch is active
|
||||
assert sientia_monitoring_mock.shutdown.call_args_list.count(call(email)) == 1
|
||||
|
||||
|
||||
def test_build_email_html(email):
|
||||
@@ -324,3 +326,37 @@ def test_send_email(mimemultipart, mimetext, email):
|
||||
)
|
||||
|
||||
assert email.try_send_email.call_count == 2
|
||||
|
||||
|
||||
@patch('orchestrator.activities.email.EmailBuilder')
|
||||
@patch('orchestrator.activities.email.smtplib')
|
||||
def test___init___without_smtp_server(smtplib, email_builder):
|
||||
email = Email(
|
||||
sender_email='test@test.com',
|
||||
sender_password='test',
|
||||
smtp_server=None,
|
||||
smtp_port=587,
|
||||
logger=MagicMock(),
|
||||
notification_handler=MagicMock(),
|
||||
metrics_controller=AsyncMock(),
|
||||
)
|
||||
|
||||
smtplib.SMTP.assert_not_called()
|
||||
assert not hasattr(email, 'server')
|
||||
|
||||
|
||||
@patch('orchestrator.activities.email.smtplib.SMTP')
|
||||
def test_try_send_email_reconnect_without_password(smtp, email):
|
||||
email.sender_password = None
|
||||
email.server.sendmail = MagicMock(side_effect=SMTPServerDisconnected('test'))
|
||||
email.server.quit = MagicMock()
|
||||
|
||||
msg = MagicMock()
|
||||
|
||||
email.try_send_email(msg, 'test')
|
||||
|
||||
smtp.return_value.starttls.assert_not_called()
|
||||
smtp.return_value.login.assert_not_called()
|
||||
smtp.return_value.sendmail.assert_called_once_with(
|
||||
'test@test.com', 'test', msg.as_string.return_value
|
||||
)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,14 +1,14 @@
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import ANY, AsyncMock, MagicMock, patch
|
||||
from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
|
||||
|
||||
from pytest import fixture, mark
|
||||
import pytest
|
||||
from sientia_do.notifications.models import NotificationLevel
|
||||
from sientia_do.temporal.constants import DATETIME_FORMAT_MS_WITH_TZ
|
||||
|
||||
from orchestrator.activities.mongo_db import MongoDB
|
||||
|
||||
|
||||
@fixture
|
||||
@pytest.fixture
|
||||
@patch('orchestrator.activities.mongo_db.MongoDBRepository')
|
||||
def mongo_db(mongo_mock):
|
||||
mongo = MongoDB(
|
||||
@@ -192,6 +192,30 @@ def test_aggregate_documents_in_mongodb_success(mongo_db):
|
||||
)
|
||||
|
||||
|
||||
def test_aggregate_documents_in_mongodb_formats_only_datetimes(mongo_db):
|
||||
"""A timestamp field stored as text, or missing, is left as it is"""
|
||||
mongo_db.mongo_db_repository.aggregate = MagicMock(
|
||||
return_value=[
|
||||
{'tag_id': 1, 'updated_at': datetime(2026, 10, 1, 12, 0, 0)},
|
||||
{'tag_id': 2, 'updated_at': '2026-10-02 00:00:00.000+0000'},
|
||||
{'tag_id': 3},
|
||||
]
|
||||
)
|
||||
|
||||
result = mongo_db.aggregate_documents_in_mongodb(
|
||||
{
|
||||
'query': {'collection': 'connector_tags', 'aggregation': [{'$match': {}}]},
|
||||
'timestamp_fields': ['updated_at'],
|
||||
}
|
||||
)
|
||||
|
||||
assert result == [
|
||||
{'tag_id': 1, 'updated_at': '2026-10-01 12:00:00.000000+0000'},
|
||||
{'tag_id': 2, 'updated_at': '2026-10-02 00:00:00.000+0000'},
|
||||
{'tag_id': 3},
|
||||
]
|
||||
|
||||
|
||||
def test_aggregate_documents_in_mongodb_failure(mongo_db):
|
||||
input_data = {
|
||||
'collection': 'test_collection',
|
||||
@@ -246,56 +270,59 @@ def test_aggregate_documents_in_mongodb_missing_aggregation(mongo_db):
|
||||
def test_update_pipelines_timestamps_success(now_mock, mongo_db):
|
||||
input_data = {
|
||||
'updated_pipelines': [
|
||||
{'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
|
||||
{'schedule_name': 'test1', 'namespace': 'test1', 'success': True, 'config_hash': 'h1'},
|
||||
# Updated by a run started before config_hash
|
||||
{'schedule_name': 'test2', 'namespace': 'test2', 'success': True},
|
||||
]
|
||||
}
|
||||
mongo_db.mongo_db_repository.update_many = MagicMock(return_value=MagicMock())
|
||||
mongo_db.mongo_db_repository.update_one = MagicMock(return_value=1)
|
||||
mongo_db.update_pipelines_timestamps(input_data)
|
||||
mongo_db.mongo_db_repository.update_many.assert_called_once_with(
|
||||
'orchestrated_schedules',
|
||||
{
|
||||
'$or': [
|
||||
{'schedule_name': 'test1', 'namespace': 'test1'},
|
||||
{'schedule_name': 'test2', 'namespace': 'test2'},
|
||||
]
|
||||
},
|
||||
{'$set': {'updated_at': now_mock.return_value}},
|
||||
{},
|
||||
)
|
||||
assert mongo_db.mongo_db_repository.update_one.call_args_list == [
|
||||
call(
|
||||
'orchestrated_schedules',
|
||||
{'schedule_name': 'test1', 'namespace': 'test1'},
|
||||
{'$set': {'updated_at': now_mock.return_value, 'config_hash': 'h1'}},
|
||||
{},
|
||||
),
|
||||
call(
|
||||
'orchestrated_schedules',
|
||||
{'schedule_name': 'test2', 'namespace': 'test2'},
|
||||
{'$set': {'updated_at': now_mock.return_value, 'config_hash': None}},
|
||||
{},
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
@patch('orchestrator.activities.mongo_db.now')
|
||||
@mark.parametrize(
|
||||
@pytest.mark.parametrize(
|
||||
'pipelines', [[], [{'schedule_name': 'test1', 'namespace': 'test1', 'success': False}]]
|
||||
)
|
||||
def test_update_pipelines_timestamps_all_failed_stamps_nothing(now_mock, mongo_db, pipelines):
|
||||
input_data = {'updated_pipelines': pipelines}
|
||||
mongo_db.mongo_db_repository.update_many = MagicMock()
|
||||
mongo_db.mongo_db_repository.update_one = MagicMock()
|
||||
|
||||
mongo_db.update_pipelines_timestamps(input_data)
|
||||
|
||||
# An empty filter would match and stamp every tracked schedule, so the failed one
|
||||
# would never be retried.
|
||||
mongo_db.mongo_db_repository.update_many.assert_not_called()
|
||||
# A failed update is not stamped, so it is retried on the next tick
|
||||
mongo_db.mongo_db_repository.update_one.assert_not_called()
|
||||
|
||||
|
||||
@patch('orchestrator.activities.mongo_db.now')
|
||||
def test_update_pipelines_timestamps_mixed_stamps_only_successes(now_mock, mongo_db):
|
||||
input_data = {
|
||||
'updated_pipelines': [
|
||||
{'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
|
||||
{'schedule_name': 'test2', 'namespace': 'test2', 'success': False},
|
||||
{'schedule_name': 'test1', 'namespace': 'test1', 'success': True, 'config_hash': 'h1'},
|
||||
{'schedule_name': 'test2', 'namespace': 'test2', 'success': False, 'config_hash': 'h2'},
|
||||
]
|
||||
}
|
||||
mongo_db.mongo_db_repository.update_many = MagicMock()
|
||||
mongo_db.mongo_db_repository.update_one = MagicMock()
|
||||
|
||||
mongo_db.update_pipelines_timestamps(input_data)
|
||||
|
||||
mongo_db.mongo_db_repository.update_many.assert_called_once_with(
|
||||
mongo_db.mongo_db_repository.update_one.assert_called_once_with(
|
||||
'orchestrated_schedules',
|
||||
{'$or': [{'schedule_name': 'test1', 'namespace': 'test1'}]},
|
||||
{'$set': {'updated_at': now_mock.return_value}},
|
||||
{'schedule_name': 'test1', 'namespace': 'test1'},
|
||||
{'$set': {'updated_at': now_mock.return_value, 'config_hash': 'h1'}},
|
||||
{},
|
||||
)
|
||||
|
||||
@@ -310,15 +337,17 @@ def test_update_pipelines_timestamps_failure(now_mock, mongo_db):
|
||||
**metadata,
|
||||
}
|
||||
|
||||
mongo_db.mongo_db_repository.update_many.side_effect = Exception('Error')
|
||||
# The first update fails: the second is still tried, and only the failure is notified
|
||||
mongo_db.mongo_db_repository.update_one.side_effect = [Exception('Error'), 1]
|
||||
try:
|
||||
mongo_db.update_pipelines_timestamps(input_data)
|
||||
except Exception as e:
|
||||
assert str(e) == 'Error'
|
||||
except RuntimeError as e:
|
||||
assert str(e) == 'Failed to update 1 of 2 pipelines timestamps'
|
||||
assert mongo_db.mongo_db_repository.update_one.call_count == 2
|
||||
mongo_db.send_notification.assert_called_once_with(
|
||||
metadata=metadata['metadata'],
|
||||
notification_id='MONGODB_UPDATE_PIPELINES_ERROR',
|
||||
message='Failed to update pipelines timestamps: Error',
|
||||
message='Failed to update pipelines timestamps: test1/test1: Error',
|
||||
block='update_pipelines_timestamps',
|
||||
level=NotificationLevel.ERROR,
|
||||
attachment_content=ANY,
|
||||
@@ -328,11 +357,37 @@ def test_update_pipelines_timestamps_failure(now_mock, mongo_db):
|
||||
raise AssertionError('Expected an exception to be raised')
|
||||
|
||||
|
||||
@patch('orchestrator.activities.mongo_db.now')
|
||||
def test_update_pipelines_timestamps_failures_notified_together(now_mock, mongo_db):
|
||||
input_data = {
|
||||
'updated_pipelines': [
|
||||
{'schedule_name': 'test1', 'namespace': 'ns', 'success': True},
|
||||
{'schedule_name': 'test2', 'namespace': 'ns', 'success': True},
|
||||
{'schedule_name': 'test3', 'namespace': 'ns', 'success': True},
|
||||
],
|
||||
**metadata,
|
||||
}
|
||||
mongo_db.mongo_db_repository.update_one.side_effect = [
|
||||
Exception('down'),
|
||||
1,
|
||||
Exception('timeout'),
|
||||
]
|
||||
|
||||
with pytest.raises(RuntimeError, match='Failed to update 2 of 3 pipelines timestamps'):
|
||||
mongo_db.update_pipelines_timestamps(input_data)
|
||||
|
||||
assert mongo_db.mongo_db_repository.update_one.call_count == 3
|
||||
mongo_db.send_notification.assert_called_once()
|
||||
assert mongo_db.send_notification.call_args.kwargs['message'] == (
|
||||
'Failed to update pipelines timestamps: ns/test1: down; ns/test3: timeout'
|
||||
)
|
||||
|
||||
|
||||
@patch('orchestrator.activities.mongo_db.now')
|
||||
def test_create_pipelines_timestamps_success(now_mock, mongo_db):
|
||||
input_data = {
|
||||
'created_pipelines': [
|
||||
{'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
|
||||
{'schedule_name': 'test1', 'namespace': 'test1', 'success': True, 'config_hash': 'h1'},
|
||||
{'schedule_name': 'test2', 'namespace': 'test2', 'success': True},
|
||||
]
|
||||
}
|
||||
@@ -341,8 +396,18 @@ def test_create_pipelines_timestamps_success(now_mock, mongo_db):
|
||||
mongo_db.mongo_db_repository.insert_many.assert_called_once_with(
|
||||
'orchestrated_schedules',
|
||||
[
|
||||
{'schedule_name': 'test1', 'namespace': 'test1', 'updated_at': now_mock.return_value},
|
||||
{'schedule_name': 'test2', 'namespace': 'test2', 'updated_at': now_mock.return_value},
|
||||
{
|
||||
'schedule_name': 'test1',
|
||||
'namespace': 'test1',
|
||||
'updated_at': now_mock.return_value,
|
||||
'config_hash': 'h1',
|
||||
},
|
||||
{
|
||||
'schedule_name': 'test2',
|
||||
'namespace': 'test2',
|
||||
'updated_at': now_mock.return_value,
|
||||
'config_hash': None,
|
||||
},
|
||||
],
|
||||
{},
|
||||
)
|
||||
@@ -397,7 +462,7 @@ def test_delete_pipelines_timestamps_success(now_mock, mongo_db):
|
||||
)
|
||||
|
||||
|
||||
@mark.parametrize(
|
||||
@pytest.mark.parametrize(
|
||||
'pipelines', [[], [{'schedule_name': 'test1', 'namespace': 'test1', 'success': False}]]
|
||||
)
|
||||
def test_delete_pipelines_timestamps_all_failed_deletes_nothing(mongo_db, pipelines):
|
||||
@@ -704,3 +769,48 @@ def test_load_latest_data_error(mongo_db):
|
||||
level=NotificationLevel.ERROR,
|
||||
attachment_content=ANY,
|
||||
)
|
||||
|
||||
|
||||
def test_find_documents_in_mongodb_leaves_documents_without_timestamp_field(mongo_db):
|
||||
mongo_db.mongo_db_repository.find = MagicMock(return_value=[{'name': 'no-timestamp'}])
|
||||
|
||||
result = mongo_db.find_documents_in_mongodb(
|
||||
{'query': {'collection': 'test_collection'}, 'timestamp_fields': ['timestamp']}
|
||||
)
|
||||
|
||||
assert result == [{'name': 'no-timestamp'}]
|
||||
|
||||
|
||||
def test_aggregate_documents_in_mongodb_leaves_documents_without_timestamp_field(mongo_db):
|
||||
mongo_db.mongo_db_repository.aggregate = MagicMock(return_value=[{'name': 'no-timestamp'}])
|
||||
|
||||
result = mongo_db.aggregate_documents_in_mongodb(
|
||||
{
|
||||
'query': {'collection': 'test_collection', 'aggregation': [{'$match': {}}]},
|
||||
'timestamp_fields': ['timestamp'],
|
||||
}
|
||||
)
|
||||
|
||||
assert result == [{'name': 'no-timestamp'}]
|
||||
|
||||
|
||||
@patch('orchestrator.activities.mongo_db.now')
|
||||
def test_create_pipelines_timestamps_all_failed_inserts_nothing(_now_mock, mongo_db):
|
||||
mongo_db.mongo_db_repository.insert_many = MagicMock()
|
||||
|
||||
mongo_db.create_pipelines_timestamps(
|
||||
{'created_pipelines': [{'schedule_name': 'a', 'namespace': 'n', 'success': False}]}
|
||||
)
|
||||
|
||||
mongo_db.mongo_db_repository.insert_many.assert_not_called()
|
||||
|
||||
|
||||
def test_create_collection_with_ttl_index_skips_pipelines_without_topic(mongo_db):
|
||||
database = mongo_db.mongo_db_repository.database
|
||||
database.list_collection_names = MagicMock(return_value=[])
|
||||
|
||||
mongo_db.create_collection_with_ttl_index(
|
||||
{**metadata, 'pipelines': {'pi-scouter': {'model_tags': {}}, 'held': {'held': True}}}
|
||||
)
|
||||
|
||||
database.create_collection.assert_not_called()
|
||||
|
||||
@@ -145,6 +145,7 @@ async def test_create_schedule(
|
||||
'data': {'test': 'test'},
|
||||
'execution_timeout_seconds': 100,
|
||||
'task_timeout_seconds': 100,
|
||||
'config_hash': 'h1',
|
||||
},
|
||||
'test-schedule-invalid-frequency': {
|
||||
'model_id': 2,
|
||||
@@ -291,24 +292,29 @@ async def test_create_schedule(
|
||||
]
|
||||
)
|
||||
|
||||
# config_hash goes to the report (orchestrated_schedules), not to the workflow
|
||||
assert 'config_hash' not in input_data['schedules']['scouter']['test-schedule']
|
||||
assert report == [
|
||||
{
|
||||
'schedule_name': 'test-schedule',
|
||||
'namespace': 'scouter',
|
||||
'success': True,
|
||||
'message': 'Schedule created successfully',
|
||||
'config_hash': 'h1',
|
||||
},
|
||||
{
|
||||
'schedule_name': 'test-schedule-invalid-frequency',
|
||||
'namespace': 'scouter',
|
||||
'success': False,
|
||||
'message': 'Invalid frequency',
|
||||
'config_hash': None,
|
||||
},
|
||||
{
|
||||
'schedule_name': 'test-schedule-laborious',
|
||||
'namespace': 'laborious',
|
||||
'success': True,
|
||||
'message': 'Schedule created successfully',
|
||||
'config_hash': None,
|
||||
},
|
||||
]
|
||||
|
||||
@@ -355,6 +361,7 @@ async def test_update_schedules(
|
||||
'workflow_type': 'scouter',
|
||||
'frequency': '1m',
|
||||
'data': {'test': 'test'},
|
||||
'config_hash': 'h1',
|
||||
},
|
||||
'test-schedule_no_handler': {
|
||||
'workflow_type': 'scouter',
|
||||
@@ -400,24 +407,28 @@ async def test_update_schedules(
|
||||
handler_scouter.update.assert_called_once()
|
||||
handler_laborious.update.assert_called_once()
|
||||
|
||||
assert 'config_hash' not in input_data['schedules']['scouter']['test-schedule']
|
||||
assert report == [
|
||||
{
|
||||
'schedule_name': 'test-schedule',
|
||||
'namespace': 'scouter',
|
||||
'success': True,
|
||||
'message': 'Schedule updated successfully',
|
||||
'config_hash': 'h1',
|
||||
},
|
||||
{
|
||||
'schedule_name': 'test-schedule_no_handler',
|
||||
'namespace': 'scouter',
|
||||
'success': False,
|
||||
'message': 'Schedule test-schedule_no_handler not found',
|
||||
'config_hash': None,
|
||||
},
|
||||
{
|
||||
'schedule_name': 'test-schedule-laborious',
|
||||
'namespace': 'laborious',
|
||||
'success': True,
|
||||
'message': 'Schedule updated successfully',
|
||||
'config_hash': None,
|
||||
},
|
||||
]
|
||||
|
||||
@@ -723,3 +734,23 @@ async def test_create_schedules_drift_keeps_default_skip_overlap(
|
||||
mock_schedule.assert_called_once()
|
||||
policy = mock_schedule.call_args.kwargs.get('policy')
|
||||
assert policy is None or policy.overlap == ScheduleOverlapPolicy.SKIP
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_normalize_schedules_keeps_schedules_known_in_mongo(temporal_manager):
|
||||
"""An orchestrated schedule still tracked in orchestrated_schedules is not deleted"""
|
||||
input_data = {
|
||||
'orchestrated_schedules': {
|
||||
'scouter': {'test-schedule-id': '2021-01-01'},
|
||||
'laborious': {'test-schedule-id': '2021-01-01'},
|
||||
},
|
||||
**metadata,
|
||||
}
|
||||
for client in temporal_manager.temporal_clients.values():
|
||||
client.list_schedules = AsyncMock(return_value=async_iter())
|
||||
client.get_schedule_handle = MagicMock()
|
||||
|
||||
await temporal_manager.normalize_schedules(input_data)
|
||||
|
||||
for client in temporal_manager.temporal_clients.values():
|
||||
client.get_schedule_handle.assert_not_called()
|
||||
|
||||
82
tests/orchestrator/utils/tag_database_fixtures.py
Normal file
82
tests/orchestrator/utils/tag_database_fixtures.py
Normal file
@@ -0,0 +1,82 @@
|
||||
"""Pipeline and tag database documents shared by the utils tests."""
|
||||
|
||||
|
||||
def scouter_pipeline_doc(schedule_name, model_id, read_tags, workflow_type='scouter'):
|
||||
return {
|
||||
'workflow_type': workflow_type,
|
||||
'schedule_name': schedule_name,
|
||||
'model_id': model_id,
|
||||
'model': {'name': f'model {model_id}'},
|
||||
'read_tags': read_tags,
|
||||
}
|
||||
|
||||
|
||||
TAGS_BY_ID = {
|
||||
7: {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1001', 'tag_name': 'Pressure'},
|
||||
8: {'tag_id': 8, 'connector_id': 5, 'address': 'ns=3;i=1002', 'tag_name': 'Level'},
|
||||
}
|
||||
|
||||
|
||||
# Connector security and client certificates (SIENTIAPDE-1974)
|
||||
|
||||
NO_SECURITY = {'mode': 'None', 'policy': 'None'}
|
||||
SECURE = {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}
|
||||
|
||||
|
||||
# A client certificate as connectors_query's $lookup brings it (certificates[])
|
||||
def certificate_doc(certificate_id=1, name='sientia-platform', **extra):
|
||||
return {
|
||||
'certificate_id': certificate_id,
|
||||
'application_uri': 'urn:sientia:opc-ingestor',
|
||||
'cert': {'bucket': 'opc-certificates', 'object_name': f'client/{name}/cert.pem'},
|
||||
'private_key': {'bucket': 'opc-certificates', 'object_name': f'client/{name}/key.pem'},
|
||||
**extra,
|
||||
}
|
||||
|
||||
|
||||
# The slot's certificate for certificate_doc(): the same fields
|
||||
CERTIFICATE = certificate_doc()
|
||||
|
||||
|
||||
def secure_connector_fields(certificate_id=1, certificates=None):
|
||||
"""security and certificates of a connector with a client certificate."""
|
||||
return {
|
||||
'security': {**SECURE, 'client_certificate_id': certificate_id},
|
||||
'certificates': [certificate_doc(certificate_id)] if certificates is None else certificates,
|
||||
}
|
||||
|
||||
|
||||
# Write tags (predictions_batch write_tags through the tag database)
|
||||
|
||||
OPC_CONNECTORS = {
|
||||
'plant': {
|
||||
'connector_id': 'plant',
|
||||
'name': 'plant',
|
||||
'url': 'opc.tcp://plant:4840',
|
||||
'security': NO_SECURITY,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def write_tag_doc(tag_id, address, category='Write', connector_id='plant', **extra):
|
||||
return {
|
||||
'tag_id': tag_id,
|
||||
'connector_id': connector_id,
|
||||
'address': address,
|
||||
'tag_name': f'tag-{tag_id}',
|
||||
'category': category,
|
||||
'status': 'Active',
|
||||
**extra,
|
||||
}
|
||||
|
||||
|
||||
def batch_pipeline_doc(schedule_name, write_tags, **fields):
|
||||
return {
|
||||
'workflow_type': 'predictions_batch',
|
||||
'schedule_name': schedule_name,
|
||||
'model_id': 'model_id',
|
||||
'model': {'name': 'model_name'},
|
||||
'query': 'SELECT 1',
|
||||
'write_tags': write_tags,
|
||||
**fields,
|
||||
}
|
||||
450
tests/orchestrator/utils/test_opc_slots.py
Normal file
450
tests/orchestrator/utils/test_opc_slots.py
Normal file
@@ -0,0 +1,450 @@
|
||||
import pytest
|
||||
|
||||
from orchestrator.utils.opc_slots import (
|
||||
build_tag_config,
|
||||
gather_read_tags,
|
||||
)
|
||||
from tests.orchestrator.utils.tag_database_fixtures import (
|
||||
CERTIFICATE,
|
||||
NO_SECURITY,
|
||||
SECURE,
|
||||
TAGS_BY_ID,
|
||||
certificate_doc,
|
||||
scouter_pipeline_doc,
|
||||
secure_connector_fields,
|
||||
)
|
||||
|
||||
NO_ISSUES = {'tags': set(), 'connectors': set(), 'invalid': set(), 'certificates': set()}
|
||||
|
||||
|
||||
def _name(schedule_name, tag_name, model_id):
|
||||
return {
|
||||
'collection': f'raw_{schedule_name}',
|
||||
'tag_name': tag_name,
|
||||
'model_id': model_id,
|
||||
'model_name': f'model {model_id}',
|
||||
}
|
||||
|
||||
|
||||
def _read(tag_id, frequency=1000, **extra):
|
||||
return {'tag_id': tag_id, 'frequency': frequency, **extra}
|
||||
|
||||
|
||||
def _connector(connector_id, name, **extra):
|
||||
return {
|
||||
'connector_id': connector_id,
|
||||
'name': name,
|
||||
'url': f'opc.tcp://{name}:4840',
|
||||
'security': NO_SECURITY,
|
||||
**extra,
|
||||
}
|
||||
|
||||
|
||||
def _db_tag(tag_id, connector_id, address, tag_name):
|
||||
return {
|
||||
'tag_id': tag_id,
|
||||
'connector_id': connector_id,
|
||||
'address': address,
|
||||
'tag_name': tag_name,
|
||||
}
|
||||
|
||||
|
||||
connectors = {
|
||||
'1': _connector(1, 'kepserver'),
|
||||
'3': _connector(3, 'plant', subscription_period_ms=1500, **secure_connector_fields()),
|
||||
}
|
||||
|
||||
|
||||
tags_by_id = {
|
||||
**TAGS_BY_ID,
|
||||
10: _db_tag(10, 1, 'ns=2;s=A', 'A'),
|
||||
11: _db_tag(11, 1, 'ns=2;s=B', 'B'),
|
||||
12: _db_tag(12, 1, 'ns=2;s=A', 'A2'),
|
||||
13: _db_tag(13, 3, 'ns=2;s=A', 'A on plant'),
|
||||
14: _db_tag(14, 1, 'ns=2;s=C', 'C'),
|
||||
}
|
||||
|
||||
|
||||
def _gather(pipelines, connector_index=None, tag_index=None):
|
||||
return gather_read_tags(
|
||||
pipelines,
|
||||
connectors if connector_index is None else connector_index,
|
||||
tags_by_id if tag_index is None else tag_index,
|
||||
)
|
||||
|
||||
|
||||
def test_gather_read_tags_multi_name():
|
||||
"""One address read by two pipelines with different names: one entry, two names"""
|
||||
pipelines = [
|
||||
scouter_pipeline_doc('sin', 'm1', [_read(10, 15000, tag_name='Counter')]),
|
||||
scouter_pipeline_doc('cos', 'm2', [_read(10, 5000, tag_name='Contador')]),
|
||||
]
|
||||
|
||||
entries, issues = _gather(pipelines)
|
||||
|
||||
assert len(entries) == 1
|
||||
assert entries[0]['address'] == 'ns=2;s=A'
|
||||
assert entries[0]['frequency'] == 5000
|
||||
assert entries[0]['names'] == [_name('cos', 'Contador', 'm2'), _name('sin', 'Counter', 'm1')]
|
||||
assert issues == NO_ISSUES
|
||||
|
||||
|
||||
def test_gather_read_tags_same_name_in_two_pipelines():
|
||||
"""The same name read by two pipelines still goes to both collections"""
|
||||
pipelines = [
|
||||
scouter_pipeline_doc('sin', 'm1', [_read(10)]),
|
||||
scouter_pipeline_doc('cos', 'm2', [_read(10)]),
|
||||
]
|
||||
|
||||
entries, _issues = _gather(pipelines)
|
||||
|
||||
assert entries[0]['names'] == [_name('cos', 'A', 'm2'), _name('sin', 'A', 'm1')]
|
||||
|
||||
|
||||
def test_gather_read_tags_duplicate_tag_in_one_pipeline():
|
||||
entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(10), _read(10)])])
|
||||
|
||||
assert entries[0]['names'] == [_name('sin', 'A', 'm1')]
|
||||
|
||||
|
||||
def test_gather_read_tags_two_tag_ids_on_one_address():
|
||||
"""Two connector_tags on the same node: one entry, one name per tag"""
|
||||
entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(10), _read(12)])])
|
||||
|
||||
assert len(entries) == 1
|
||||
assert entries[0]['names'] == [_name('sin', 'A', 'm1'), _name('sin', 'A2', 'm1')]
|
||||
|
||||
|
||||
def test_gather_read_tags_same_address_on_two_servers():
|
||||
entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(13), _read(10)])])
|
||||
|
||||
assert [(e['server']['server_id'], e['address']) for e in entries] == [
|
||||
('1', 'ns=2;s=A'),
|
||||
('3', 'ns=2;s=A'),
|
||||
]
|
||||
|
||||
|
||||
def test_gather_read_tags_with_tag_database():
|
||||
"""Address, name and server come from connector_tags + connectors"""
|
||||
entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(7)])])
|
||||
|
||||
assert entries == [
|
||||
{
|
||||
'server': {
|
||||
'server_id': '3',
|
||||
'name': 'plant',
|
||||
'url': 'opc.tcp://plant:4840',
|
||||
'subscription_period_ms': 1500,
|
||||
'security': SECURE,
|
||||
'certificate': CERTIFICATE,
|
||||
'key': 'plant',
|
||||
},
|
||||
'address': 'ns=3;i=1001',
|
||||
'frequency': 1000,
|
||||
'active_read': {'enabled': False},
|
||||
'names': [_name('sin', 'Pressure', 'm1')],
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def test_gather_read_tags_pipeline_name_wins_over_tag_database():
|
||||
entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(7, tag_name='P1')])])
|
||||
|
||||
assert entries[0]['names'] == [_name('sin', 'P1', 'm1')]
|
||||
|
||||
|
||||
def test_gather_read_tags_unknown_tag_id():
|
||||
entries, issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(404)])])
|
||||
|
||||
assert entries == []
|
||||
assert issues == {**NO_ISSUES, 'tags': {'404'}}
|
||||
|
||||
|
||||
def test_gather_read_tags_inactive_connector():
|
||||
"""Connector 5 is not among the active connectors: its tag is left out"""
|
||||
entries, issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(8)])])
|
||||
|
||||
assert entries == []
|
||||
assert issues == {**NO_ISSUES, 'connectors': {'5'}}
|
||||
|
||||
|
||||
def test_gather_read_tags_without_tag_id():
|
||||
"""connectors is the only source of the servers: a read tag needs a tag_id"""
|
||||
pipelines = [
|
||||
scouter_pipeline_doc(
|
||||
'sin',
|
||||
'm1',
|
||||
[
|
||||
_read(10),
|
||||
{'tag_name': 'b', 'frequency': 1000},
|
||||
{'tag_id': '', 'tag_name': 'c', 'frequency': 1000},
|
||||
{'tag_id': None, 'tag_name': 'd', 'frequency': 1000},
|
||||
# tag_id is int in connector_tags: text is no reference
|
||||
{'tag_id': '11', 'frequency': 1000},
|
||||
],
|
||||
)
|
||||
]
|
||||
|
||||
entries, issues = _gather(pipelines)
|
||||
|
||||
assert [e['address'] for e in entries] == ['ns=2;s=A']
|
||||
assert issues == {**NO_ISSUES, 'invalid': {'sin/b', 'sin/c', 'sin/d', 'sin/11'}}
|
||||
|
||||
|
||||
def test_gather_read_tags_server_name_clash():
|
||||
"""Two servers with the same name get distinct slot keys, stable by server_id"""
|
||||
connector_index = {**connectors, '2': _connector(2, 'kepserver')}
|
||||
tag_index = {**tags_by_id, 20: _db_tag(20, 2, 'ns=2;s=A', 'A')}
|
||||
pipelines = [scouter_pipeline_doc('sin', 'm1', [_read(20), _read(11)])]
|
||||
|
||||
entries, _issues = _gather(pipelines, connector_index, tag_index)
|
||||
|
||||
assert [e['server']['key'] for e in entries] == ['kepserver', 'kepserver:2']
|
||||
assert [e['server']['name'] for e in entries] == ['kepserver', 'kepserver']
|
||||
|
||||
|
||||
def test_gather_read_tags_active_read():
|
||||
"""Active read: on if any pipeline asks, with the lowest cycle_multiple (D3)"""
|
||||
pipelines = [
|
||||
scouter_pipeline_doc('p1', 'm1', [_read(10)]),
|
||||
scouter_pipeline_doc(
|
||||
'p2', 'm2', [_read(10, active_read={'enabled': True, 'cycle_multiple': 4})]
|
||||
),
|
||||
scouter_pipeline_doc(
|
||||
'p3', 'm3', [_read(10, active_read={'enabled': True, 'cycle_multiple': 2})]
|
||||
),
|
||||
scouter_pipeline_doc(
|
||||
'p4', 'm4', [_read(11, active_read={'enabled': False, 'cycle_multiple': 1})]
|
||||
),
|
||||
scouter_pipeline_doc('p5', 'm5', [_read(14, active_read={'enabled': True})]),
|
||||
]
|
||||
|
||||
entries, _issues = _gather(pipelines)
|
||||
|
||||
assert [e['active_read'] for e in entries] == [
|
||||
{'enabled': True, 'cycle_multiple': 2},
|
||||
{'enabled': False},
|
||||
{'enabled': True, 'cycle_multiple': 1},
|
||||
]
|
||||
|
||||
|
||||
def test_gather_read_tags_filters_non_scouter_workflows():
|
||||
"""Only scouter pipelines feed the slots"""
|
||||
pipelines = [
|
||||
scouter_pipeline_doc('sin', 'm1', [_read(10)]),
|
||||
scouter_pipeline_doc('pred', 'm2', [_read(11)], workflow_type='predictions_batch'),
|
||||
scouter_pipeline_doc('retrain', 'm3', [_read(14)], workflow_type='minimal_retrain'),
|
||||
scouter_pipeline_doc('drift', 'm4', [_read(7)], workflow_type='drift'),
|
||||
]
|
||||
|
||||
entries, _issues = _gather(pipelines)
|
||||
|
||||
assert [e['address'] for e in entries] == ['ns=2;s=A']
|
||||
|
||||
|
||||
def test_gather_read_tags_without_read_tags():
|
||||
pipeline = scouter_pipeline_doc('sin', 'm1', [])
|
||||
del pipeline['read_tags']
|
||||
|
||||
assert _gather([pipeline]) == ([], NO_ISSUES)
|
||||
|
||||
|
||||
def test_build_tag_config():
|
||||
"""Slot v2 (contract C1): security, certificate and subscription period per server"""
|
||||
pipelines = [
|
||||
scouter_pipeline_doc('sin', 'm1', [_read(10, 1000), _read(11, '300'), _read(7, 100)])
|
||||
]
|
||||
entries, _issues = _gather(pipelines)
|
||||
|
||||
result = build_tag_config(entries)
|
||||
|
||||
assert result == {
|
||||
'kepserver': {
|
||||
'server_id': '1',
|
||||
'name': 'kepserver',
|
||||
'url': 'opc.tcp://kepserver:4840',
|
||||
# No client certificate: no security (D5)
|
||||
'security': NO_SECURITY,
|
||||
'certificate': None,
|
||||
# No subscription_period_ms on the connector: half its lowest tag frequency
|
||||
'subscription_period_ms': 150,
|
||||
'tags': {
|
||||
'ns=2;s=A': {
|
||||
'frequency': 1000,
|
||||
'active_read': {'enabled': False},
|
||||
'names': [_name('sin', 'A', 'm1')],
|
||||
},
|
||||
'ns=2;s=B': {
|
||||
'frequency': 300,
|
||||
'active_read': {'enabled': False},
|
||||
'names': [_name('sin', 'B', 'm1')],
|
||||
},
|
||||
},
|
||||
},
|
||||
'plant': {
|
||||
'server_id': '3',
|
||||
'name': 'plant',
|
||||
'url': 'opc.tcp://plant:4840',
|
||||
'subscription_period_ms': 1500,
|
||||
'security': SECURE,
|
||||
'certificate': CERTIFICATE,
|
||||
'tags': {
|
||||
'ns=3;i=1001': {
|
||||
'frequency': 100,
|
||||
'active_read': {'enabled': False},
|
||||
'names': [_name('sin', 'Pressure', 'm1')],
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def test_build_tag_config_empty():
|
||||
assert build_tag_config([]) == {}
|
||||
|
||||
|
||||
def test_gather_read_tags_invalid_read_tags():
|
||||
"""Unusable read tags are reported and skipped; the others still make the slot"""
|
||||
pipelines = [
|
||||
scouter_pipeline_doc('sin', 'm1', [_read(10), {'tag_id': 11}, _read(7, frequency='fast')])
|
||||
]
|
||||
|
||||
entries, issues = _gather(pipelines)
|
||||
|
||||
assert [e['address'] for e in entries] == ['ns=2;s=A']
|
||||
assert issues['invalid'] == {'sin/11', 'sin/7'}
|
||||
|
||||
|
||||
def test_gather_read_tags_frequency_type_does_not_depend_on_order():
|
||||
"""15000 and '15000' give the same int frequency in any pipeline order (O6)"""
|
||||
as_int = scouter_pipeline_doc('sin', 'm1', [_read(10, 15000)])
|
||||
as_text = scouter_pipeline_doc('cos', 'm2', [_read(10, '15000')])
|
||||
|
||||
first, _ = _gather([as_int, as_text])
|
||||
second, _ = _gather([as_text, as_int])
|
||||
|
||||
assert first[0]['frequency'] == second[0]['frequency'] == 15000
|
||||
assert isinstance(first[0]['frequency'], int)
|
||||
assert isinstance(second[0]['frequency'], int)
|
||||
|
||||
|
||||
def test_gather_read_tags_server_key_never_collides():
|
||||
"""A server literally named 'x:3' does not share its key with server 3"""
|
||||
connector_index = {
|
||||
'1': _connector(1, 'x:3'),
|
||||
'2': _connector(2, 'x'),
|
||||
'3': _connector(3, 'x'),
|
||||
}
|
||||
tag_index = {
|
||||
1: _db_tag(1, 1, 'ns=2;s=A', 'a'),
|
||||
2: _db_tag(2, 2, 'ns=2;s=B', 'b'),
|
||||
3: _db_tag(3, 3, 'ns=2;s=C', 'c'),
|
||||
}
|
||||
pipelines = [scouter_pipeline_doc('sin', 'm1', [_read(1), _read(2), _read(3)])]
|
||||
|
||||
entries, _issues = _gather(pipelines, connector_index, tag_index)
|
||||
slot = build_tag_config(entries)
|
||||
|
||||
keys = [e['server']['key'] for e in entries]
|
||||
assert len(set(keys)) == 3
|
||||
assert {server['server_id'] for server in slot.values()} == {'1', '2', '3'}
|
||||
assert all(len(server['tags']) == 1 for server in slot.values())
|
||||
|
||||
|
||||
def test_gather_read_tags_invalid_cycle_multiple_does_not_fail_the_slot():
|
||||
"""A bad active_read is reported as invalid instead of failing every pipeline's slot"""
|
||||
pipelines = [
|
||||
scouter_pipeline_doc(
|
||||
'sin',
|
||||
'm1',
|
||||
[
|
||||
_read(10),
|
||||
_read(11, active_read={'enabled': True, 'cycle_multiple': 'x'}),
|
||||
_read(7, active_read={'enabled': True, 'cycle_multiple': None}),
|
||||
# Not a dict: treated as no active read
|
||||
_read(14, active_read=True),
|
||||
],
|
||||
)
|
||||
]
|
||||
|
||||
entries, issues = _gather(pipelines)
|
||||
|
||||
assert [e['address'] for e in entries] == ['ns=2;s=A', 'ns=2;s=C']
|
||||
assert entries[1]['active_read'] == {'enabled': False}
|
||||
assert issues['invalid'] == {'sin/11', 'sin/7'}
|
||||
|
||||
|
||||
def test_gather_read_tags_skips_incomplete_pipelines():
|
||||
"""A pipeline without schedule_name, model_id or model.name is reported, not raised"""
|
||||
no_model = scouter_pipeline_doc('no-model', 'm1', [_read(10)])
|
||||
del no_model['model']
|
||||
no_name = scouter_pipeline_doc('no-name', 'm2', [_read(10)])
|
||||
no_name['model'] = {}
|
||||
no_model_id = scouter_pipeline_doc('no-id', None, [_read(10)])
|
||||
no_schedule = scouter_pipeline_doc(None, 'm4', [_read(10)])
|
||||
ok = scouter_pipeline_doc('ok', 'm5', [_read(11)])
|
||||
|
||||
entries, issues = _gather([no_model, no_name, no_model_id, no_schedule, ok])
|
||||
|
||||
assert [e['address'] for e in entries] == ['ns=2;s=B']
|
||||
assert issues['invalid'] == {'no-model', 'no-name', 'no-id', 'None'}
|
||||
|
||||
|
||||
def _plant_with(**fields):
|
||||
return {'1': connectors['1'], '3': {**connectors['3'], **fields}}
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'fields',
|
||||
[
|
||||
# $lookup found nothing: missing id, inactive or not a client certificate
|
||||
pytest.param(secure_connector_fields(certificates=[]), id='not-found'),
|
||||
pytest.param(
|
||||
secure_connector_fields(certificates=[certificate_doc(application_uri='')]),
|
||||
id='no-application-uri',
|
||||
),
|
||||
pytest.param(
|
||||
secure_connector_fields(
|
||||
certificates=[{**certificate_doc(), 'private_key': {'bucket': 'b'}}]
|
||||
),
|
||||
id='key-without-object',
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_gather_read_tags_registered_certificate_missing(fields):
|
||||
"""An id without a usable certificate: in the slot without a certificate, with a warning (D5)"""
|
||||
entries, issues = _gather(
|
||||
[scouter_pipeline_doc('sin', 'm1', [_read(7)])], _plant_with(**fields)
|
||||
)
|
||||
|
||||
assert entries[0]['server']['certificate'] is None
|
||||
assert entries[0]['server']['security'] == SECURE
|
||||
assert issues == {**NO_ISSUES, 'certificates': {'3'}}
|
||||
|
||||
|
||||
@pytest.mark.parametrize('certificate_id', [None, 'absent'])
|
||||
def test_gather_read_tags_without_certificate_id(certificate_id):
|
||||
"""No client_certificate_id: no certificate and nothing to report (D5)"""
|
||||
security = {**NO_SECURITY, 'client_certificate_id': certificate_id}
|
||||
if certificate_id == 'absent':
|
||||
security = NO_SECURITY
|
||||
entries, issues = _gather(
|
||||
[scouter_pipeline_doc('sin', 'm1', [_read(7)])],
|
||||
_plant_with(security=security, certificates=[]),
|
||||
)
|
||||
|
||||
assert entries[0]['server']['certificate'] is None
|
||||
assert issues == NO_ISSUES
|
||||
|
||||
|
||||
def test_gather_read_tags_reports_a_certificate_once_per_connector():
|
||||
entries, issues = _gather(
|
||||
[
|
||||
scouter_pipeline_doc('sin', 'm1', [_read(7)]),
|
||||
scouter_pipeline_doc('cos', 'm2', [_read(7)]),
|
||||
],
|
||||
_plant_with(**secure_connector_fields(certificates=[])),
|
||||
)
|
||||
|
||||
assert len(entries) == 1
|
||||
assert issues['certificates'] == {'3'}
|
||||
@@ -2,14 +2,12 @@ from unittest.mock import call, patch
|
||||
|
||||
from pytest import mark
|
||||
|
||||
from orchestrator.utils.orchestrator_functions import (
|
||||
from orchestrator.utils.pipeline_configs import (
|
||||
base_scouter,
|
||||
build_tag_config,
|
||||
chunk_period_warning,
|
||||
common_config,
|
||||
drift,
|
||||
frequency_alert_warning,
|
||||
gather_read_tags,
|
||||
minimal_retrain,
|
||||
overlap_filter_config,
|
||||
pi_web_api_scouter,
|
||||
@@ -345,11 +343,11 @@ def test_process_path_priority():
|
||||
|
||||
|
||||
@patch(
|
||||
'orchestrator.utils.orchestrator_functions.overlap_filter_config',
|
||||
'orchestrator.utils.pipeline_configs.overlap_filter_config',
|
||||
return_value={'test_filter_name': {'policy': 'test_policy', 'config': {}}},
|
||||
)
|
||||
@patch(
|
||||
'orchestrator.utils.orchestrator_functions.process_path_priority',
|
||||
'orchestrator.utils.pipeline_configs.process_path_priority',
|
||||
return_value=['STOP', 'CONTINUE', 'REPEAT'],
|
||||
)
|
||||
def test_predictions_batch(mock_process_path_priority, mock_overlap_filter_config):
|
||||
@@ -359,10 +357,14 @@ def test_predictions_batch(mock_process_path_priority, mock_overlap_filter_confi
|
||||
'model_id': 'test_model_id',
|
||||
'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
|
||||
'query': 'test_query',
|
||||
'write_tags': [
|
||||
{'server_id': 'test_server_id', 'type': 'prediction', 'addr': 'test_addr'},
|
||||
{'server_id': 'test_server_id', 'type': 'confidence', 'addr': 'test_addr'},
|
||||
],
|
||||
'write_tags': [{'tag_id': 77}, {'tag_id': 78}],
|
||||
# Resolved from write_tags by process_schedules
|
||||
'opc_output_config': {
|
||||
'test_server_id': {
|
||||
'prediction_tags': {'test_addr': {'data_type': 'float'}},
|
||||
'confidence_tags': {'test_addr': {'data_type': 'float'}},
|
||||
}
|
||||
},
|
||||
'pi_web_api_output_config': {
|
||||
'endpoint': 'test_endpoint',
|
||||
'prediction_tags': {
|
||||
@@ -445,249 +447,17 @@ def test_predictions_batch(mock_process_path_priority, mock_overlap_filter_confi
|
||||
assert result == expected
|
||||
|
||||
|
||||
def test_gather_read_tags():
|
||||
pipelines = [
|
||||
{
|
||||
'workflow_type': 'scouter',
|
||||
'schedule_name': 'test_schedule',
|
||||
'read_tags': [
|
||||
{
|
||||
'server_id': '1',
|
||||
'server_name': 'test_server_name',
|
||||
'tag_address': 'test_tag_address',
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
'workflow_type': 'scouter',
|
||||
'schedule_name': 'test_schedule2',
|
||||
'read_tags': [
|
||||
{
|
||||
'server_id': '2',
|
||||
'server_name': 'test_server_name2',
|
||||
'tag_address': 'test_tag_address2',
|
||||
},
|
||||
{
|
||||
'server_id': '2',
|
||||
'server_name': 'test_server_name2',
|
||||
'tag_address': 'test_tag_address3',
|
||||
},
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
result = gather_read_tags(pipelines)
|
||||
|
||||
expected = {
|
||||
'1:test_tag_address': {
|
||||
'server_id': '1',
|
||||
'server_name': 'test_server_name',
|
||||
'tag_address': 'test_tag_address',
|
||||
'topics': ['raw_test_schedule'],
|
||||
},
|
||||
'2:test_tag_address2': {
|
||||
'server_id': '2',
|
||||
'server_name': 'test_server_name2',
|
||||
'tag_address': 'test_tag_address2',
|
||||
'topics': ['raw_test_schedule2'],
|
||||
},
|
||||
'2:test_tag_address3': {
|
||||
'server_id': '2',
|
||||
'server_name': 'test_server_name2',
|
||||
'tag_address': 'test_tag_address3',
|
||||
'topics': ['raw_test_schedule2'],
|
||||
},
|
||||
def test_scouter_skips_tags_without_name():
|
||||
"""A tag_id the tag database cannot resolve has no name and stays out of model_tags"""
|
||||
config = {
|
||||
'workflow_type': 'scouter',
|
||||
'schedule_name': 'sin',
|
||||
'model_id': 'm1',
|
||||
'model': {'name': 'model m1'},
|
||||
'read_tags': [{'tag_name': 'a'}, {'tag_id': 404}],
|
||||
}
|
||||
|
||||
assert result == expected
|
||||
|
||||
|
||||
def test_gather_read_tags_filters_non_scouter_workflows():
|
||||
"""Test that gather_read_tags only processes scouter workflow types and ignores others"""
|
||||
pipelines = [
|
||||
{
|
||||
'workflow_type': 'scouter',
|
||||
'schedule_name': 'test_scouter_schedule',
|
||||
'read_tags': [
|
||||
{
|
||||
'server_id': '1',
|
||||
'server_name': 'test_server_name',
|
||||
'tag_address': 'test_tag_address',
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
'workflow_type': 'predictions_batch',
|
||||
'schedule_name': 'test_predictions_schedule',
|
||||
'read_tags': [
|
||||
{
|
||||
'server_id': '2',
|
||||
'server_name': 'test_server_name2',
|
||||
'tag_address': 'test_tag_address_predictions',
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
'workflow_type': 'minimal_retrain',
|
||||
'schedule_name': 'test_retrain_schedule',
|
||||
'read_tags': [
|
||||
{
|
||||
'server_id': '3',
|
||||
'server_name': 'test_server_name3',
|
||||
'tag_address': 'test_tag_address_retrain',
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
'workflow_type': 'pi_web_api_scouter',
|
||||
'schedule_name': 'test_pi_web_api_schedule',
|
||||
'read_tags': [
|
||||
{
|
||||
'server_id': '4',
|
||||
'server_name': 'test_server_name4',
|
||||
'tag_address': 'test_tag_address_pi_web_api',
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
'workflow_type': 'drift',
|
||||
'schedule_name': 'test_drift_schedule',
|
||||
'read_tags': [
|
||||
{
|
||||
'server_id': '5',
|
||||
'server_name': 'test_server_name5',
|
||||
'tag_address': 'test_tag_address_drift',
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
'workflow_type': 'scouter',
|
||||
'schedule_name': 'test_scouter_schedule2',
|
||||
'read_tags': [
|
||||
{
|
||||
'server_id': '1',
|
||||
'server_name': 'test_server_name',
|
||||
'tag_address': 'test_tag_address2',
|
||||
}
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
result = gather_read_tags(pipelines)
|
||||
|
||||
# Only scouter workflow types should be included
|
||||
expected = {
|
||||
'1:test_tag_address': {
|
||||
'server_id': '1',
|
||||
'server_name': 'test_server_name',
|
||||
'tag_address': 'test_tag_address',
|
||||
'topics': ['raw_test_scouter_schedule'],
|
||||
},
|
||||
'1:test_tag_address2': {
|
||||
'server_id': '1',
|
||||
'server_name': 'test_server_name',
|
||||
'tag_address': 'test_tag_address2',
|
||||
'topics': ['raw_test_scouter_schedule2'],
|
||||
},
|
||||
}
|
||||
|
||||
assert result == expected
|
||||
# Ensure non-scouter pipelines are not included
|
||||
assert '2:test_tag_address_predictions' not in result
|
||||
assert '3:test_tag_address_retrain' not in result
|
||||
assert '4:test_tag_address_pi_web_api' not in result
|
||||
assert '5:test_tag_address_drift' not in result
|
||||
|
||||
|
||||
def test_build_tag_config():
|
||||
tags = [
|
||||
{
|
||||
'server_id': '1',
|
||||
'server_name': 'test_server_name',
|
||||
'tag_address': 'test_tag_address',
|
||||
'frequency': 1000,
|
||||
},
|
||||
{
|
||||
'server_id': '2',
|
||||
'server_name': 'test_server_name2',
|
||||
'tag_address': 'test_tag_address2',
|
||||
'frequency': 1000,
|
||||
},
|
||||
{
|
||||
'server_id': '2',
|
||||
'server_name': 'test_server_name2',
|
||||
'tag_address': 'test_tag_address3',
|
||||
'frequency': 300,
|
||||
},
|
||||
{
|
||||
'server_id': '3',
|
||||
'server_name': 'test_server_name3',
|
||||
'tag_address': 'test_tag_address4',
|
||||
'frequency': 200,
|
||||
},
|
||||
]
|
||||
|
||||
opc_servers = {
|
||||
'1': {
|
||||
'server_name': 'test_server_name',
|
||||
'url': 'test_url',
|
||||
'uri': 'test_uri',
|
||||
},
|
||||
'2': {
|
||||
'server_name': 'test_server_name2',
|
||||
'url': 'test_url2',
|
||||
'uri': 'test_uri2',
|
||||
},
|
||||
}
|
||||
|
||||
result = build_tag_config(tags, opc_servers)
|
||||
|
||||
expected = {
|
||||
'test_server_name': {
|
||||
'server_id': '1',
|
||||
'name': 'test_server_name',
|
||||
'url': 'test_url',
|
||||
'server_uri': 'test_uri',
|
||||
'cert_path': None,
|
||||
'private_key_path': None,
|
||||
'server_cert_path': None,
|
||||
'subscription_period_ms': 500,
|
||||
'tags': {
|
||||
'test_tag_address': {
|
||||
'server_id': '1',
|
||||
'server_name': 'test_server_name',
|
||||
'tag_address': 'test_tag_address',
|
||||
'frequency': 1000,
|
||||
},
|
||||
},
|
||||
},
|
||||
'test_server_name2': {
|
||||
'server_id': '2',
|
||||
'name': 'test_server_name2',
|
||||
'url': 'test_url2',
|
||||
'server_uri': 'test_uri2',
|
||||
'cert_path': None,
|
||||
'private_key_path': None,
|
||||
'server_cert_path': None,
|
||||
'subscription_period_ms': 150,
|
||||
'tags': {
|
||||
'test_tag_address2': {
|
||||
'server_id': '2',
|
||||
'server_name': 'test_server_name2',
|
||||
'tag_address': 'test_tag_address2',
|
||||
'frequency': 1000,
|
||||
},
|
||||
'test_tag_address3': {
|
||||
'server_id': '2',
|
||||
'server_name': 'test_server_name2',
|
||||
'tag_address': 'test_tag_address3',
|
||||
'frequency': 300,
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
assert result == (expected, ['3'])
|
||||
assert list(scouter(config)['model_tags']) == ['a']
|
||||
|
||||
|
||||
def test_base_scouter():
|
||||
@@ -912,3 +682,61 @@ def test_pi_web_api_scouter_with_no_timeout():
|
||||
)
|
||||
def test_frequency_alert_warning(frequency_alert, warns):
|
||||
assert (frequency_alert_warning(frequency_alert) is not None) is warns
|
||||
|
||||
|
||||
def test_pi_web_api_scouter_caps_api_timeout_at_frequency_only_when_larger():
|
||||
config = {
|
||||
'workflow_type': 'pi_web_api_scouter',
|
||||
'schedule_name': 'pi_schedule',
|
||||
'model_id': 'model_id',
|
||||
'model': {'name': 'model_name'},
|
||||
'frequency': '1m',
|
||||
'read_tags': {'TI-101': {'webid': 'F1DP-X'}},
|
||||
'pi_web_api_config': {
|
||||
'endpoint': '/streamsets/recorded',
|
||||
'max_count': 5,
|
||||
'api_timeout': 10,
|
||||
},
|
||||
}
|
||||
|
||||
result = pi_web_api_scouter(config)
|
||||
|
||||
assert result['model_tags'] == {
|
||||
'TI-101': {'webid': 'F1DP-X', 'aggr_func': 'lts', 'data_range': [-100, 100]}
|
||||
}
|
||||
assert result['pi_web_api_query'] == {
|
||||
'endpoint': '/streamsets/recorded',
|
||||
'period': '*-1d',
|
||||
'max_count': 5,
|
||||
'api_timeout': 10,
|
||||
}
|
||||
|
||||
|
||||
def test_predictions_batch_copies_the_resolved_opc_output_config():
|
||||
opc_output_config = {'plant': {'prediction_tags': {'ns=2;s=P1': {'data_type': 'int'}}}}
|
||||
config = {
|
||||
'workflow_type': 'predictions_batch',
|
||||
'schedule_name': 'batch',
|
||||
'model_id': 'model_id',
|
||||
'model': {'name': 'model_name'},
|
||||
'query': 'SELECT 1',
|
||||
'write_tags': [{'tag_id': 77}],
|
||||
'opc_output_config': opc_output_config,
|
||||
}
|
||||
|
||||
result = predictions_batch(config)
|
||||
|
||||
assert result['opc_output_config'] == opc_output_config
|
||||
assert 'write_tags' not in result
|
||||
|
||||
|
||||
def test_predictions_batch_without_write_tags_has_no_opc_output():
|
||||
config = {
|
||||
'workflow_type': 'predictions_batch',
|
||||
'schedule_name': 'batch',
|
||||
'model_id': 'model_id',
|
||||
'model': {'name': 'model_name'},
|
||||
'query': 'SELECT 1',
|
||||
}
|
||||
|
||||
assert predictions_batch(config)['opc_output_config'] == {}
|
||||
279
tests/orchestrator/utils/test_tag_database.py
Normal file
279
tests/orchestrator/utils/test_tag_database.py
Normal file
@@ -0,0 +1,279 @@
|
||||
import pytest
|
||||
|
||||
from orchestrator.utils.tag_database import (
|
||||
WRITE_CATEGORIES,
|
||||
client_certificate_id,
|
||||
filter_by_tag_ids,
|
||||
index_connector_tags,
|
||||
index_connectors,
|
||||
index_tag_documents,
|
||||
referenced_tag_ids,
|
||||
resolve_certificate,
|
||||
resolve_tag_names,
|
||||
resolve_write_tags,
|
||||
tag_id_of,
|
||||
)
|
||||
from tests.orchestrator.utils.tag_database_fixtures import (
|
||||
CERTIFICATE,
|
||||
NO_SECURITY,
|
||||
OPC_CONNECTORS,
|
||||
TAGS_BY_ID,
|
||||
batch_pipeline_doc,
|
||||
certificate_doc,
|
||||
scouter_pipeline_doc,
|
||||
secure_connector_fields,
|
||||
write_tag_doc,
|
||||
)
|
||||
|
||||
|
||||
def test_index_connector_tags():
|
||||
tags = [
|
||||
{'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1', 'tag_name': 'a'},
|
||||
# tag_id is int in connector_tags: a text id is no tag
|
||||
{'tag_id': '8', 'connector_id': 3, 'address': 'ns=3;i=2', 'tag_name': 'b'},
|
||||
]
|
||||
|
||||
assert index_connector_tags(tags) == {7: tags[0]}
|
||||
|
||||
|
||||
def test_resolve_tag_names():
|
||||
read_tags = [
|
||||
{'tag_id': 7},
|
||||
{'tag_id': 7, 'tag_name': 'own name'},
|
||||
{'tag_id': '7'},
|
||||
{'tag_id': 404},
|
||||
{'tag_name': 'no tag_id'},
|
||||
]
|
||||
|
||||
assert resolve_tag_names(read_tags, TAGS_BY_ID) == [
|
||||
{'tag_id': 7, 'tag_name': 'Pressure'},
|
||||
{'tag_id': 7, 'tag_name': 'own name'},
|
||||
{'tag_id': '7'},
|
||||
{'tag_id': 404},
|
||||
{'tag_name': 'no tag_id'},
|
||||
]
|
||||
|
||||
|
||||
def test_referenced_tag_ids():
|
||||
"""Int ids only, scouter pipelines only; text, empty or null tag_id ignored"""
|
||||
pipelines = [
|
||||
scouter_pipeline_doc(
|
||||
'sin', 'm1', [{'tag_id': 7}, {'tag_id': '12'}, {'tag_id': ''}, {'tag_id': None}]
|
||||
),
|
||||
scouter_pipeline_doc(
|
||||
'cos', 'm2', [{'tag_id': 'abc'}, {'tag_name': 'a', 'frequency': 1000}]
|
||||
),
|
||||
scouter_pipeline_doc('pred', 'm3', [{'tag_id': 99}], workflow_type='predictions_batch'),
|
||||
]
|
||||
|
||||
# Sorted, so the query is the same on every run
|
||||
assert referenced_tag_ids(pipelines) == [7]
|
||||
|
||||
|
||||
def test_referenced_tag_ids_without_tag_ids():
|
||||
pipeline = scouter_pipeline_doc('sin', 'm1', [{'tag_name': 'a', 'frequency': 1000}])
|
||||
no_read_tags = scouter_pipeline_doc('cos', 'm2', [])
|
||||
del no_read_tags['read_tags']
|
||||
|
||||
assert referenced_tag_ids([pipeline, no_read_tags]) == []
|
||||
|
||||
|
||||
def test_filter_by_tag_ids():
|
||||
"""A tag_id $match goes in front, on a new aggregation list"""
|
||||
query = {'collection': 'connector_tags', 'aggregation': [{'$project': {'_id': 0}}]}
|
||||
|
||||
result = filter_by_tag_ids(query, [7, 8])
|
||||
|
||||
assert result == {
|
||||
'collection': 'connector_tags',
|
||||
'aggregation': [{'$match': {'tag_id': {'$in': [7, 8]}}}, {'$project': {'_id': 0}}],
|
||||
}
|
||||
assert result['aggregation'] is not query['aggregation']
|
||||
assert query['aggregation'] == [{'$project': {'_id': 0}}]
|
||||
|
||||
|
||||
def test_index_skips_incomplete_documents():
|
||||
"""A malformed tag database document is left out instead of failing the run"""
|
||||
tags = [
|
||||
{'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1', 'tag_name': 'a'},
|
||||
{'connector_id': 3, 'address': 'ns=3;i=2', 'tag_name': 'no id'},
|
||||
{'tag_id': 9, 'connector_id': 3, 'address': '', 'tag_name': 'no address'},
|
||||
]
|
||||
connector_list = [
|
||||
{'connector_id': 3, 'name': 'plant', 'url': 'opc.tcp://p:4840', 'security': NO_SECURITY},
|
||||
{'connector_id': 4, 'name': 'no url', 'security': NO_SECURITY},
|
||||
{'name': 'no id', 'url': 'opc.tcp://x:4840', 'security': NO_SECURITY},
|
||||
# security.mode and security.policy are the source of the OPC security (D4)
|
||||
{'connector_id': 5, 'name': 'no security', 'url': 'opc.tcp://x:4840'},
|
||||
{
|
||||
'connector_id': 6,
|
||||
'name': 'no mode',
|
||||
'url': 'opc.tcp://x:4840',
|
||||
'security': {'policy': 'None'},
|
||||
},
|
||||
{
|
||||
'connector_id': 7,
|
||||
'name': 'no policy',
|
||||
'url': 'opc.tcp://x:4840',
|
||||
'security': {'mode': 'None'},
|
||||
},
|
||||
]
|
||||
|
||||
assert list(index_connector_tags(tags)) == [7]
|
||||
assert list(index_connectors(connector_list)) == ['3']
|
||||
|
||||
|
||||
def test_tag_id_of_accepts_only_int():
|
||||
"""connector_tags stores tag_id as int: anything else is no reference"""
|
||||
assert tag_id_of({'tag_id': 7}) == 7
|
||||
assert tag_id_of({'tag_id': '7'}) is None
|
||||
assert tag_id_of({'tag_id': 'abc'}) is None
|
||||
assert tag_id_of({'tag_id': 7.0}) is None
|
||||
assert tag_id_of({'tag_id': True}) is None
|
||||
assert tag_id_of({'tag_id': ''}) is None
|
||||
assert tag_id_of({'tag_id': None}) is None
|
||||
assert tag_id_of({}) is None
|
||||
assert tag_id_of('ns=2;s=raw') is None
|
||||
|
||||
|
||||
def test_referenced_tag_ids_includes_write_tags():
|
||||
pipelines = [
|
||||
scouter_pipeline_doc('sin', 'm1', [{'tag_id': 7}]),
|
||||
batch_pipeline_doc('pred', [{'tag_id': 77}, {'tag_id': 78}, {'server_id': '1'}]),
|
||||
batch_pipeline_doc('xgb', [{'tag_id': 79}], workflow_type='xgboost_predictions_batch'),
|
||||
batch_pipeline_doc('retrain', [{'tag_id': 99}], workflow_type='minimal_retrain'),
|
||||
]
|
||||
|
||||
assert referenced_tag_ids(pipelines) == [7, 77, 78, 79]
|
||||
|
||||
|
||||
def test_index_connector_tags_filters_status_and_category():
|
||||
tags = [
|
||||
write_tag_doc(1, 'ns=2;s=R', category='Read'),
|
||||
write_tag_doc(2, 'ns=2;s=W'),
|
||||
write_tag_doc(3, 'ns=2;s=C', category='Prediction Confidence'),
|
||||
write_tag_doc(4, 'ns=2;s=I', status='Inactive'),
|
||||
# An override that projects neither status nor category
|
||||
{'tag_id': 5, 'connector_id': 'plant', 'address': 'ns=2;s=N', 'tag_name': 'n'},
|
||||
]
|
||||
|
||||
assert list(index_connector_tags(tags)) == [1, 5]
|
||||
assert list(index_connector_tags(tags, WRITE_CATEGORIES)) == [2, 3, 5]
|
||||
# Every document, whatever its status or category
|
||||
assert list(index_tag_documents(tags)) == [1, 2, 3, 4, 5]
|
||||
|
||||
|
||||
def test_index_connectors_leaves_out_inactive():
|
||||
connectors = [
|
||||
{**OPC_CONNECTORS['plant'], 'status': 'Active'},
|
||||
{**OPC_CONNECTORS['plant'], 'connector_id': 'old', 'status': 'Inactive'},
|
||||
]
|
||||
|
||||
assert list(index_connectors(connectors)) == ['plant']
|
||||
|
||||
|
||||
def test_resolve_write_tags():
|
||||
tags = index_connector_tags(
|
||||
[
|
||||
write_tag_doc(77, 'ns=2;s=P1'),
|
||||
write_tag_doc(78, 'ns=2;s=C1', category='Prediction Confidence', data_type='double'),
|
||||
write_tag_doc(79, 'ns=2;s=P2', connector_id='other', data_type='int'),
|
||||
],
|
||||
WRITE_CATEGORIES,
|
||||
)
|
||||
connectors = {**OPC_CONNECTORS, 'other': {**OPC_CONNECTORS['plant'], 'connector_id': 'other'}}
|
||||
|
||||
output_config, issues = resolve_write_tags(
|
||||
[{'tag_id': 77}, {'tag_id': 78}, {'tag_id': 79}], tags, connectors
|
||||
)
|
||||
|
||||
assert output_config == {
|
||||
'plant': {
|
||||
'prediction_tags': {'ns=2;s=P1': {'data_type': 'float'}},
|
||||
'confidence_tags': {'ns=2;s=C1': {'data_type': 'double'}},
|
||||
},
|
||||
'other': {'prediction_tags': {'ns=2;s=P2': {'data_type': 'int'}}},
|
||||
}
|
||||
assert not any(issues.values())
|
||||
|
||||
|
||||
def test_resolve_write_tags_issues():
|
||||
tags = index_connector_tags(
|
||||
[
|
||||
write_tag_doc(1, 'ns=2;s=R', category='Read'),
|
||||
write_tag_doc(2, 'ns=2;s=X', connector_id='gone'),
|
||||
write_tag_doc(3, 'ns=2;s=B', data_type='Double'),
|
||||
write_tag_doc(4, 'ns=2;s=I', status='Inactive'),
|
||||
# No category: a write tag cannot tell prediction from confidence
|
||||
{'tag_id': 5, 'connector_id': 'plant', 'address': 'ns=2;s=N', 'tag_name': 'n'},
|
||||
write_tag_doc(6, 'ns=2;s=OK'),
|
||||
],
|
||||
WRITE_CATEGORIES,
|
||||
)
|
||||
write_tags = [
|
||||
{'tag_id': 1},
|
||||
{'tag_id': 2},
|
||||
{'tag_id': 3},
|
||||
{'tag_id': 4},
|
||||
{'tag_id': 5},
|
||||
{'tag_id': 404},
|
||||
# Every write tag is a tag_id
|
||||
{'tag_name': 'L'},
|
||||
{'tag_id': None},
|
||||
# tag_id is int: a text id is invalid
|
||||
{'tag_id': '6'},
|
||||
'ns=2;s=raw',
|
||||
{'tag_id': 6},
|
||||
]
|
||||
|
||||
output_config, issues = resolve_write_tags(write_tags, tags, OPC_CONNECTORS)
|
||||
|
||||
assert output_config == {'plant': {'prediction_tags': {'ns=2;s=OK': {'data_type': 'float'}}}}
|
||||
assert issues['tags'] == {'1', '4', '5', '404'}
|
||||
assert issues['connectors'] == {'gone'}
|
||||
assert issues['invalid'] == {
|
||||
'3',
|
||||
"{'tag_name': 'L'}",
|
||||
"{'tag_id': '6'}",
|
||||
"{'tag_id': None}",
|
||||
'ns=2;s=raw',
|
||||
}
|
||||
|
||||
|
||||
def test_resolve_certificate():
|
||||
"""Only the fields the gateway needs: dates and other fields stay out of the slot"""
|
||||
connector = {
|
||||
**OPC_CONNECTORS['plant'],
|
||||
**secure_connector_fields(
|
||||
certificates=[certificate_doc(thumbprint_sha1='ab', not_after=None)]
|
||||
),
|
||||
}
|
||||
|
||||
assert resolve_certificate(connector) == CERTIFICATE
|
||||
assert client_certificate_id(connector) == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'certificates',
|
||||
[
|
||||
pytest.param(None, id='no-lookup'),
|
||||
pytest.param([], id='not-found'),
|
||||
pytest.param([certificate_doc(application_uri=None)], id='no-application-uri'),
|
||||
pytest.param([{**certificate_doc(), 'cert': None}], id='no-cert'),
|
||||
pytest.param(
|
||||
[{**certificate_doc(), 'private_key': {'object_name': 'k'}}], id='key-no-bucket'
|
||||
),
|
||||
pytest.param(['not a document'], id='not-a-document'),
|
||||
],
|
||||
)
|
||||
def test_resolve_certificate_without_a_usable_certificate(certificates):
|
||||
connector = {**OPC_CONNECTORS['plant'], **secure_connector_fields()}
|
||||
connector['certificates'] = certificates
|
||||
|
||||
assert resolve_certificate(connector) is None
|
||||
|
||||
|
||||
def test_client_certificate_id_without_one():
|
||||
assert client_certificate_id(OPC_CONNECTORS['plant']) is None
|
||||
assert client_certificate_id({'connector_id': 'x'}) is None
|
||||
assert client_certificate_id({'security': None}) is None
|
||||
@@ -3,7 +3,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
from pytest import mark
|
||||
|
||||
from orchestrator.worker.worker import main
|
||||
from orchestrator.worker.worker import main, start_prometheus_server
|
||||
from orchestrator.workflows.alerts import Alerts
|
||||
from orchestrator.workflows.orchestrator import Orchestrator
|
||||
from orchestrator.workflows.reports import Reports
|
||||
@@ -47,3 +47,43 @@ async def test_main_starts_three_workers_for_expected_workflows(
|
||||
assert main_workflows == [Orchestrator, Alerts, Reports]
|
||||
|
||||
assert worker_mock.run.call_count == 3
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('orchestrator.worker.worker.Runtime')
|
||||
@patch('orchestrator.worker.worker.sys')
|
||||
@patch('orchestrator.worker.worker.start_http_server')
|
||||
@patch('orchestrator.worker.worker.NotificationHandler')
|
||||
@patch('orchestrator.worker.worker.Activities')
|
||||
@patch('orchestrator.worker.worker.prepare_worker')
|
||||
@patch('orchestrator.worker.worker.client.Client.connect', new_callable=AsyncMock)
|
||||
async def test_main_exits_with_error_when_a_worker_fails(
|
||||
_connect_mock,
|
||||
prepare_worker_mock,
|
||||
activities_mock,
|
||||
notification_handler_mock,
|
||||
_start_http_server,
|
||||
sys_mock,
|
||||
_runtime_mock,
|
||||
):
|
||||
"""A worker failure shuts everything down and exits with code 1."""
|
||||
activities_instance = activities_mock.return_value
|
||||
activities_instance.connect_to_temporal = AsyncMock()
|
||||
|
||||
worker_mock = MagicMock()
|
||||
worker_mock.run = AsyncMock(side_effect=RuntimeError('worker crashed'))
|
||||
prepare_worker_mock.return_value = worker_mock
|
||||
|
||||
await main()
|
||||
|
||||
notification_handler_mock.return_value.shutdown.assert_called_once()
|
||||
activities_instance.shutdown.assert_called_once()
|
||||
sys_mock.exit.assert_called_once_with(1)
|
||||
|
||||
|
||||
@patch('orchestrator.worker.worker.os._exit')
|
||||
@patch('orchestrator.worker.worker.start_http_server', side_effect=OSError('port in use'))
|
||||
def test_start_prometheus_server_exits_on_failure(_start_http_server, exit_mock):
|
||||
start_prometheus_server()
|
||||
|
||||
exit_mock.assert_called_once_with(1)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -3,7 +3,11 @@ from unittest.mock import ANY, AsyncMock, call, patch
|
||||
from pytest import fixture, mark
|
||||
|
||||
from orchestrator.activities.activities import Activities
|
||||
from orchestrator.workflows.orchestrator import Orchestrator
|
||||
from orchestrator.workflows.orchestrator import (
|
||||
DEFAULT_CONNECTOR_TAGS_QUERY,
|
||||
DEFAULT_CONNECTORS_QUERY,
|
||||
Orchestrator,
|
||||
)
|
||||
|
||||
|
||||
@fixture
|
||||
@@ -22,11 +26,11 @@ metadata = {
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[7, '7'])
|
||||
@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
|
||||
async def test_run(workflow_mock, orchestrator):
|
||||
async def test_run(workflow_mock, referenced_tag_ids_mock, orchestrator):
|
||||
input_data = {
|
||||
'pipelines_query': 'SELECT * FROM bucket',
|
||||
'opc_servers_query': 'SELECT * FROM servers',
|
||||
'schedule_name': 'test-schedule-name',
|
||||
}
|
||||
|
||||
@@ -47,16 +51,13 @@ async def test_run(workflow_mock, orchestrator):
|
||||
]
|
||||
)
|
||||
|
||||
workflow_mock.start_local_activity_method.assert_has_calls(
|
||||
[
|
||||
call(
|
||||
Activities.find_documents_in_mongodb,
|
||||
{**metadata, 'query': input_data['opc_servers_query']},
|
||||
retry_policy=ANY,
|
||||
start_to_close_timeout=ANY,
|
||||
)
|
||||
]
|
||||
)
|
||||
# connectors is the only source of the OPC servers: opc-servers is never loaded
|
||||
assert not [
|
||||
c
|
||||
for c in workflow_mock.start_local_activity_method.call_args_list
|
||||
if c.args[0] is Activities.find_documents_in_mongodb
|
||||
and c.args[1]['query'] != {'collection': 'orchestrated_schedules'}
|
||||
]
|
||||
|
||||
workflow_mock.start_local_activity_method.assert_has_calls(
|
||||
[
|
||||
@@ -109,11 +110,54 @@ async def test_run(workflow_mock, orchestrator):
|
||||
]
|
||||
)
|
||||
|
||||
referenced_tag_ids_mock.assert_called_once_with(
|
||||
workflow_mock.start_local_activity_method.return_value
|
||||
)
|
||||
|
||||
workflow_mock.start_local_activity_method.assert_has_calls(
|
||||
[
|
||||
call(
|
||||
Activities.aggregate_documents_in_mongodb,
|
||||
{**metadata, 'query': DEFAULT_CONNECTORS_QUERY},
|
||||
retry_policy=ANY,
|
||||
start_to_close_timeout=ANY,
|
||||
),
|
||||
call(
|
||||
Activities.aggregate_documents_in_mongodb,
|
||||
{
|
||||
**metadata,
|
||||
'query': {
|
||||
**DEFAULT_CONNECTOR_TAGS_QUERY,
|
||||
'aggregation': [
|
||||
{'$match': {'tag_id': {'$in': [7, '7']}}},
|
||||
*DEFAULT_CONNECTOR_TAGS_QUERY['aggregation'],
|
||||
],
|
||||
},
|
||||
},
|
||||
retry_policy=ANY,
|
||||
start_to_close_timeout=ANY,
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
# The activity appends a stage to the aggregation it gets: never the module default
|
||||
tag_database_calls = _tag_database_calls(workflow_mock)
|
||||
assert len(tag_database_calls) == 2
|
||||
for c in tag_database_calls:
|
||||
aggregation = c.args[1]['query']['aggregation']
|
||||
assert aggregation is not DEFAULT_CONNECTORS_QUERY['aggregation']
|
||||
assert aggregation is not DEFAULT_CONNECTOR_TAGS_QUERY['aggregation']
|
||||
|
||||
workflow_mock.start_local_activity_method.assert_has_calls(
|
||||
[
|
||||
call(
|
||||
Activities.process_schedules,
|
||||
{**metadata, 'pipelines': workflow_mock.start_local_activity_method.return_value},
|
||||
{
|
||||
**metadata,
|
||||
'pipelines': workflow_mock.start_local_activity_method.return_value,
|
||||
'connectors': workflow_mock.start_local_activity_method.return_value,
|
||||
'connector_tags': workflow_mock.start_local_activity_method.return_value,
|
||||
},
|
||||
retry_policy=ANY,
|
||||
start_to_close_timeout=ANY,
|
||||
)
|
||||
@@ -126,7 +170,8 @@ async def test_run(workflow_mock, orchestrator):
|
||||
Activities.process_slots,
|
||||
{
|
||||
**metadata,
|
||||
'opc_servers': workflow_mock.start_local_activity_method.return_value,
|
||||
'connectors': workflow_mock.start_local_activity_method.return_value,
|
||||
'connector_tags': workflow_mock.start_local_activity_method.return_value,
|
||||
'active_ingestors': workflow_mock.start_local_activity_method.return_value,
|
||||
'pipelines': workflow_mock.start_local_activity_method.return_value,
|
||||
},
|
||||
@@ -346,3 +391,136 @@ async def test_run(workflow_mock, orchestrator):
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _tag_database_calls(workflow_mock):
|
||||
return [
|
||||
c
|
||||
for c in workflow_mock.start_local_activity_method.call_args_list
|
||||
if c.args[0] is Activities.aggregate_documents_in_mongodb
|
||||
and isinstance(c.args[1]['query'], dict)
|
||||
and c.args[1]['query'].get('collection') in ('connectors', 'connector_tags')
|
||||
]
|
||||
|
||||
|
||||
def _process_slots_input(workflow_mock):
|
||||
return next(
|
||||
c.args[1]
|
||||
for c in workflow_mock.start_local_activity_method.call_args_list
|
||||
if c.args[0] is Activities.process_slots
|
||||
)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[7])
|
||||
@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
|
||||
async def test_run_with_tag_database_queries_from_input(workflow_mock, _ids, orchestrator):
|
||||
"""The input overrides the default tag database aggregations (SIENTIAPDE-1974, D9)"""
|
||||
connectors_query = {'collection': 'connectors', 'aggregation': [{'$match': {'type': 'opc_ua'}}]}
|
||||
connector_tags_query = {
|
||||
'collection': 'connector_tags',
|
||||
'aggregation': [{'$match': {'category': 'Read'}}],
|
||||
}
|
||||
input_data = {
|
||||
'pipelines_query': 'SELECT * FROM bucket',
|
||||
'connectors_query': connectors_query,
|
||||
'connector_tags_query': connector_tags_query,
|
||||
'schedule_name': 'test-schedule-name',
|
||||
}
|
||||
|
||||
await orchestrator.run(input_data)
|
||||
|
||||
queries = [c.args[1]['query'] for c in _tag_database_calls(workflow_mock)]
|
||||
assert queries == [
|
||||
connectors_query,
|
||||
{
|
||||
'collection': 'connector_tags',
|
||||
'aggregation': [
|
||||
{'$match': {'tag_id': {'$in': [7]}}},
|
||||
{'$match': {'category': 'Read'}},
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[])
|
||||
@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
|
||||
async def test_run_without_tag_ids_skips_tag_database(workflow_mock, _ids, orchestrator):
|
||||
"""No read tag with tag_id: the tag database is not queried at all (P4)"""
|
||||
input_data = {
|
||||
'pipelines_query': 'SELECT * FROM bucket',
|
||||
'schedule_name': 'test-schedule-name',
|
||||
}
|
||||
|
||||
await orchestrator.run(input_data)
|
||||
|
||||
assert _tag_database_calls(workflow_mock) == []
|
||||
slots_input = _process_slots_input(workflow_mock)
|
||||
assert slots_input['connectors'] == []
|
||||
assert slots_input['connector_tags'] == []
|
||||
|
||||
|
||||
def _process_schedules_input(workflow_mock):
|
||||
return next(
|
||||
c.args[1]
|
||||
for c in workflow_mock.start_local_activity_method.call_args_list
|
||||
if c.args[0] is Activities.process_schedules
|
||||
)
|
||||
|
||||
|
||||
def test_default_connector_tags_query_loads_every_category_and_status():
|
||||
"""
|
||||
The status / category filters are applied in the code. updated_at is not loaded: a
|
||||
tag change reaches the schedules through their config_hash.
|
||||
"""
|
||||
stages = DEFAULT_CONNECTOR_TAGS_QUERY['aggregation']
|
||||
assert all('$match' not in stage for stage in stages)
|
||||
projected = stages[-1]['$project']
|
||||
for field in ('category', 'data_type', 'status'):
|
||||
assert projected[field] == 1
|
||||
assert 'updated_at' not in projected
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[77])
|
||||
@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
|
||||
async def test_run_hands_the_tag_database_to_process_schedules(workflow_mock, _ids, orchestrator):
|
||||
"""The write tags of process_schedules need the connectors as well as the tags"""
|
||||
input_data = {
|
||||
'pipelines_query': 'SELECT * FROM bucket',
|
||||
'schedule_name': 'test-schedule-name',
|
||||
}
|
||||
|
||||
await orchestrator.run(input_data)
|
||||
|
||||
loaded = workflow_mock.start_local_activity_method.return_value
|
||||
schedules_input = _process_schedules_input(workflow_mock)
|
||||
assert schedules_input['connectors'] is loaded
|
||||
assert schedules_input['connector_tags'] is loaded
|
||||
assert 'pi_connectors' not in schedules_input
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[])
|
||||
@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
|
||||
async def test_run_without_changes_skips_reports_and_timestamps(workflow_mock, _ids, orchestrator):
|
||||
"""Nothing created, updated or deleted: no report and no pipeline timestamp activity"""
|
||||
workflow_mock.start_activity_method = AsyncMock(return_value=[])
|
||||
input_data = {
|
||||
'pipelines_query': 'SELECT * FROM bucket',
|
||||
'schedule_name': 'test-schedule-name',
|
||||
}
|
||||
|
||||
await orchestrator.run(input_data)
|
||||
|
||||
started = [c.args[0] for c in workflow_mock.start_activity_method.call_args_list]
|
||||
assert started == [
|
||||
Activities.normalize_schedules,
|
||||
Activities.create_collection_with_ttl_index,
|
||||
Activities.delete_slots,
|
||||
Activities.update_slots,
|
||||
Activities.delete_schedules,
|
||||
Activities.create_schedules,
|
||||
Activities.update_schedules,
|
||||
]
|
||||
|
||||
27
tests/orchestrator/workflows/test_orchestrator_replay.py
Normal file
27
tests/orchestrator/workflows/test_orchestrator_replay.py
Normal file
@@ -0,0 +1,27 @@
|
||||
"""
|
||||
Replay a real Orchestrator history against the current workflow code.
|
||||
|
||||
orchestrator_tag_database.json was recorded on the e2e stack (real Temporal, Mongo and
|
||||
Redis), with the connectors / connector_tags loads of read and write tags by tag_id.
|
||||
|
||||
A NondeterminismError here means a change to the workflow breaks runs in flight.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from temporalio.client import WorkflowHistory
|
||||
from temporalio.worker import Replayer
|
||||
|
||||
from orchestrator.workflows.orchestrator import Orchestrator
|
||||
|
||||
HISTORIES = Path(__file__).parent / 'histories'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_replay_recorded_history():
|
||||
history = WorkflowHistory.from_json(
|
||||
'orchestrator-history', (HISTORIES / 'orchestrator_tag_database.json').read_text()
|
||||
)
|
||||
|
||||
await Replayer(workflows=[Orchestrator]).replay_workflow(history)
|
||||
Reference in New Issue
Block a user