diff --git a/CLAUDE.md b/CLAUDE.md
index cfa191c..4b6fb8c 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -47,7 +47,8 @@ CI quality gate (`.github/workflows/quality-gate.yml`) calls the shared reusable
- **`orchestrator/worker/worker.py`** — entrypoint (`python -m orchestrator.worker.worker`). Builds a single `Activities` instance, connects to Temporal, starts Prometheus (`HTTP_METRICS_PORT`) and Temporal SDK metrics (`HTTP_SDK_METRICS_PORT`) servers, then runs three `Worker`s concurrently via `asyncio.gather`, each on its own task queue: `orchestrator-queue`, `alerts-queue`, `reports-queue`. `prepare_worker.py` derives the queue name from the workflow class name (`camel_to_snake`) and reads per-queue concurrency/poller tuning from env vars prefixed with the workflow name (e.g. `ORCHESTRATOR_MAX_CONCURRENT_ACTIVITIES`).
- **`orchestrator/workflows/`** — Temporal workflow definitions: `orchestrator.py` (pipeline/slot/schedule orchestration), `alerts.py` (real-time ERROR notifications), `reports.py` (scheduled all-level reports). `alerts.py` and `reports.py` both delegate to the same two subworkflows in `workflows/subworkflows/`: `load_notification_package.py` (load notifications + receiver configs from Mongo, incremental via Redis timestamp) and `process_notifications.py` (build HTML email, send, log to Postgres). The difference between alerts and reports is which filter activity they call (`filter_notification_alerts` vs `filter_notification_reports`) and their base data filter (`level: ERROR` by default vs all levels) — `load_notification_package.py`'s `_compute_query_time_filter` can widen either beyond that default to the union of active receiver groups' own `levels` (SIENTIAPDE-2011); the ERROR-only default only holds when no active group narrows further. Receiver groups also support a `model_ids` allow-list (SIENTIAPDE-2149), independent of the pre-existing `ignore_models_id` deny-list — both apply together (AND) when a group configures both, and `model_ids` follows the same union-at-query-time / precise-per-group-slice split as `levels`. `load_latest_data`'s Mongo query is capped at `NOTIFICATION_QUEUE_BATCH_SIZE` documents per run (default `0`, meaning unlimited — unchanged from pre-SIENTIAPDE-2013 behavior unless a deployment opts in), sorted by `timestamp` ascending so a capped run always returns the oldest pending batch first (SIENTIAPDE-2013) — a hard cap with no drain loop, since the incremental Redis cursor picks up any remainder on the next run.
- **`orchestrator/activities/activities.py`** — `Activities` is a single class combining every activity mixin via multiple inheritance: `TemporalManager, SlotManager, Formatters, MongoDB, Email, Postgres` (Postgres comes from `sientia_do.temporal.activities.postgres`, the external `sientia-dataops-library` dependency). One instance is constructed in `worker.py` and its bound methods are handed to each `Worker` as the activity list — activities are plain methods on this shared object, not separate classes. Each mixin's `__init__` takes its own config dict plus shared `logger`, `notification_handler`, `metrics_controller`. `Activities.shutdown()` closes every mixin's connections.
-- **`orchestrator/utils/`** — `connectors_config.py` builds each config dict from env vars (Temporal/Redis/MongoDB/Postgres/Email); `email_builder.py` renders Jinja2 templates (`utils/templates/email_template.html`, `general_template.html`); `orchestrator_functions.py` transforms pipeline documents into pipeline-type-specific configs (`scouter`, `pi_web_api_scouter`, `predictions_batch`, `minimal_retrain`, `drift`, `simple_metrics`); `converters.py` handles Temporal schedule frequency parsing.
+- **`orchestrator/utils/`** — `connectors_config.py` builds each config dict from env vars (Temporal/Redis/MongoDB/Postgres/Email); `email_builder.py` renders Jinja2 templates (`utils/templates/email_template.html`, `general_template.html`); `pipeline_configs.py` transforms pipeline documents into pipeline-type-specific configs (`scouter`, `pi_web_api_scouter`, `predictions_batch`, `minimal_retrain`, `drift`, `simple_metrics`); `tag_database.py` indexes the tag database (`connectors` + `connector_tags`) and resolves tags by `tag_id` (scouter read tag names and the `predictions_batch` write tags → `opc_output_config` keyed by `connector_id`, via `resolve_write_tags`; tag changes reach the schedules through `config_hash`, see below); `opc_slots.py` builds the OPC slot v2 the gateway reads (`gather_read_tags` / `build_tag_config`, contract C1: one entry per address with a `names[]` list, and per server `security` {mode, policy} plus the `certificate` from `tag_database.resolve_certificate`; see the README section "OPC slots"). The certificate comes from `opc_certificates` through the `$lookup` in `DEFAULT_CONNECTORS_QUERY` (only `active` + `client`); no certificate means the gateway connects without security, and a `client_certificate_id` without one is a `WARNING` (SIENTIAPDE-1974, D5). `resolve_tag_names` (in `tag_database.py`) is the single source of tag names for both the scouter's `model_tags` and the slot's `names` (SIENTIAPDE-1974 SC1); `converters.py` handles Temporal schedule frequency parsing.
+- **Schedule redeploys** — `orchestrated_schedules` stores, per schedule, the last deploy time (`updated_at`) and the `config_hash` it was deployed with (`formatters.schedule_config_hash`: SHA-256 of the built schedule without `updated_at`). `compare_config_timestamps` redeploys when the pipeline's `updated_at` is newer (manual force) or the hash differs (any content change, including tag database edits/deletes made without bumping `updated_at`). `create_schedules`/`update_schedules` pop `config_hash` before sending the schedule to Temporal and return it in their report for `create_pipelines_timestamps`/`update_pipelines_timestamps`.
### Temporal-specific conventions (all workflow/activity files follow this)
- Every workflow and activity-consuming module wraps non-deterministic/side-effecting imports in `with workflow.unsafe.imports_passed_through(): ...` right after `from temporalio import workflow`, because Temporal's sandbox restricts what workflow code can import directly. When adding a new workflow file, follow the existing pattern (see `orchestrator/workflows/orchestrator.py`, `orchestrator/activities/activities.py`) rather than importing at module top level.
diff --git a/README.md b/README.md
index a0d7666..c4f2170 100644
--- a/README.md
+++ b/README.md
@@ -97,13 +97,13 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
- **Infrastructure Management**: Creates, updates, and deletes workflow schedules
**Execution Flow**:
-1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, and active ingestors
+1. **Parallel Data Loading**: Concurrently loads pipelines, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`, the only source of the OPC servers and tags of the slot and the schedules)
2. **Parallel Processing**: Formats orchestrated schedules and processes new schedules and slots
3. **Parallel Config Creation**: Creates schedule and slot action configurations, normalizes schedules, and creates collections with TTL indexes
4. **Parallel Operations**: Executes slot deletions, slot updates, schedule deletions, schedule creations, and schedule updates concurrently
5. **Parallel Reports & Timestamps**: Generates orchestration reports and updates MongoDB timestamps for created/updated/deleted pipelines
-**Input Parameters**:
+**Input Parameters** (minimal: each pipeline must already carry `model.name`; see "Workflow Configuration" for the `$lookup` on `models` that adds it):
```json
{
"schedule_name": "hourly_orchestration",
@@ -113,21 +113,153 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
{"$match": {"active": true}},
{"$sort": {"updated_at": -1}}
]
- },
- "opc_servers_query": {
- "collection": "opc_servers",
- "filters": {"active": true}
}
}
```
+Optional: `connectors_query` and `connector_tags_query` override the tag database aggregations. The defaults live in the code (`orchestrator/workflows/orchestrator.py`), so the schedule input does not need them:
+
+```json
+{
+ "connectors_query": {
+ "collection": "connectors",
+ "aggregation": [
+ {"$match": {"type": "opc_ua", "status": "Active"}},
+ {"$lookup": {"from": "opc_certificates",
+ "let": {"certificate_id": "$security.client_certificate_id"},
+ "pipeline": [
+ {"$match": {"$expr": {"$eq": ["$certificate_id", "$$certificate_id"]},
+ "status": "active", "kind": "client"}},
+ {"$project": {"_id": 0, "certificate_id": 1, "application_uri": 1,
+ "cert.bucket": 1, "cert.object_name": 1,
+ "private_key.bucket": 1, "private_key.object_name": 1}}
+ ],
+ "as": "certificates"}},
+ {"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "subscription_period_ms": 1,
+ "security.mode": 1, "security.policy": 1,
+ "security.client_certificate_id": 1, "certificates": 1}}
+ ]
+ },
+ "connector_tags_query": {
+ "collection": "connector_tags",
+ "aggregation": [
+ {"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1,
+ "category": 1, "data_type": 1, "status": 1}}
+ ]
+ }
+}
+```
+
+The workflow puts `{"$match": {"tag_id": {"$in": [...]}}}` in front of `connector_tags_query`, so it loads only the tags the scouter read tags and `predictions_batch` write tags reference. When no pipeline references a `tag_id`, the tag database is not queried. An override should keep a `$project` of these fields: any other attribute may hold a BSON type a Temporal payload cannot carry (such as `ObjectId`).
+
+The referenced tags are loaded whatever their category and status: the code uses only the active ones (`Read` for the slot and the scouter names, `Write` / `Prediction Confidence` for the write tags), and a change to a referenced tag that reaches a schedule redeploys it through its `config_hash` (see "Redeploy on changes" below). A document without `status` counts as active.
+
+#### OPC slots (slot v2, contract C1)
+
+`process_slots` writes one slot per active ingestor to Redis (`slot:opc_tags:{n}`). The OPC Gateway reads it (SIENTIAPDE-1974). Each server carries its connection fields, its security and the MinIO objects of its client certificate, and each tag address carries the list of names it is written under:
+
+```json
+{
+ "kepserver": {
+ "server_id": "1", "name": "kepserver", "url": "opc.tcp://host:49320",
+ "subscription_period_ms": 7500.0,
+ "security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256"},
+ "certificate": {
+ "certificate_id": 1, "application_uri": "urn:sientia:opc-ingestor",
+ "cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-cert.pem"},
+ "private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-key.pem"}
+ },
+ "tags": {
+ "ns=2;s=Channel1.Device1.Tag1": {
+ "frequency": 15000,
+ "active_read": {"enabled": false},
+ "names": [
+ {"collection": "raw_model_a", "tag_name": "Counter", "model_id": "1", "model_name": "model a"},
+ {"collection": "raw_model_b", "tag_name": "Contador", "model_id": "2", "model_name": "model b"}
+ ]
+ }
+ }
+ }
+}
+```
+
+Each scouter `read_tags` entry references its tag by `tag_id` (`{"tag_id": 7, "frequency": 15000}`). `connectors` is the only source of the OPC servers:
+
+| Field | Source |
+|---|---|
+| Address | `connector_tags.address` |
+| Server (`server_id`, `name`, `url`, `subscription_period_ms`, `security.mode`, `security.policy`) | The tag's `connector_tags.connector_id` in `connectors` (`server_id` = `connector_id`) |
+| `certificate` | The `opc_certificates` document of the connector's `security.client_certificate_id`, when it is `active` and of kind `client` (`connectors_query`'s `$lookup`) |
+| Name | The read tag's `tag_name`, else the `connector_tags` one |
+
+Rules:
+- **One entry per address.** An address read by N pipelines is read once and written to N collections: one `names` item per pipeline (`collection` = `raw_{schedule_name}`).
+- **Same name source as the scouter.** `resolve_tag_names` names both the slot's `names` and the scouter's `model_tags`, so the gateway always writes the name the scouter looks for.
+- **`frequency`:** the lowest among the pipelines of the address, as an integer. Every read tag needs one.
+- **`tag_id`:** required, an integer like in `connector_tags`. A read tag whose `tag_id` is missing, `null` or not an integer (`"7"` included) is left out and notified as invalid.
+- **`subscription_period_ms`:** the connector's value. A connector without one gets half its lowest tag frequency in the slot.
+- **`security`:** the connector's `mode` and `policy`, the source of the OPC security (`use_security` and `server_uri` no longer exist). A connector without them is left out like an incomplete one.
+- **`certificate`:** `certificate_id`, `application_uri` (the SAN URI of the certificate, used as the client application URI) and the `bucket`/`object_name` of `cert` and `private_key`; the gateway downloads them from MinIO. The certificate dates, `sha256` and audit fields stay out of the slot. There is no default certificate: `null` means the gateway connects without security, and a server that requires a certificate refuses the connection. A connector whose `client_certificate_id` is set but has no active `client` certificate (missing id, `retired`, another kind or an incomplete document) still goes into the slot with `certificate: null`, and is reported (see below).
+- **`active_read`:** off by default. It is on when any pipeline sets `read_tags[].active_read.enabled`, with the lowest `cycle_multiple`.
+- **Server keys:** two different connectors with the same name get distinct slot keys (`name`, then `name:server_id`).
+- **Slot division:** the addresses are split across the slots, sorted by server and address, and the names of an address always stay in the same slot.
+- **Unresolved tags:** a read tag that cannot be resolved is left out of the slot, and the rest of the slot is still written. It is notified once per run:
+ - `ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the `tag_id` is not an active Read tag, or its document lacks `connector_id`, `address` or `tag_name`;
+ - `ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the connector is missing or inactive, or lacks `name`, `url`, `security.mode` or `security.policy`;
+ - `ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION`: the read tag has no integer `tag_id`, no integer `frequency`, or an enabled `active_read` without an integer `cycle_multiple`; or its pipeline lacks `schedule_name`, `model_id` or `model.name` (the value is then the `schedule_name`).
+ - `ORCHESTRATOR_CERTIFICATE_NOT_FOUND_DURING_SLOT_CONFIGURATION` (`WARNING`): the connector's `client_certificate_id` has no active `client` certificate, so its tags go into the slot without a certificate. Most likely a registration error.
+- **Only changed slots are written.** The slot is built in a stable order, so unchanged pipelines produce an identical slot, and the gateway does not re-subscribe.
+
+#### Write tags (predictions_batch, through the tag database)
+
+A `predictions_batch` (or `xgboost_predictions_batch`) references its write tags by `tag_id` only, exactly like the read tags. Everything else comes from the tag database, so the pipeline does not say which server, address or type a tag has:
+
+```json
+{"workflow_type": "predictions_batch", "write_tags": [{"tag_id": 101}, {"tag_id": 102}]}
+```
+
+`process_schedules` turns them into the `opc_output_config` the Laborious already reads:
+
+```json
+{"plant": {
+ "prediction_tags": {"ns=3;i=2001": {"data_type": "double"}},
+ "confidence_tags": {"ns=3;i=2002": {"data_type": "float"}}}}
+```
+
+| Field | Source |
+|---|---|
+| Server key | `connector_tags.connector_id`. The Laborious publishes it as `server_id` on NATS, and the OPC Gateway resolves it in `connectors` |
+| `prediction_tags` / `confidence_tags` | `connector_tags.category`: `Write` / `Prediction Confidence` |
+| Address | `connector_tags.address` |
+| `data_type` | `connector_tags.data_type` (`float`, `double`, `real`, `int`, `integer`, `bool`, `boolean`, `str`, `string`), else `float` |
+
+A write tag that cannot be resolved is left out, the rest of the pipeline is still scheduled, and it is notified once per run (values are `schedule_name/tag_id`):
+- `ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the `tag_id` is not an active `Write` / `Prediction Confidence` tag, or its document lacks `connector_id`, `address`, `tag_name` or `category`;
+- `ORCHESTRATOR_WRITE_TAG_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the tag's connector is missing, inactive or incomplete in `connectors`;
+- `ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION`: the entry has no integer `tag_id`, or the tag's `data_type` is not one the gateway accepts.
+
+#### Redeploy on changes (`updated_at` and `config_hash`)
+
+`orchestrated_schedules` keeps, per schedule, when it was last deployed (`updated_at`) and the `config_hash` it was deployed with: a SHA-256 of the schedule as the Orchestrator builds it (the workflow input, frequency and offset, with tags and connectors already resolved), without `updated_at`. A schedule is redeployed when either:
+
+- the pipeline's `updated_at` is newer than its last deploy. Bumping `updated_at` by hand forces a redeploy even when nothing else changed; or
+- its `config_hash` differs from the stored one. Any change that reaches the schedule redeploys it, whether or not someone bumped an `updated_at`: a pipeline edited by hand, a tag renamed, moved, retyped, deactivated or deleted in `connector_tags`, a connector changed, or a new Orchestrator version that builds the schedule differently.
+
+A change that does not reach the schedule (e.g. a tag's `description`) does not redeploy. The slot is compared by content and follows on its own. A pipeline without `updated_at` is still redeployed on every run. Schedules deployed before `config_hash` existed have none stored, so the first run of this version redeploys all of them once and records it.
+
+#### Deploying
+
+1. Run the migrations that create the tag database and add `data_type` to the `connector_tags` validator (`sientia-dataops-database-migrations`).
+2. Every scouter `read_tags` and every `predictions_batch` / `xgboost_predictions_batch` `write_tags` entry is a `tag_id` of `connector_tags`, as an integer. Anything else is left out and notified.
+3. Deploy this version. Its first run redeploys every schedule once, to record their `config_hash` (see above).
+
#### Architecture
```mermaid
flowchart LR
subgraph "Parallel Data Loading"
A[1. aggregate_documents_in_mongodb]
- B[2. find_documents_in_mongodb
OPC Servers]
+ B[2. aggregate_documents_in_mongodb
connectors + connector_tags]
C[3. find_documents_in_mongodb
Orchestrated Schedules]
D[4. load_opc_slots]
E[5. load_active_ingestors]
@@ -556,6 +688,112 @@ python -m orchestrator.worker.worker
| `LOG_LEVEL` | Application logging level | `INFO` | No |
| `KAFKA_BOOTSTRAP_SERVERS` | Kafka bootstrap servers | - | No |
+### MongoDB Collections
+
+| Collection | Read / written by | Purpose |
+|---|---|---|
+| `pipelines` | Orchestrator (`pipelines_query`) | One document per Temporal schedule (`schedule_name`, `workflow_type`, `model_id`, `active`, `frequency`, ...) |
+| `models` | Orchestrator (`$lookup` in `pipelines_query`) | Gives each pipeline its `model` (`name`, `active`, `model_config`) |
+| `connectors` | Orchestrator (`connectors_query`) | Tag database: the OPC UA servers (`connector_id`, `type`, `name`, `url`, `security`, `subscription_period_ms`, `status`) |
+| `connector_tags` | Orchestrator (`connector_tags_query`) | Tag database: the tags (`tag_id`, `connector_id`, `address`, `tag_name`, `category`, `data_type`, `status`) |
+| `opc_certificates` | Orchestrator (`$lookup` in `connectors_query`) | The client certificate of each connector (`security.client_certificate_id` → `certificate_id`), only when `active` and of kind `client`: `application_uri` and the MinIO objects of `cert` and `private_key` go into the slot |
+| `orchestrated_schedules` | Orchestrator (read and written) | Per deployed schedule: `schedule_name`, `namespace`, `updated_at` (last deploy) and `config_hash` |
+| `raw_{schedule_name}` | Orchestrator (created) | One per scouter `topic`, where the OPC Gateway writes its tags. Created with a TTL index on `inserted_at` (`MONGODB_TTL_INDEX_HOURS`) |
+| `notification_queue` | Alerts / Reports (read) | Notifications (`notification_id`, `trigger`, `level`, `model_id`, `timestamp`, ...), loaded incrementally by `timestamp` |
+| `receiver_groups` | Alerts / Reports (read, `active: true`) | Who receives what (see below) |
+
+#### Tags in a pipeline
+
+Pipelines reference tags only by `tag_id` (an integer, as in `connector_tags`). Servers, addresses and data types come from the tag database, never from the pipeline.
+
+Scouter (`scouter` namespace), `workflow_type: "scouter"`:
+
+```json
+{
+ "schedule_name": "scouter-linreg",
+ "workflow_type": "scouter",
+ "model_id": "1",
+ "active": true,
+ "frequency": "1m",
+ "read_tags": [
+ {
+ "tag_id": 7,
+ "frequency": 15000,
+ "tag_name": "Pressure",
+ "aggr_func": "avg",
+ "data_range": [0, 100],
+ "active_read": {"enabled": true, "cycle_multiple": 2}
+ }
+ ]
+}
+```
+
+| Field | Required | Meaning |
+|---|---|---|
+| `tag_id` | Yes | An active `Read` tag of `connector_tags` |
+| `frequency` | Yes | Read period in ms, an integer. The slot uses the lowest among the pipelines of the address |
+| `tag_name` | No | Overrides the `connector_tags` name: the name the gateway writes and the scouter looks for |
+| `aggr_func` | No | Default `lts` |
+| `data_range` | No | Default `[-100, 100]` |
+| `active_read` | No | Off by default. When `enabled`, `cycle_multiple` must be an integer (default `1`) |
+
+Laborious (`laborious` namespace): only `predictions_batch` and `xgboost_predictions_batch` reference tags; `minimal_retrain`, `drift` and `simple_metrics` have none.
+
+```json
+{
+ "schedule_name": "pred-linreg",
+ "workflow_type": "predictions_batch",
+ "model_id": "1",
+ "active": true,
+ "frequency": "1m",
+ "query": "SELECT ...",
+ "write_tags": [{"tag_id": 101}, {"tag_id": 102}]
+}
+```
+
+Each `write_tags` entry is only a `tag_id`, of an active `Write` (prediction) or `Prediction Confidence` tag. The Orchestrator builds `opc_output_config` from them (see "Write tags" above) and overwrites any `opc_output_config` stored in the pipeline.
+
+The `connector_tags` documents those pipelines reference:
+
+```json
+{"tag_id": 7, "connector_id": "plant", "address": "ns=3;i=1001", "tag_name": "Pressure", "category": "Read", "status": "Active"}
+{"tag_id": 101, "connector_id": "plant", "address": "ns=3;i=2001", "tag_name": "PRED-101", "category": "Write", "data_type": "double", "status": "Active"}
+{"tag_id": 102, "connector_id": "plant", "address": "ns=3;i=2002", "tag_name": "CONF-101", "category": "Prediction Confidence", "status": "Active"}
+```
+
+`pi_web_api_scouter` (`scouter` namespace) does not use the tag database: its `read_tags` is an object keyed by tag name, and it is not part of the OPC slot:
+
+```json
+{
+ "workflow_type": "pi_web_api_scouter",
+ "read_tags": {"Pressure": {"webid": "F1DP...", "aggr_func": "lts", "data_range": [0, 100]}},
+ "pi_web_api_config": {"endpoint": "https://pi/piwebapi", "period": "*-1d", "max_count": 1, "api_timeout": 30}
+}
+```
+
+#### Receiver groups
+
+```json
+{
+ "group_name": "admin_team",
+ "active": true,
+ "members": ["admin@example.com"],
+ "contents": ["core_alerts", "persistent_alerts", "reports"],
+ "levels": ["ERROR", "WARNING"],
+ "model_ids": ["1"],
+ "ignore_models_id": ["2"],
+ "ignore": ["SOME_NOTIFICATION_ID"]
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `contents` | What the group gets: `core_alerts` / `persistent_alerts` (Alerts workflow), `reports` (Reports workflow) |
+| `levels` | Optional allow-list of notification levels. Absent = every level the workflow loads |
+| `model_ids` | Optional allow-list of `model_id`s |
+| `ignore_models_id` | Optional deny-list of `model_id`s (applies together with `model_ids`) |
+| `ignore` | Optional deny-list of `notification_id`s |
+
### Workflow Configuration
Temporal input configuration sample:
@@ -583,7 +821,7 @@ Temporal input configuration sample:
},
{
"$addFields": {
- "models": {
+ "model": {
"$arrayElemAt": [
"$model_docs",
0
@@ -593,7 +831,7 @@ Temporal input configuration sample:
},
{
"$match": {
- "models.active": True
+ "model.active": True
}
},
{
@@ -603,15 +841,39 @@ Temporal input configuration sample:
}
]
},
- "opc_servers_query": {
- "collection": "opc-servers",
- "filters": {
-
- }
+ "connectors_query": {
+ "collection": "connectors",
+ "aggregation": [
+ {"$match": {"type": "opc_ua", "status": "Active"}},
+ {"$lookup": {"from": "opc_certificates",
+ "let": {"certificate_id": "$security.client_certificate_id"},
+ "pipeline": [
+ {"$match": {"$expr": {"$eq": ["$certificate_id", "$$certificate_id"]},
+ "status": "active", "kind": "client"}},
+ {"$project": {"_id": 0, "certificate_id": 1, "application_uri": 1,
+ "cert.bucket": 1, "cert.object_name": 1,
+ "private_key.bucket": 1, "private_key.object_name": 1}}
+ ],
+ "as": "certificates"}},
+ {"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "subscription_period_ms": 1,
+ "security.mode": 1, "security.policy": 1,
+ "security.client_certificate_id": 1, "certificates": 1}}
+ ]
+ },
+ "connector_tags_query": {
+ "collection": "connector_tags",
+ "aggregation": [
+ {"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1,
+ "category": 1, "data_type": 1, "status": 1}}
+ ]
}
}
```
+`connectors_query` and `connector_tags_query` are optional: the values above are the defaults in the code (see "OPC slots" and "Write tags" above).
+
+The `$lookup` must end in a `model` field (singular): every pipeline type reads `model.name` (and `model.model_config`, when present), so a pipeline without `model` fails to build its schedule.
+
#### Simple metrics: parallel windows & adaptive frequency
One `pipelines` doc per window, all with the same `model_id`. Name them `-simple_metrics-m`; each doc becomes its own Temporal schedule. Thresholds are shared through `model.model_config`. The adaptive state is per doc.
@@ -823,7 +1085,9 @@ orchestrator/
├── utils/ # Utility functions
│ ├── connectors_config.py # Database configuration
│ ├── email_builder.py # Email template generation
-│ ├── orchestrator_functions.py # Pipeline utilities
+│ ├── pipeline_configs.py # Schedule input builders per pipeline type
+│ ├── tag_database.py # Tag database indexes, OPC read names and write tags
+│ ├── opc_slots.py # OPC slot v2 (contract C1)
│ ├── converters.py # Data type conversion utilities
│ └── templates/ # Email HTML templates
│ ├── email_template.html # Report email template
diff --git a/e2e/helpers.py b/e2e/helpers.py
index 451148b..e60070f 100644
--- a/e2e/helpers.py
+++ b/e2e/helpers.py
@@ -25,8 +25,8 @@ MONGO_COLLECTIONS = (
'receiver_groups',
'orchestrated_schedules',
'pipelines',
- 'opc_servers',
- 'opc-servers',
+ 'connectors',
+ 'connector_tags',
)
DATETIME_FORMAT_MS_WITH_TZ = '%Y-%m-%d %H:%M:%S.%f%z'
@@ -143,19 +143,25 @@ def seed_pipelines(
client.close()
-def seed_opc_servers(
+def seed_tag_database(
mongo_uri: str,
database: str,
- servers: list[dict[str, Any]],
- *,
- collection: str = 'opc_servers',
+ connectors: list[dict[str, Any]],
+ connector_tags: list[dict[str, Any]],
+ opc_certificates: list[dict[str, Any]] | None = None,
) -> None:
- """Insert OPC server documents into the test Mongo database."""
+ """
+ Insert tag database documents (connectors, connector_tags and the opc_certificates
+ the connectors point to) into the test Mongo database.
+ """
client = MongoClient(mongo_uri)
try:
- coll = client[database][collection]
- if servers:
- coll.insert_many(servers)
+ if connectors:
+ client[database]['connectors'].insert_many(connectors)
+ if connector_tags:
+ client[database]['connector_tags'].insert_many(connector_tags)
+ if opc_certificates:
+ client[database]['opc_certificates'].insert_many(opc_certificates)
finally:
client.close()
diff --git a/e2e/scenario_inputs/orchestrator_conflict.json b/e2e/scenario_inputs/orchestrator_conflict.json
index fdca0f7..fd36eb6 100644
--- a/e2e/scenario_inputs/orchestrator_conflict.json
+++ b/e2e/scenario_inputs/orchestrator_conflict.json
@@ -4,10 +4,6 @@
"pipelines_query": {
"collection": "pipelines",
"aggregation": [{"$match": {"active": true}}]
- },
- "opc_servers_query": {
- "collection": "opc_servers",
- "filters": {"active": true}
}
},
"pipelines": [
@@ -22,13 +18,16 @@
"frequency": "1m",
"offset": "0m",
"query": "SELECT 1",
- "write_tags": [
- {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
- ]
+ "write_tags": [{"tag_id": 101}]
}
],
- "opc_servers": [
- {"id": "srv-1", "active": true}
+ "connectors": [
+ {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+ "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+ ],
+ "connector_tags": [
+ {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+ "category": "Write", "status": "Active", "updated_at": "@now-1d"}
],
"active_ingestors": ["heartbeat:ingestor:1"]
}
diff --git a/e2e/scenario_inputs/orchestrator_create_only.json b/e2e/scenario_inputs/orchestrator_create_only.json
index 8d7494b..2d5d94a 100644
--- a/e2e/scenario_inputs/orchestrator_create_only.json
+++ b/e2e/scenario_inputs/orchestrator_create_only.json
@@ -4,10 +4,6 @@
"pipelines_query": {
"collection": "pipelines",
"aggregation": [{"$match": {"active": true}}]
- },
- "opc_servers_query": {
- "collection": "opc_servers",
- "filters": {"active": true}
}
},
"pipelines": [
@@ -22,13 +18,16 @@
"frequency": "1m",
"offset": "0m",
"query": "SELECT 1",
- "write_tags": [
- {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
- ]
+ "write_tags": [{"tag_id": 101}]
}
],
- "opc_servers": [
- {"id": "srv-1", "active": true}
+ "connectors": [
+ {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+ "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+ ],
+ "connector_tags": [
+ {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+ "category": "Write", "status": "Active", "updated_at": "@now-1d"}
],
"active_ingestors": ["heartbeat:ingestor:1"]
}
diff --git a/e2e/scenario_inputs/orchestrator_delete_only.json b/e2e/scenario_inputs/orchestrator_delete_only.json
index 750d359..99ade41 100644
--- a/e2e/scenario_inputs/orchestrator_delete_only.json
+++ b/e2e/scenario_inputs/orchestrator_delete_only.json
@@ -4,10 +4,6 @@
"pipelines_query": {
"collection": "pipelines",
"aggregation": [{"$match": {"active": true}}]
- },
- "opc_servers_query": {
- "collection": "opc_servers",
- "filters": {"active": true}
}
},
"pipelines": [
@@ -24,8 +20,5 @@
"interval_minutes": 60
}
],
- "opc_servers": [
- {"id": "srv-1", "active": true}
- ],
"active_ingestors": ["heartbeat:ingestor:1"]
}
diff --git a/e2e/scenario_inputs/orchestrator_drift_window.json b/e2e/scenario_inputs/orchestrator_drift_window.json
index a034a9c..1671f6d 100644
--- a/e2e/scenario_inputs/orchestrator_drift_window.json
+++ b/e2e/scenario_inputs/orchestrator_drift_window.json
@@ -4,10 +4,6 @@
"pipelines_query": {
"collection": "pipelines",
"aggregation": [{"$match": {"active": true}}]
- },
- "opc_servers_query": {
- "collection": "opc_servers",
- "filters": {"active": true}
}
},
"pipelines": [
@@ -33,6 +29,5 @@
"chunk_period": "s"
}
],
- "opc_servers": [],
"active_ingestors": ["heartbeat:ingestor:1"]
}
diff --git a/e2e/scenario_inputs/orchestrator_empty.json b/e2e/scenario_inputs/orchestrator_empty.json
index 67f8bce..28123cd 100644
--- a/e2e/scenario_inputs/orchestrator_empty.json
+++ b/e2e/scenario_inputs/orchestrator_empty.json
@@ -4,13 +4,8 @@
"pipelines_query": {
"collection": "pipelines",
"aggregation": [{"$match": {"active": true}}]
- },
- "opc_servers_query": {
- "collection": "opc_servers",
- "filters": {"active": true}
}
},
"pipelines": [],
- "opc_servers": [],
"active_ingestors": []
}
diff --git a/e2e/scenario_inputs/orchestrator_happy_path.json b/e2e/scenario_inputs/orchestrator_happy_path.json
index 0486b12..5499f75 100644
--- a/e2e/scenario_inputs/orchestrator_happy_path.json
+++ b/e2e/scenario_inputs/orchestrator_happy_path.json
@@ -4,10 +4,6 @@
"pipelines_query": {
"collection": "pipelines",
"aggregation": [{"$match": {"active": true}}]
- },
- "opc_servers_query": {
- "collection": "opc_servers",
- "filters": {"active": true}
}
},
"pipelines": [
@@ -22,9 +18,7 @@
"frequency": "1m",
"offset": "0m",
"query": "SELECT 1",
- "write_tags": [
- {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
- ]
+ "write_tags": [{"tag_id": 101}]
},
{
"schedule_name": "drift-gpu",
@@ -39,8 +33,13 @@
"interval_minutes": 60
}
],
- "opc_servers": [
- {"id": "srv-1", "active": true, "name": "opc-1"}
+ "connectors": [
+ {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+ "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+ ],
+ "connector_tags": [
+ {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+ "category": "Write", "status": "Active", "updated_at": "@now-1d"}
],
"active_ingestors": ["heartbeat:ingestor:1"]
}
diff --git a/e2e/scenario_inputs/orchestrator_noop.json b/e2e/scenario_inputs/orchestrator_noop.json
index c782136..d56ae25 100644
--- a/e2e/scenario_inputs/orchestrator_noop.json
+++ b/e2e/scenario_inputs/orchestrator_noop.json
@@ -4,10 +4,6 @@
"pipelines_query": {
"collection": "pipelines",
"aggregation": [{"$match": {"active": true}}]
- },
- "opc_servers_query": {
- "collection": "opc_servers",
- "filters": {"active": true}
}
},
"pipelines": [
@@ -22,13 +18,16 @@
"frequency": "1m",
"offset": "0m",
"query": "SELECT 1",
- "write_tags": [
- {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
- ]
+ "write_tags": [{"tag_id": 101}]
}
],
- "opc_servers": [
- {"id": "srv-1", "active": true}
+ "connectors": [
+ {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+ "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+ ],
+ "connector_tags": [
+ {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+ "category": "Write", "status": "Active", "updated_at": "@now-1d"}
],
"active_ingestors": ["heartbeat:ingestor:1"],
"orchestrated_schedules": [
diff --git a/e2e/scenario_inputs/orchestrator_simple_metrics_windows.json b/e2e/scenario_inputs/orchestrator_simple_metrics_windows.json
index 8f93c8b..4fe6f57 100644
--- a/e2e/scenario_inputs/orchestrator_simple_metrics_windows.json
+++ b/e2e/scenario_inputs/orchestrator_simple_metrics_windows.json
@@ -4,10 +4,6 @@
"pipelines_query": {
"collection": "pipelines",
"aggregation": [{"$match": {"active": true}}]
- },
- "opc_servers_query": {
- "collection": "opc_servers",
- "filters": {"active": true}
}
},
"pipelines": [
@@ -40,6 +36,5 @@
"interval_minutes": 1440
}
],
- "opc_servers": [],
"active_ingestors": ["heartbeat:ingestor:1"]
}
diff --git a/e2e/scenario_inputs/orchestrator_tag_database.json b/e2e/scenario_inputs/orchestrator_tag_database.json
new file mode 100644
index 0000000..cab7da2
--- /dev/null
+++ b/e2e/scenario_inputs/orchestrator_tag_database.json
@@ -0,0 +1,71 @@
+{
+ "workflow_input": {
+ "schedule_name": "orchestrator-e2e-tag-database",
+ "pipelines_query": {
+ "collection": "pipelines",
+ "aggregation": [{"$match": {"active": true}}]
+ }
+ },
+ "connectors": [
+ {
+ "connector_id": "plant",
+ "type": "opc_ua",
+ "name": "plant",
+ "url": "opc.tcp://plant:4840",
+ "subscription_period_ms": 500,
+ "security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256", "client_certificate_id": 1},
+ "status": "Active",
+ "updated_at": "@now-1d"
+ }
+ ],
+ "opc_certificates": [
+ {"certificate_id": 1, "kind": "client", "name": "sientia-platform", "status": "active",
+ "application_uri": "urn:sientia:opc-ingestor",
+ "cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/cert.pem", "sha256": null},
+ "private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/key.pem", "sha256": null},
+ "thumbprint_sha1": null, "not_before": "@now-1d", "not_after": "@now+365d",
+ "created_at": "@now-1d", "updated_at": "@now-1d", "created_by": "e2e", "updated_by": "e2e"},
+ {"certificate_id": 2, "kind": "client", "name": "retired", "status": "retired",
+ "application_uri": "urn:sientia:old",
+ "cert": {"bucket": "opc-certificates", "object_name": "client/retired/cert.pem"},
+ "private_key": {"bucket": "opc-certificates", "object_name": "client/retired/key.pem"},
+ "created_at": "@now-1d", "updated_at": "@now-1d", "created_by": "e2e", "updated_by": "e2e"}
+ ],
+ "connector_tags": [
+ {"tag_id": 7, "connector_id": "plant", "address": "ns=3;i=1001", "tag_name": "Pressure", "category": "Read", "status": "Active", "updated_at": "@now-1d"},
+ {"tag_id": 101, "connector_id": "plant", "address": "ns=3;i=2001", "tag_name": "PRED-101", "category": "Write", "data_type": "double", "status": "Active", "updated_at": "@now-1d"},
+ {"tag_id": 102, "connector_id": "plant", "address": "ns=3;i=2002", "tag_name": "CONF-101", "category": "Prediction Confidence", "status": "Active", "updated_at": "@now-1d"},
+ {"tag_id": 103, "connector_id": "plant", "address": "ns=3;i=2003", "tag_name": "PRED-OLD", "category": "Write", "status": "Inactive", "updated_at": "@now-1d"}
+ ],
+ "pipelines": [
+ {
+ "schedule_name": "opc-tagdb",
+ "workflow_type": "scouter",
+ "model_id": "model-opc",
+ "model": {"name": "Model model-opc"},
+ "active": true,
+ "updated_at": "@now-2d",
+ "frequency": "1m",
+ "read_tags": [{"tag_id": 7, "frequency": 1000, "aggr_func": "avg"}]
+ },
+ {
+ "schedule_name": "pred-tagdb",
+ "workflow_type": "predictions_batch",
+ "model_id": "model-opc",
+ "model": {"name": "Model model-opc"},
+ "active": true,
+ "updated_at": "@now-2d",
+ "frequency": "1m",
+ "query": "SELECT 1",
+ "write_tags": [
+ {"tag_id": 101},
+ {"tag_id": 102},
+ {"tag_id": 7},
+ {"tag_id": 103},
+ {"tag_id": 404},
+ {"tag_name": "PRED-NO-ID"}
+ ]
+ }
+ ],
+ "active_ingestors": ["heartbeat:ingestor:1"]
+}
diff --git a/e2e/scenario_inputs/orchestrator_ttl_index.json b/e2e/scenario_inputs/orchestrator_ttl_index.json
index b17b468..d2f6318 100644
--- a/e2e/scenario_inputs/orchestrator_ttl_index.json
+++ b/e2e/scenario_inputs/orchestrator_ttl_index.json
@@ -4,10 +4,6 @@
"pipelines_query": {
"collection": "pipelines",
"aggregation": [{"$match": {"active": true}}]
- },
- "opc_servers_query": {
- "collection": "opc_servers",
- "filters": {"active": true}
}
},
"pipelines": [
@@ -22,24 +18,17 @@
"frequency": "1m",
"offset": "0m",
"read_tags": [
- {
- "server_id": "srv-1",
- "tag_name": "Read1",
- "tag_address": "ns=2;s=Read1",
- "aggr_func": "lts",
- "frequency": 1000
- }
+ {"tag_id": 1, "aggr_func": "lts", "frequency": 1000}
]
}
],
- "opc_servers": [
- {
- "id": "srv-1",
- "active": true,
- "server_name": "opc-1",
- "url": "opc.tcp://localhost:4840",
- "uri": "urn:opcfoundation:UA:DemoServer"
- }
+ "connectors": [
+ {"connector_id": "opc-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://localhost:4840",
+ "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+ ],
+ "connector_tags": [
+ {"tag_id": 1, "connector_id": "opc-1", "address": "ns=2;s=Read1", "tag_name": "Read1",
+ "category": "Read", "status": "Active"}
],
"active_ingestors": ["heartbeat:ingestor:1"]
}
diff --git a/e2e/scenario_inputs/orchestrator_update_only.json b/e2e/scenario_inputs/orchestrator_update_only.json
index 3cff4ba..65b4f16 100644
--- a/e2e/scenario_inputs/orchestrator_update_only.json
+++ b/e2e/scenario_inputs/orchestrator_update_only.json
@@ -4,10 +4,6 @@
"pipelines_query": {
"collection": "pipelines",
"aggregation": [{"$match": {"active": true}}]
- },
- "opc_servers_query": {
- "collection": "opc_servers",
- "filters": {"active": true}
}
},
"pipelines": [
@@ -22,13 +18,16 @@
"frequency": "5m",
"offset": "0m",
"query": "SELECT 1",
- "write_tags": [
- {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
- ]
+ "write_tags": [{"tag_id": 101}]
}
],
- "opc_servers": [
- {"id": "srv-1", "active": true}
+ "connectors": [
+ {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+ "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+ ],
+ "connector_tags": [
+ {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+ "category": "Write", "status": "Active", "updated_at": "@now-1d"}
],
"active_ingestors": ["heartbeat:ingestor:1"],
"orchestrated_schedules": [
diff --git a/e2e/scenarios.md b/e2e/scenarios.md
index aa3e9af..b550372 100644
--- a/e2e/scenarios.md
+++ b/e2e/scenarios.md
@@ -26,7 +26,9 @@ Source: `e2e/test_orchestrator_main_workflow.py`
### 1.1.1 Happy path
-Pipelines in Mongo → Temporal schedule created in the `laborious` namespace, `orchestrated_schedules` entries written.
+Pipelines in Mongo → both Temporal schedules created in the `laborious` namespace, one
+`orchestrated_schedules` entry per pipeline, and the `predictions_batch` write tag (`tag_id`)
+resolved into `opc_output_config` through the tag database.
### 1.3.1 Create-only
@@ -64,6 +66,34 @@ Two `drift` pipelines on `frequency='1m'`: one without `interval_minutes` and wi
`interval_minutes=60` and `chunk_period='s'` → both forwarded unchanged, no warning
(SIENTIAPDE-2212).
+### 1.9 OPC tag database: read and write tags by tag_id
+
+Scenario input: `orchestrator_tag_database.json` (one `opc_ua` connector, its `connector_tags` of
+every category, one inactive, an OPC scouter reading by `tag_id` and a `predictions_batch` writing by
+`tag_id`, with a Read tag, an inactive tag, an unknown `tag_id` and an entry without `tag_id` among its
+write tags).
+
+#### 1.9.1 Write tags through the tag database
+
+`write_tags: [{tag_id}]` → `opc_output_config` keyed by `connector_id`, address and `data_type`
+from `connector_tags` (`float` when absent), `Write` → `prediction_tags`, `Prediction Confidence` →
+`confidence_tags`. The Read, inactive and unknown tags are left out and notified once
+(`ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`); the entry without `tag_id` is notified
+(`ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION`).
+
+#### 1.9.2 A tag change redeploys its pipelines
+
+A second run with nothing changed redeploys nothing (`orchestrated_schedules` is not stamped
+again). Then renaming the read tag, moving the write tag and deleting the confidence tag in
+`connector_tags` by hand (no `updated_at` bumped anywhere, pipelines untouched) → the next run
+redeploys both through their `config_hash`: the scouter `model_tags` and the slot `names` take
+the new name, the `opc_output_config` the new address without the confidence tag, and
+`orchestrated_schedules` records the new `config_hash`.
+
+#### 1.9.3 OPC slot from the tag database
+
+The slot v2 written to Redis holds only the active Read tag, with the connector's server fields.
+
---
## 2. Alerts workflow
diff --git a/e2e/test_orchestrator_main_workflow.py b/e2e/test_orchestrator_main_workflow.py
index 86d0091..fe87e9a 100644
--- a/e2e/test_orchestrator_main_workflow.py
+++ b/e2e/test_orchestrator_main_workflow.py
@@ -1,5 +1,6 @@
"""E2E tests for the Orchestrator main workflow."""
+import json
from datetime import UTC, datetime, timedelta
import pytest
@@ -14,9 +15,9 @@ from e2e.helpers import (
load_scenario_input,
make_workflow_id,
seed_active_ingestors,
- seed_opc_servers,
seed_orchestrated_schedules,
seed_pipelines,
+ seed_tag_database,
start_and_await_workflow,
)
from orchestrator.workflows.orchestrator import Orchestrator
@@ -41,7 +42,7 @@ def _apply_orchestrator_seeds(
redis_client: Redis,
) -> None:
"""
- Seed Mongo and Redis with the pipelines/opc_servers/ingestors declared in a scenario.
+ Seed Mongo and Redis with the pipelines/tag database/ingestors of a scenario.
Args:
scenario: Scenario payload returned by ``load_scenario_input``.
@@ -49,7 +50,13 @@ def _apply_orchestrator_seeds(
redis_client: Redis client connected to the test instance.
"""
seed_pipelines(mongo_uri, E2E_DATABASE, scenario.get('pipelines', []))
- seed_opc_servers(mongo_uri, E2E_DATABASE, scenario.get('opc_servers', []))
+ seed_tag_database(
+ mongo_uri,
+ E2E_DATABASE,
+ scenario.get('connectors', []),
+ scenario.get('connector_tags', []),
+ scenario.get('opc_certificates', []),
+ )
seed_active_ingestors(redis_client, scenario.get('active_ingestors', []))
if scenario.get('orchestrated_schedules'):
seed_orchestrated_schedules(
@@ -84,14 +91,19 @@ async def test_scenario_1_1_1_happy_path(
try:
tracked = list(mongo[E2E_DATABASE]['orchestrated_schedules'].find())
names = {doc['schedule_name'] for doc in tracked}
- assert names, f'Expected orchestrated_schedules rows, got {tracked}'
+ assert names == {'pred-legacy', 'drift-gpu'}, f'Unexpected orchestrated_schedules {tracked}'
finally:
mongo.close()
laborious_schedules = await _schedule_ids(temporal_host, 'laborious')
- assert 'pred-legacy' in laborious_schedules or 'drift-gpu' in laborious_schedules, (
+ assert {'pred-legacy', 'drift-gpu'} <= set(laborious_schedules), (
f'Expected Temporal schedules in laborious, got {laborious_schedules}'
)
+ # The write tag comes from the tag database, keyed by the connector the gateway writes to
+ pred = await _schedule_input(temporal_host, 'pred-legacy')
+ assert pred['opc_output_config'] == {
+ 'srv-1': {'prediction_tags': {'ns=2;s=Tag1': {'data_type': 'float'}}}
+ }
@pytest.mark.e2e
@@ -315,9 +327,9 @@ async def test_scenario_1_7_2_simple_metrics_adaptive_frequency_round_trip(
assert await _schedule_every(temporal_host, short) == timedelta(hours=1)
-async def _schedule_input(host: str, schedule_id: str) -> dict:
- """Return the decoded workflow input of a laborious-namespace schedule."""
- client = await Client.connect(host, namespace='laborious')
+async def _schedule_input(host: str, schedule_id: str, namespace: str = 'laborious') -> dict:
+ """Return the decoded workflow input of a schedule (laborious namespace by default)."""
+ client = await Client.connect(host, namespace=namespace)
description = await client.get_schedule_handle(schedule_id).describe()
action = description.schedule.action
assert isinstance(action, ScheduleActionStartWorkflow)
@@ -364,3 +376,195 @@ async def test_scenario_1_8_1_drift_window_and_chunk_period(
)
assert len(warnings) == 1
assert warnings[0]['model_id'] == 'model-dr'
+
+
+# 1.9 OPC tag database: read and write tags by tag_id (SIENTIAPDE-1974)
+
+
+async def _run_tag_database_scenario(temporal_env: WorkflowEnvironment, tag: str) -> None:
+ await start_and_await_workflow(
+ temporal_env.client,
+ Orchestrator.run,
+ load_scenario_input('orchestrator_tag_database')['workflow_input'],
+ make_workflow_id(f'orchestrator-tagdb-{tag}'),
+ )
+
+
+def _notifications(notification_handler, notification_id: str) -> list[dict]:
+ return list(notification_handler.mongo_collection.find({'notification_id': notification_id}))
+
+
+def _opc_slots(redis_client: Redis) -> dict[str, dict]:
+ return {
+ key if isinstance(key, str) else key.decode(): json.loads(redis_client.get(key))
+ for key in redis_client.keys('slot:opc_tags:*')
+ }
+
+
+def _orchestrated_schedules(mongo_uri: str) -> dict[str, dict]:
+ """orchestrated_schedules by schedule_name: updated_at (last deploy) and config_hash."""
+ mongo = MongoClient(mongo_uri)
+ try:
+ return {
+ doc['schedule_name']: {
+ 'updated_at': doc['updated_at'],
+ 'config_hash': doc.get('config_hash'),
+ }
+ for doc in mongo[E2E_DATABASE]['orchestrated_schedules'].find()
+ }
+ finally:
+ mongo.close()
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_9_1_write_tags_through_tag_database(
+ temporal_env: WorkflowEnvironment,
+ orchestrator_worker,
+ stub_workers,
+ mongo_uri: str,
+ temporal_host: str,
+ redis_client: Redis,
+ notification_handler,
+):
+ """1.9.1: write_tags by tag_id become the opc_output_config; the rest is notified."""
+ _apply_orchestrator_seeds(
+ load_scenario_input('orchestrator_tag_database'), mongo_uri, redis_client
+ )
+
+ await _run_tag_database_scenario(temporal_env, 'write')
+
+ pred = await _schedule_input(temporal_host, 'pred-tagdb')
+ assert pred['opc_output_config'] == {
+ 'plant': {
+ 'prediction_tags': {'ns=3;i=2001': {'data_type': 'double'}},
+ 'confidence_tags': {'ns=3;i=2002': {'data_type': 'float'}},
+ }
+ }
+
+ not_found = _notifications(
+ notification_handler, 'ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION'
+ )
+ assert len(not_found) == 1
+ # A Read tag, an inactive tag and an unknown tag_id
+ for value in ('pred-tagdb/7', 'pred-tagdb/103', 'pred-tagdb/404'):
+ assert value in not_found[0]['message']
+ invalid = _notifications(
+ notification_handler, 'ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION'
+ )
+ assert len(invalid) == 1
+ assert 'PRED-NO-ID' in invalid[0]['message']
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_9_2_tag_change_redeploys_pipelines(
+ temporal_env: WorkflowEnvironment,
+ orchestrator_worker,
+ stub_workers,
+ mongo_uri: str,
+ temporal_host: str,
+ redis_client: Redis,
+):
+ """
+ 1.9.2: changing or deleting a tag in connector_tags, without touching any updated_at,
+ redeploys the pipelines that reference it (config_hash); an unchanged run does not.
+ """
+ _apply_orchestrator_seeds(
+ load_scenario_input('orchestrator_tag_database'), mongo_uri, redis_client
+ )
+ await _run_tag_database_scenario(temporal_env, 'before-change')
+
+ scouter = await _schedule_input(temporal_host, 'opc-tagdb', namespace='scouter')
+ assert set(scouter['model_tags']) == {'Pressure'}
+ deployed = _orchestrated_schedules(mongo_uri)
+ assert all(doc.get('config_hash') for doc in deployed.values()), deployed
+
+ # Nothing changed: no redeploy, so orchestrated_schedules is not stamped again
+ await _run_tag_database_scenario(temporal_env, 'unchanged')
+ assert _orchestrated_schedules(mongo_uri) == deployed
+
+ # As an edit by hand in Compass: rename the read tag, move a write tag and delete the
+ # confidence tag, with no updated_at bumped anywhere
+ mongo = MongoClient(mongo_uri)
+ try:
+ tags = mongo[E2E_DATABASE]['connector_tags']
+ tags.update_one({'tag_id': 7}, {'$set': {'tag_name': 'Pressao'}})
+ tags.update_one({'tag_id': 101}, {'$set': {'address': 'ns=3;i=3001'}})
+ tags.delete_one({'tag_id': 102})
+ finally:
+ mongo.close()
+
+ await _run_tag_database_scenario(temporal_env, 'after-change')
+
+ scouter = await _schedule_input(temporal_host, 'opc-tagdb', namespace='scouter')
+ assert set(scouter['model_tags']) == {'Pressao'}
+ pred = await _schedule_input(temporal_host, 'pred-tagdb')
+ assert pred['opc_output_config'] == {
+ 'plant': {'prediction_tags': {'ns=3;i=3001': {'data_type': 'double'}}}
+ }
+ redeployed = _orchestrated_schedules(mongo_uri)
+ for name in ('opc-tagdb', 'pred-tagdb'):
+ assert redeployed[name]['config_hash'] != deployed[name]['config_hash']
+ assert redeployed[name]['updated_at'] > deployed[name]['updated_at']
+ # The slot names the read tag like the scouter does (SC1)
+ [slot] = _opc_slots(redis_client).values()
+ [name] = slot['plant']['tags']['ns=3;i=1001']['names']
+ assert name['tag_name'] == 'Pressao'
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_9_3_opc_slot_from_tag_database(
+ temporal_env: WorkflowEnvironment,
+ orchestrator_worker,
+ stub_workers,
+ mongo_uri: str,
+ redis_client: Redis,
+):
+ """1.9.3: the slot v2 holds only the active Read tags of the tag database."""
+ _apply_orchestrator_seeds(
+ load_scenario_input('orchestrator_tag_database'), mongo_uri, redis_client
+ )
+
+ await _run_tag_database_scenario(temporal_env, 'slot')
+
+ assert _opc_slots(redis_client) == {
+ 'slot:opc_tags:1': {
+ 'plant': {
+ 'server_id': 'plant',
+ 'name': 'plant',
+ 'url': 'opc.tcp://plant:4840',
+ 'subscription_period_ms': 500,
+ # The active client certificate of client_certificate_id, without its
+ # dates, sha256 or audit fields (D5)
+ 'security': {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'},
+ 'certificate': {
+ 'certificate_id': 1,
+ 'application_uri': 'urn:sientia:opc-ingestor',
+ 'cert': {
+ 'bucket': 'opc-certificates',
+ 'object_name': 'client/sientia-platform/cert.pem',
+ },
+ 'private_key': {
+ 'bucket': 'opc-certificates',
+ 'object_name': 'client/sientia-platform/key.pem',
+ },
+ },
+ 'tags': {
+ 'ns=3;i=1001': {
+ 'frequency': 1000,
+ 'active_read': {'enabled': False},
+ 'names': [
+ {
+ 'collection': 'raw_opc-tagdb',
+ 'tag_name': 'Pressure',
+ 'model_id': 'model-opc',
+ 'model_name': 'Model model-opc',
+ }
+ ],
+ }
+ },
+ }
+ }
+ }
diff --git a/orchestrator/activities/email.py b/orchestrator/activities/email.py
index 6fa0b08..f40e969 100644
--- a/orchestrator/activities/email.py
+++ b/orchestrator/activities/email.py
@@ -177,14 +177,13 @@ class Email(SientiaMonitoring):
self.logger.error(f'SMTP server disconnected: {e}')
self.logger.info(f'Reconnecting to {self.smtp_server}:{self.smtp_port}')
- if self.server:
- try:
- self.server.quit()
- except SMTPServerDisconnected as e:
- self.logger.info(f'Server already disconnected: {e}')
- except Exception as e:
- self.logger.error(f'Failed to quit server: {e}')
- raise e
+ try:
+ self.server.quit()
+ except SMTPServerDisconnected as e:
+ self.logger.info(f'Server already disconnected: {e}')
+ except Exception as e:
+ self.logger.error(f'Failed to quit server: {e}')
+ raise e
self.server = smtplib.SMTP(self.smtp_server, self.smtp_port, timeout=20)
if self.sender_password:
diff --git a/orchestrator/activities/formatters.py b/orchestrator/activities/formatters.py
index 8a06fea..0d7909b 100644
--- a/orchestrator/activities/formatters.py
+++ b/orchestrator/activities/formatters.py
@@ -1,6 +1,7 @@
from temporalio import activity, workflow
with workflow.unsafe.imports_passed_through():
+ import hashlib
import json
from collections.abc import Callable
from logging import Logger
@@ -17,21 +18,95 @@ with workflow.unsafe.imports_passed_through():
now,
)
- from orchestrator.utils.orchestrator_functions import (
- build_tag_config,
+ from orchestrator.utils.opc_slots import build_tag_config, gather_read_tags
+ from orchestrator.utils.pipeline_configs import (
chunk_period_warning,
drift,
frequency_alert_warning,
- gather_read_tags,
minimal_retrain,
pi_web_api_scouter,
predictions_batch,
scouter,
simple_metrics,
)
+ from orchestrator.utils.tag_database import (
+ WRITE_CATEGORIES,
+ WRITE_WORKFLOW_TYPES,
+ index_connector_tags,
+ index_connectors,
+ resolve_tag_names,
+ resolve_write_tags,
+ )
topic_separator = '\n ========== \n'
+# Schedule fields left out of config_hash: updated_at is compared on its own
+CONFIG_HASH_EXCLUDED_FIELDS = ('updated_at', 'config_hash')
+
+# Read tags left out of the slots, by gather_read_tags issue kind:
+# (kind, notification_id, message, level)
+SLOT_ISSUES = (
+ (
+ 'tags',
+ 'ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+ 'Tags {} not found among the active, complete Read tags of connector_tags',
+ NotificationLevel.ERROR,
+ ),
+ (
+ 'connectors',
+ 'ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+ 'Connectors {} not found, not active or incomplete in connectors',
+ NotificationLevel.ERROR,
+ ),
+ (
+ 'invalid',
+ 'ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION',
+ 'Read tags {} have no integer tag_id, no integer frequency or an active_read without an '
+ 'integer cycle_multiple; or their pipeline lacks schedule_name, model_id or model.name',
+ NotificationLevel.ERROR,
+ ),
+ (
+ 'certificates',
+ 'ORCHESTRATOR_CERTIFICATE_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+ 'Connectors {} have a security.client_certificate_id without an active client '
+ 'certificate in opc_certificates: they go into the slot without a certificate, so '
+ 'without security',
+ NotificationLevel.WARNING,
+ ),
+)
+
+# Write tags left out of the predictions_batch opc_output_config, by resolve_write_tags
+# issue kind; values are schedule_name/
+WRITE_TAG_ISSUES = (
+ (
+ 'tags',
+ 'ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION',
+ 'Write tags {} not found among the active, complete Write and Prediction Confidence '
+ 'tags of connector_tags',
+ NotificationLevel.ERROR,
+ ),
+ (
+ 'connectors',
+ 'ORCHESTRATOR_WRITE_TAG_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION',
+ 'Connectors {} of write tags not found, not active or incomplete in connectors',
+ NotificationLevel.ERROR,
+ ),
+ (
+ 'invalid',
+ 'ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION',
+ 'Write tags {} have no integer tag_id or a data_type the OPC Gateway does not accept',
+ NotificationLevel.ERROR,
+ ),
+)
+
+
+def _merge_issues(
+ issues: dict[str, set[str]], pipeline_issues: dict[str, set[str]], schedule_name: str
+) -> None:
+ """Add one pipeline's issues to the run's, each value prefixed with schedule_name/."""
+ for kind, values in pipeline_issues.items():
+ issues.setdefault(kind, set()).update(f'{schedule_name}/{value}' for value in values)
+
class ScheduleType(TypedDict):
"""
@@ -103,6 +178,20 @@ Each entry specifies:
"""
+def schedule_config_hash(schedule: dict[str, Any]) -> str:
+ """
+ Fingerprint of a schedule as process_schedules builds it, without updated_at.
+
+ Stored in orchestrated_schedules when the schedule is deployed: any change to what
+ reaches Temporal (a pipeline edited without bumping updated_at, a tag renamed, moved,
+ retyped, deactivated or deleted, a connector changed, or a new Orchestrator version
+ building it differently) changes the hash and redeploys the schedule.
+ """
+ content = {k: v for k, v in schedule.items() if k not in CONFIG_HASH_EXCLUDED_FIELDS}
+ serialized = json.dumps(content, sort_keys=True, separators=(',', ':'), default=str)
+ return hashlib.sha256(serialized.encode()).hexdigest()
+
+
class Formatters(SientiaMonitoring):
"""
Schedule and slot configuration formatting and notification filtering activity.
@@ -175,6 +264,16 @@ class Formatters(SientiaMonitoring):
- input_data (dict[str, Any]): The input data containing
the schedules to process.
- pipelines (list[dict[str, Any]]): The schedules to process.
+ - connectors (list[dict[str, Any]], optional): OPC UA connectors of the
+ tag database, for the write tags.
+ - connector_tags (list[dict[str, Any]], optional): The connector_tags the
+ pipelines reference, of every category and status.
+
+ Scouter read tags that carry only a tag_id are named from the tag database (the
+ same names as the slot). A predictions_batch gets its opc_output_config from its
+ write_tags ([{tag_id}]) through the tag database; what cannot be resolved is left
+ out and notified once per run. A change to a tag reaches the schedule through
+ create_schedule_config, which compares the resolved schedule (config_hash).
Returns:
- dict[str, Any]: The schedule configuration dictionary keyed by namespace
@@ -185,6 +284,11 @@ class Formatters(SientiaMonitoring):
self.info('Processing schedules...', metadata=metadata)
pipelines = input_data['pipelines']
+ connector_tags = input_data.get('connector_tags', [])
+ read_tags_by_id = index_connector_tags(connector_tags)
+ write_tags_by_id = index_connector_tags(connector_tags, WRITE_CATEGORIES)
+ connectors = index_connectors(input_data.get('connectors', []))
+ write_issues: dict[str, set[str]] = {}
schedule_config: dict[str, dict[str, Any]] = {
self.scouter_namespace: {},
@@ -201,14 +305,31 @@ class Formatters(SientiaMonitoring):
schedule_type = schedule_types[workflow_type]
namespace = schedule_type['namespace']
function = schedule_type['function']
+ schedule_name = pipeline['schedule_name']
- schedule_config[namespace][pipeline['schedule_name']] = {
+ if workflow_type == 'scouter' and 'read_tags' in pipeline:
+ # Same names as the slot's names[] (process_slots): SIENTIAPDE-1974 SC1
+ pipeline = {
+ **pipeline,
+ 'read_tags': resolve_tag_names(pipeline['read_tags'], read_tags_by_id),
+ }
+
+ if workflow_type in WRITE_WORKFLOW_TYPES:
+ opc_output_config, issues = resolve_write_tags(
+ pipeline.get('write_tags') or [], write_tags_by_id, connectors
+ )
+ _merge_issues(write_issues, issues, schedule_name)
+ pipeline = {**pipeline, 'opc_output_config': opc_output_config}
+
+ schedule_config[namespace][schedule_name] = {
**function(pipeline),
- 'updated_at': pipeline.get(
- 'updated_at', now().strftime(DATETIME_FORMAT_MS_WITH_TZ)
- ),
+ # A pipeline without updated_at is redeployed on every run, as before
+ 'updated_at': pipeline.get('updated_at')
+ or now().strftime(DATETIME_FORMAT_MS_WITH_TZ),
}
+ self._notify_issues(write_issues, WRITE_TAG_ISSUES, metadata)
+
self.info('Processed schedules', metadata=metadata)
self.debug(json.dumps(schedule_config, indent=4, sort_keys=True), metadata=metadata)
@@ -218,14 +339,25 @@ class Formatters(SientiaMonitoring):
def process_slots(self, input_data: dict[str, Any]) -> dict[str, Any]:
"""
Extracts all read tags from input pipelines, divides them into slots and
- returns a slot config dictionary. If no ingestor is available, only one slot
- is created.
+ returns a slot config dictionary in the slot v2 format (contract C1). If no
+ ingestor is available, only one slot is created.
+
+ The division is by address: all the names of an address stay in the same slot
+ (SIENTIAPDE-1974, O5). Read tags that cannot be resolved are left out and
+ notified once per run: a read tag without tag_id or a usable frequency, a tag_id
+ missing from connector_tags, or a connector missing or inactive. A connector whose
+ client_certificate_id has no active client certificate goes into the slot without
+ a certificate, with a warning (D5).
Args:
- input_data (dict[str, Any]): The input data containing
the schedules to process.
- pipelines (list[dict[str, Any]]): The schedules to process.
- - opc_servers (list[str]): The OPC servers to create ingestor config.
+ - connectors (list[dict[str, Any]], optional): Active OPC UA connectors of
+ the tag database, the only source of the OPC servers, each with the
+ certificates of its client_certificate_id (connectors_query).
+ - connector_tags (list[dict[str, Any]], optional): The connector_tags the
+ pipelines reference; only the active Read tags are used.
- active_ingestors (list[str]): The active ingestors to divide into slots.
Returns:
@@ -237,55 +369,52 @@ class Formatters(SientiaMonitoring):
self.info('Processing slots...', metadata=metadata)
pipelines = input_data['pipelines']
- opc_servers_list = input_data['opc_servers']
active_ingestors = input_data['active_ingestors']
+ connectors = index_connectors(input_data.get('connectors', []))
+ tags_by_id = index_connector_tags(input_data.get('connector_tags', []))
- opc_servers = {}
- for server in opc_servers_list:
- opc_servers[server['id']] = {
- **server,
- }
+ entries, issues = gather_read_tags(pipelines, connectors, tags_by_id)
+ self._notify_issues(issues, SLOT_ISSUES, metadata)
- tags = list(gather_read_tags(pipelines).values())
-
- number_of_tags = len(tags)
number_of_slots = len(active_ingestors) if active_ingestors else 1
- tags_per_slot = ceil(number_of_tags / number_of_slots)
+ entries_per_slot = ceil(len(entries) / number_of_slots)
slot_config: dict[str, Any] = {}
- last_index = 0
-
- for i in range(1, number_of_slots):
- slot_tags = tags[last_index : last_index + tags_per_slot]
- slot_config[f'{i}'], notifications = build_tag_config(slot_tags, opc_servers)
-
- if notifications:
- self.send_notification(
- metadata=metadata,
- notification_id='ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION',
- message=f'Servers {", ".join(notifications)} not found in opc_servers',
- block='orchestrator',
- level=NotificationLevel.ERROR,
- )
-
- last_index += tags_per_slot
-
- slot_tags = tags[last_index:]
- slot_config[f'{number_of_slots}'], notifications = build_tag_config(slot_tags, opc_servers)
- if notifications:
- self.send_notification(
- metadata=metadata,
- notification_id='ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION',
- message=f'Servers {", ".join(notifications)} not found in opc_servers',
- block='orchestrator',
- level=NotificationLevel.ERROR,
- )
+ for i in range(number_of_slots):
+ slot_entries = entries[i * entries_per_slot : (i + 1) * entries_per_slot]
+ slot_config[f'{i + 1}'] = build_tag_config(slot_entries)
self.info('Processed slots', metadata=metadata)
self.debug(json.dumps(slot_config, indent=4, sort_keys=True), metadata=metadata)
return slot_config
+ def _notify_issues(
+ self,
+ issues: dict[str, set[str]],
+ kinds: tuple[tuple[str, str, str, str], ...],
+ metadata: dict[str, Any],
+ ) -> None:
+ """
+ Notify what was left out of the slots or schedules, one notification per kind.
+
+ Args:
+ issues (dict[str, set[str]]): What was left out, by kind
+ kinds (tuple[tuple[str, str, str, str], ...]): (kind, notification_id, message
+ with one {} for the joined values, level), e.g. SLOT_ISSUES or
+ WRITE_TAG_ISSUES
+ metadata (dict[str, Any]): Metadata for the notifications
+ """
+ for kind, notification_id, message, level in kinds:
+ if issues.get(kind):
+ self.send_notification(
+ metadata=metadata,
+ notification_id=notification_id,
+ message=message.format(', '.join(sorted(issues[kind]))),
+ block='orchestrator',
+ level=level,
+ )
+
@activity.defn(name='format_schedule_config')
def format_schedule_config(self, input_data: dict[str, Any]) -> dict[str, Any]:
"""
@@ -293,6 +422,8 @@ class Formatters(SientiaMonitoring):
Transforms a list of schedule configurations into a nested dictionary structure
organized by namespace and schedule name for efficient lookup and comparison.
+ Each schedule keeps when it was last deployed (updated_at) and the config_hash
+ it was deployed with (None for schedules deployed before config_hash existed).
Args:
input_data (dict[str, Any]): The input data containing:
@@ -300,7 +431,7 @@ class Formatters(SientiaMonitoring):
- metadata (dict): Metadata for logging purposes
Returns:
- dict[str, Any]: The formatted schedule config organized by namespace and schedule name
+ dict[str, Any]: {namespace: {schedule_name: {'updated_at', 'config_hash'}}}
"""
metadata = input_data['metadata']
@@ -308,16 +439,18 @@ class Formatters(SientiaMonitoring):
schedule_config = input_data['schedule_config']
- config: dict[str, dict[str, str]] = {}
+ config: dict[str, dict[str, dict[str, Any]]] = {}
for schedule in schedule_config:
namespace = schedule['namespace']
schedule_name = schedule['schedule_name']
- updated_at = schedule['updated_at']
if namespace not in config:
config[namespace] = {}
- config[namespace][schedule_name] = updated_at
+ config[namespace][schedule_name] = {
+ 'updated_at': schedule['updated_at'],
+ 'config_hash': schedule.get('config_hash'),
+ }
self.info('Formatted schedule config', metadata=metadata)
self.debug(json.dumps(config, indent=4, sort_keys=True), metadata=metadata)
@@ -336,32 +469,42 @@ class Formatters(SientiaMonitoring):
"""
Compare new and current schedules to determine which should be updated or created.
- Compares schedule timestamps to identify schedules that need updating (newer timestamp)
- or creating (schedule doesn't exist). Results are accumulated in the provided dictionaries.
+ A deployed schedule is updated when its updated_at is newer than its last deploy
+ (bumping updated_at forces a redeploy) or when its config_hash differs from the one
+ it was deployed with (any change to the resolved schedule, including tag database
+ changes and deleted tags). A schedule not deployed yet is created. Each schedule
+ sent to update or create carries its config_hash, stored once it is deployed.
+ Results are accumulated in the provided dictionaries.
Args:
schedules (dict[str, Any]): New schedules to compare
- current_schedules (dict[str, Any]): Existing schedules to compare against
+ current_schedules (dict[str, Any]): Existing schedules to compare against, as
+ format_schedule_config gives them
to_update (dict[str, Any]): Output accumulator for schedules that need updating
to_create (dict[str, Any]): Output accumulator for schedules that need creating
namespace (str): Namespace for the schedules being compared
metadata (dict[str, Any]): Metadata for logging purposes
"""
for schedule_name, schedule in schedules.items():
+ config_hash = schedule_config_hash(schedule)
+ deployed = {**schedule, 'config_hash': config_hash}
+
if schedule_name in current_schedules:
update_timestamp = schedule.get('updated_at', now())
- old_timestamp = current_schedules[schedule_name]
+ current = current_schedules[schedule_name]
+ old_timestamp = current['updated_at']
self.debug(
- f'Comparing schedule {schedule_name}:{update_timestamp} vs {old_timestamp}',
+ f'Comparing schedule {schedule_name}:{update_timestamp} vs {old_timestamp}'
+ f' (config_hash {config_hash} vs {current.get("config_hash")})',
metadata=metadata,
)
- if update_timestamp > old_timestamp:
- to_update[namespace][schedule_name] = schedule
+ if update_timestamp > old_timestamp or config_hash != current.get('config_hash'):
+ to_update[namespace][schedule_name] = deployed
else:
- to_create[namespace][schedule_name] = schedule
+ to_create[namespace][schedule_name] = deployed
@activity.defn(name='create_schedule_config')
def create_schedule_config(self, input_data: dict[str, Any]) -> dict[str, Any]:
@@ -370,7 +513,8 @@ class Formatters(SientiaMonitoring):
Compares the current schedule configuration in Temporal with the new schedule
configuration to determine which schedules need to be created, updated, or deleted.
- Uses timestamp comparison to identify schedules that have changed.
+ Uses the updated_at timestamps and the config_hash (compare_config_timestamps) to
+ identify schedules that have changed.
Also sends a SIMPLE_METRICS_FREQUENCY_ALERT_BELOW_FLOOR WARNING for laborious schedules
whose frequency_alert is below 1m (the value is kept) or unparseable (it is ignored),
and a DRIFT_CHUNK_PERIOD_INVALID WARNING for drift schedules whose configured
@@ -415,8 +559,8 @@ class Formatters(SientiaMonitoring):
schedules, current_schedules, to_update, to_create, namespace, metadata
)
- # Fires whenever the schedule is created or updated (each `updated_at` change,
- # including escalation flips).
+ # Fires whenever the schedule is created or updated (each `updated_at` or
+ # config_hash change, including escalation flips).
changed = {**to_create[self.laborious_namespace], **to_update[self.laborious_namespace]}
for schedule_name, schedule in changed.items():
warning = frequency_alert_warning(schedule.get('frequency_alert'))
@@ -469,15 +613,21 @@ class Formatters(SientiaMonitoring):
to determine which slots need to be inserted or deleted. Slots are identified
by numeric IDs, and excess slots are marked for deletion.
+ Only slots whose content changed are inserted, so an unchanged slot is not
+ rewritten (nor reported) on every run. gather_read_tags builds the slot in a
+ stable order, so the same pipelines always give the same slot (SIENTIAPDE-1974, O6).
+ A slot in the v1 format differs from its v2 rewrite and is replaced once.
+
Args:
input_data (dict[str, Any]): The input data containing:
- - current_slot_config (dict[str, Any]): The current slot config in Redis
+ - current_slot_config (dict[str, Any]): The current slot config in Redis,
+ keyed by Redis key (slot:opc_tags:{id}) or by slot id
- slot_config (dict[str, Any]): The slot config to process
- metadata (dict): Metadata for logging purposes
Returns:
dict[str, Any]: A dictionary containing:
- - to_insert (dict): Slots that need to be inserted/updated
+ - to_insert (dict): Slots that are new or changed
- to_delete (list[str]): Slot IDs that need to be deleted
"""
@@ -495,7 +645,16 @@ class Formatters(SientiaMonitoring):
if number_of_current_slots > number_of_slots:
to_delete = [str(i) for i in range(number_of_slots + 1, number_of_current_slots + 1)]
- output = {'to_delete': to_delete, 'to_insert': slot_config}
+ current_by_id = {
+ str(key).rsplit(':', 1)[-1]: config for key, config in current_slot_config.items()
+ }
+ to_insert = {
+ slot: config
+ for slot, config in slot_config.items()
+ if slot not in current_by_id or current_by_id[slot] != config
+ }
+
+ output = {'to_delete': to_delete, 'to_insert': to_insert}
self.info('Created slot config', metadata=metadata)
self.debug(json.dumps(output, indent=4, sort_keys=True), metadata=metadata)
diff --git a/orchestrator/activities/mongo_db.py b/orchestrator/activities/mongo_db.py
index d0256d0..bfa9a0b 100644
--- a/orchestrator/activities/mongo_db.py
+++ b/orchestrator/activities/mongo_db.py
@@ -193,7 +193,8 @@ class MongoDB(SientiaMonitoring):
for document in aggregated_documents:
for timestamp_field in timestamp_fields:
- if timestamp_field in document:
+ # Only BSON dates: a field already stored as text is left as it is
+ if isinstance(document.get(timestamp_field), datetime):
document[timestamp_field] = (
document[timestamp_field]
.replace(tzinfo=UTC)
@@ -223,12 +224,14 @@ class MongoDB(SientiaMonitoring):
"""
Update `updated_at` timestamps for successfully updated pipelines.
- Updates the `updated_at` field in the `orchestrated_schedules` collection
- for all pipelines that were successfully updated in Temporal.
+ Updates the `updated_at` and `config_hash` fields in the `orchestrated_schedules`
+ collection for all pipelines that were successfully updated in Temporal, so the
+ next run compares against what was deployed.
Args:
input_data (dict[str, Any]): Input data containing:
- - updated_pipelines (list[dict]): Pipelines with success flags to consider
+ - updated_pipelines (list[dict]): Pipelines with success flags (and
+ config_hash) to consider
- metadata (dict): Metadata for logging purposes
"""
updated_pipelines = input_data.get('updated_pipelines', [])
@@ -237,34 +240,53 @@ class MongoDB(SientiaMonitoring):
self.info('Updating pipelines timestamps...', metadata=metadata)
- argument = [
- {'schedule_name': pipeline['schedule_name'], 'namespace': pipeline['namespace']}
- for pipeline in updated_pipelines
- if pipeline['success']
- ]
- if not argument:
+ succeeded = [pipeline for pipeline in updated_pipelines if pipeline['success']]
+ if not succeeded:
# Every update failed: stamp nothing so they are retried next tick (same guard as
- # create_pipelines_timestamps). An empty filter would match every tracked schedule.
+ # create_pipelines_timestamps).
self.info('No successful pipeline updates to timestamp', metadata=metadata)
return
- data_filter = {'$or': argument}
- try:
- self.mongo_db_repository.update_many(
- 'orchestrated_schedules', data_filter, {'$set': {'updated_at': date_now}}, metadata
- )
- except Exception as e:
- trace = traceback.format_exc()
+ # One update per schedule (each has its own config_hash). A failure does not stop
+ # the others: they are all tried, and the failures are notified together.
+ failures: dict[str, str] = {}
+ traces: list[str] = []
+ for pipeline in succeeded:
+ schedule = f'{pipeline["namespace"]}/{pipeline["schedule_name"]}'
+ try:
+ self.mongo_db_repository.update_one(
+ 'orchestrated_schedules',
+ {
+ 'schedule_name': pipeline['schedule_name'],
+ 'namespace': pipeline['namespace'],
+ },
+ {
+ '$set': {
+ 'updated_at': date_now,
+ 'config_hash': pipeline.get('config_hash'),
+ }
+ },
+ metadata,
+ )
+ except Exception as e:
+ failures[schedule] = str(e)
+ traces.append(f'{schedule}\n{traceback.format_exc()}')
+
+ if failures:
+ trace = '\n'.join(traces)
+ details = '; '.join(f'{schedule}: {error}' for schedule, error in failures.items())
self.send_notification(
metadata=metadata,
notification_id='MONGODB_UPDATE_PIPELINES_ERROR',
- message=f'Failed to update pipelines timestamps: {e}',
+ message=f'Failed to update pipelines timestamps: {details}',
block='update_pipelines_timestamps',
level=NotificationLevel.ERROR,
attachment_content=trace,
)
self.error(trace, metadata=metadata)
- raise e
+ raise RuntimeError(
+ f'Failed to update {len(failures)} of {len(succeeded)} pipelines timestamps'
+ )
self.info(
f'Updated {len(updated_pipelines)} pipelines timestamps',
@@ -277,11 +299,13 @@ class MongoDB(SientiaMonitoring):
Insert `updated_at` timestamps for newly created pipelines.
Inserts new documents into the `orchestrated_schedules` collection
- for all pipelines that were successfully created in Temporal.
+ for all pipelines that were successfully created in Temporal, with the
+ `config_hash` they were created with.
Args:
input_data (dict[str, Any]): Input data containing:
- - created_pipelines (list[dict]): Pipelines with success flags to consider
+ - created_pipelines (list[dict]): Pipelines with success flags (and
+ config_hash) to consider
- metadata (dict): Metadata for logging purposes
"""
created_pipelines = input_data.get('created_pipelines', [])
@@ -296,6 +320,7 @@ class MongoDB(SientiaMonitoring):
'schedule_name': pipeline['schedule_name'],
'namespace': pipeline['namespace'],
'updated_at': date_now,
+ 'config_hash': pipeline.get('config_hash'),
}
for pipeline in created_pipelines
if pipeline['success']
diff --git a/orchestrator/activities/temporal_manager.py b/orchestrator/activities/temporal_manager.py
index ec622e4..ea2ce28 100644
--- a/orchestrator/activities/temporal_manager.py
+++ b/orchestrator/activities/temporal_manager.py
@@ -177,7 +177,9 @@ class TemporalManager(SientiaMonitoring):
- schedules (dict[str, Any]): The schedules to create.
Returns:
- - list[dict[str, Any]]: Report entries for each attempted schedule creation.
+ - list[dict[str, Any]]: Report entries for each attempted schedule creation,
+ with the schedule's config_hash (create_schedule_config) for
+ create_pipelines_timestamps.
"""
schedules_to_create = input_data['schedules']
@@ -198,6 +200,8 @@ class TemporalManager(SientiaMonitoring):
)
for schedule_name, schedule in schedules.items():
+ # Stored in orchestrated_schedules, not sent to the workflow
+ config_hash = schedule.pop('config_hash', None)
search_attributes = TypedSearchAttributes(
[
SearchAttributePair(key=self.model_id_id_key, value=schedule['model_id']),
@@ -258,6 +262,7 @@ class TemporalManager(SientiaMonitoring):
'schedule_name': schedule_name,
'success': True,
'message': 'Schedule created successfully',
+ 'config_hash': config_hash,
}
)
@@ -272,6 +277,7 @@ class TemporalManager(SientiaMonitoring):
'schedule_name': schedule_name,
'success': False,
'message': str(e),
+ 'config_hash': config_hash,
}
)
@@ -360,7 +366,9 @@ class TemporalManager(SientiaMonitoring):
- schedules (dict[str, Any]): The schedules to update.
Returns:
- - list[dict[str, Any]]: Report entries for each attempted schedule update.
+ - list[dict[str, Any]]: Report entries for each attempted schedule update,
+ with the schedule's config_hash (create_schedule_config) for
+ update_pipelines_timestamps.
"""
schedules_to_update = input_data['schedules']
@@ -380,6 +388,8 @@ class TemporalManager(SientiaMonitoring):
)
for schedule_name, schedule in schedules.items():
+ # Stored in orchestrated_schedules, not sent to the workflow
+ config_hash = schedule.pop('config_hash', None)
try:
await self._update_single_schedule(client, schedule_name, schedule, metadata)
@@ -389,6 +399,7 @@ class TemporalManager(SientiaMonitoring):
'schedule_name': schedule_name,
'success': True,
'message': 'Schedule updated successfully',
+ 'config_hash': config_hash,
}
)
@@ -403,6 +414,7 @@ class TemporalManager(SientiaMonitoring):
'schedule_name': schedule_name,
'success': False,
'message': str(e),
+ 'config_hash': config_hash,
}
)
diff --git a/orchestrator/utils/opc_slots.py b/orchestrator/utils/opc_slots.py
new file mode 100644
index 0000000..0bc6814
--- /dev/null
+++ b/orchestrator/utils/opc_slots.py
@@ -0,0 +1,301 @@
+"""OPC slot v2 (contract C1): read tags grouped by server and address for the gateway."""
+
+from typing import Any, cast
+
+from orchestrator.utils.tag_database import (
+ client_certificate_id,
+ is_complete,
+ resolve_certificate,
+ resolve_tag_names,
+ tag_id_of,
+)
+
+
+def _connector_server(connector: dict[str, Any], issues: dict[str, set[str]]) -> dict[str, Any]:
+ """
+ Slot server fields of a tag database connector (connectors collection).
+
+ A connector whose client_certificate_id has no active client certificate still goes
+ into the slot, without a certificate, and is recorded in issues['certificates'] (D5).
+ """
+ certificate = resolve_certificate(connector)
+ if certificate is None and client_certificate_id(connector) is not None:
+ issues['certificates'].add(str(connector['connector_id']))
+ return {
+ 'server_id': str(connector['connector_id']),
+ 'name': connector['name'],
+ 'url': connector['url'],
+ 'subscription_period_ms': connector.get('subscription_period_ms'),
+ 'security': {
+ 'mode': connector['security']['mode'],
+ 'policy': connector['security']['policy'],
+ },
+ 'certificate': certificate,
+ }
+
+
+def _frequency(tag: dict[str, Any]) -> int | None:
+ """The read tag's frequency as int, or None when it is missing or not an integer."""
+ try:
+ return int(tag['frequency'])
+ except (KeyError, TypeError, ValueError):
+ return None
+
+
+def _locate_read_tag(
+ tag_id: int,
+ connectors: dict[str, dict[str, Any]],
+ tags_by_id: dict[int, dict[str, Any]],
+ issues: dict[str, set[str]],
+) -> tuple[dict[str, Any], str] | None:
+ """
+ Server and address of a read tag, or None when they cannot be resolved.
+
+ The address and connector come from connector_tags and the server from connectors
+ (RF3). What cannot be resolved is recorded in issues ('tags' or 'connectors'), and a
+ server without its registered certificate in issues['certificates'].
+ """
+ connector_tag = tags_by_id.get(tag_id)
+ if connector_tag is None:
+ issues['tags'].add(str(tag_id))
+ return None
+ connector_id = str(connector_tag['connector_id'])
+ connector = connectors.get(connector_id)
+ if connector is None:
+ issues['connectors'].add(connector_id)
+ return None
+ return _connector_server(connector, issues), connector_tag['address']
+
+
+def _cycle_multiple(active_read: Any) -> int | None:
+ """The cycle_multiple of an enabled active_read as int (default 1), or None when invalid."""
+ try:
+ return int(active_read.get('cycle_multiple', 1))
+ except (AttributeError, TypeError, ValueError):
+ return None
+
+
+def _invalid_read_tag(tag: dict[str, Any]) -> bool:
+ """
+ A read tag the slot cannot use: no integer frequency, or an enabled active_read
+ without an integer cycle_multiple.
+ """
+ if _frequency(tag) is None:
+ return True
+ active_read = tag.get('active_read')
+ return (
+ isinstance(active_read, dict)
+ and bool(active_read.get('enabled'))
+ and _cycle_multiple(active_read) is None
+ )
+
+
+def _read_tag_ref(tag: dict[str, Any]) -> Any:
+ """How an invalid read tag is reported: its tag_id, or its tag_name without one."""
+ return tag.get('tag_id') if tag.get('tag_id') not in (None, '') else tag.get('tag_name')
+
+
+def _is_slot_pipeline(pipeline: dict[str, Any]) -> bool:
+ """A scouter pipeline whose read tags can be named in the slot (schedule, model id and name)."""
+ model = pipeline.get('model')
+ return (
+ is_complete(pipeline, ('schedule_name', 'model_id'))
+ and isinstance(model, dict)
+ and is_complete(model, ('name',))
+ )
+
+
+def _merge_active_read(current: dict[str, Any], requested: dict[str, Any] | None) -> dict[str, Any]:
+ """Active read of an address: on if any pipeline asks, with the lowest cycle_multiple (D3)."""
+ if not isinstance(requested, dict) or not requested.get('enabled'):
+ return current
+ cycle_multiple = cast(int, _cycle_multiple(requested))
+ if current.get('enabled'):
+ cycle_multiple = min(cycle_multiple, current['cycle_multiple'])
+ return {'enabled': True, 'cycle_multiple': cycle_multiple}
+
+
+def _assign_server_keys(servers: dict[str, dict[str, Any]]) -> None:
+ """
+ Give each server a unique slot key: its name, or name:server_id (extended until
+ free) when another server already took it. Assigned in server_id order, so the keys
+ are the same on every run and in every slot.
+ """
+ taken: set[str] = set()
+ for server_id in sorted(servers):
+ server = servers[server_id]
+ key = server['name']
+ while key in taken:
+ key = f'{key}:{server_id}'
+ server['key'] = key
+ taken.add(key)
+
+
+def _add_read_tag(
+ entries: dict[tuple[str, str], dict[str, Any]],
+ servers: dict[str, dict[str, Any]],
+ pipeline: dict[str, Any],
+ tag: dict[str, Any],
+ server: dict[str, Any],
+ address: str,
+) -> None:
+ """Merge one resolved read tag into its (server, address) entry and its server."""
+ server_id = server['server_id']
+ servers[server_id] = server
+
+ # int, so the slot does not depend on how each pipeline stored the number
+ frequency = cast(int, _frequency(tag))
+ entry = entries.setdefault(
+ (server_id, address),
+ {
+ 'address': address,
+ 'frequency': frequency,
+ 'active_read': {'enabled': False},
+ 'names': [],
+ },
+ )
+ entry['frequency'] = min(entry['frequency'], frequency)
+ entry['active_read'] = _merge_active_read(entry['active_read'], tag.get('active_read'))
+
+ name = {
+ 'collection': f'raw_{pipeline["schedule_name"]}',
+ 'tag_name': tag['tag_name'],
+ 'model_id': pipeline['model_id'],
+ 'model_name': pipeline['model']['name'],
+ }
+ if name not in entry['names']:
+ entry['names'].append(name)
+
+
+def _add_pipeline_read_tags(
+ entries: dict[tuple[str, str], dict[str, Any]],
+ servers: dict[str, dict[str, Any]],
+ issues: dict[str, set[str]],
+ pipeline: dict[str, Any],
+ connectors: dict[str, dict[str, Any]],
+ tags_by_id: dict[int, dict[str, Any]],
+) -> None:
+ """Add the read tags of one slot pipeline to entries, reporting the ones left out in issues."""
+ for tag in resolve_tag_names(pipeline.get('read_tags') or [], tags_by_id):
+ tag_id = tag_id_of(tag)
+ if tag_id is None or _invalid_read_tag(tag):
+ issues['invalid'].add(f'{pipeline["schedule_name"]}/{_read_tag_ref(tag)}')
+ continue
+
+ located = _locate_read_tag(tag_id, connectors, tags_by_id, issues)
+ if located is None:
+ continue
+ _add_read_tag(entries, servers, pipeline, tag, *located)
+
+
+def gather_read_tags(
+ pipelines: list[dict[str, Any]],
+ connectors: dict[str, dict[str, Any]],
+ tags_by_id: dict[int, dict[str, Any]],
+) -> tuple[list[dict[str, Any]], dict[str, set[str]]]:
+ """
+ Group the read tags of all scouter pipelines by (server, address) for the slot v2.
+
+ An address read by N pipelines becomes ONE entry with N items in names[]: the
+ gateway reads it once and writes it to N collections (SIENTIAPDE-1974, RF2). Each
+ read tag is resolved through the tag database by its tag_id (RF3): connectors is the
+ only source of the OPC servers.
+
+ Args:
+ pipelines (list[dict[str, Any]]): Pipeline configurations. Only workflow_type
+ 'scouter' is read; each needs schedule_name, model_id, model.name and
+ read_tags (with tag_id and frequency, and optionally active_read)
+ connectors (dict[str, dict[str, Any]]): Active OPC UA connectors indexed by
+ index_connectors
+ tags_by_id (dict[int, dict[str, Any]]): Active Read connector_tags indexed by
+ index_connector_tags
+
+ Returns:
+ tuple[list[dict[str, Any]], dict[str, set[str]]]:
+ - One entry per (server, address), sorted by server_id then address, with:
+ - server (dict): server_id, name, url, subscription_period_ms (None =
+ derive from the frequencies), security ({mode, policy}), certificate
+ (resolve_certificate; None = connect without security) and key (the
+ server's unique slot key)
+ - address (str): The OPC node address
+ - frequency (int): The lowest frequency among the pipelines (RF6)
+ - active_read (dict): {'enabled': False} unless a pipeline enables it (D3)
+ - names (list[dict]): collection ('raw_{schedule_name}'), tag_name,
+ model_id and model_name, one per pipeline name, sorted
+ - What was left out: 'tags' (tag_id), 'connectors' (connector_id) and
+ 'invalid' (schedule_name/read tag without an int tag_id, a usable frequency or
+ cycle_multiple, or the schedule_name of a pipeline without schedule_name,
+ model_id or model.name); and 'certificates': connector_ids that went into the
+ slot without the certificate their client_certificate_id points to
+ """
+ issues: dict[str, set[str]] = {
+ 'tags': set(),
+ 'connectors': set(),
+ 'invalid': set(),
+ 'certificates': set(),
+ }
+ servers: dict[str, dict[str, Any]] = {}
+ entries: dict[tuple[str, str], dict[str, Any]] = {}
+
+ for pipeline in pipelines:
+ if pipeline.get('workflow_type') != 'scouter':
+ continue
+ if not _is_slot_pipeline(pipeline):
+ issues['invalid'].add(str(pipeline.get('schedule_name')))
+ continue
+ _add_pipeline_read_tags(entries, servers, issues, pipeline, connectors, tags_by_id)
+
+ _assign_server_keys(servers)
+
+ gathered = []
+ for (server_id, _address), entry in sorted(entries.items()):
+ # A stable order keeps the slot identical between runs, so the gateway does
+ # not re-subscribe an unchanged slot (O6)
+ entry['names'].sort(key=lambda n: (n['collection'], n['tag_name'], str(n['model_id'])))
+ gathered.append({'server': servers[server_id], **entry})
+
+ return gathered, issues
+
+
+def build_tag_config(entries: list[dict[str, Any]]) -> dict[str, Any]:
+ """
+ Build one slot in the slot v2 format (contract C1) from address entries. Each server
+ carries its security and the MinIO objects of its client certificate, which the
+ gateway downloads (SIENTIAPDE-1974); certificate None means no security (D5).
+
+ Args:
+ entries (list[dict[str, Any]]): Address entries from gather_read_tags for this slot
+
+ Returns:
+ dict[str, Any]: The slot keyed by server key, each server with server_id, name,
+ url, subscription_period_ms, security, certificate and tags. Each tag (keyed
+ by address) has frequency, active_read and names. A connector without a
+ subscription_period_ms gets half its lowest tag frequency in this slot.
+ """
+ slot_config: dict[str, Any] = {}
+
+ for entry in entries:
+ server = entry['server']
+ if server['key'] not in slot_config:
+ slot_config[server['key']] = {
+ 'server_id': server['server_id'],
+ 'name': server['name'],
+ 'url': server['url'],
+ 'subscription_period_ms': server['subscription_period_ms'],
+ 'security': server['security'],
+ 'certificate': server['certificate'],
+ 'tags': {},
+ }
+
+ slot_config[server['key']]['tags'][entry['address']] = {
+ 'frequency': entry['frequency'],
+ 'active_read': entry['active_read'],
+ 'names': entry['names'],
+ }
+
+ for server_config in slot_config.values():
+ if server_config['subscription_period_ms'] is None:
+ frequencies = [tag['frequency'] for tag in server_config['tags'].values()]
+ server_config['subscription_period_ms'] = min(frequencies) / 2
+
+ return slot_config
diff --git a/orchestrator/utils/orchestrator_functions.py b/orchestrator/utils/pipeline_configs.py
similarity index 78%
rename from orchestrator/utils/orchestrator_functions.py
rename to orchestrator/utils/pipeline_configs.py
index ac13082..be44505 100644
--- a/orchestrator/utils/orchestrator_functions.py
+++ b/orchestrator/utils/pipeline_configs.py
@@ -1,3 +1,5 @@
+"""Pipeline configuration builders: one Temporal schedule input per pipeline type."""
+
from math import ceil
from typing import Any
@@ -264,8 +266,10 @@ def scouter(config: dict[str, Any]):
Args:
config (dict[str, Any]): Pipeline configuration containing:
- schedule_name (str): Name of the schedule (used for topic generation)
- - read_tags (list[dict]): List of tag configurations with:
- - tag_name (str): Name of the tag to read
+ - read_tags (list[dict]): List of tag configurations, already passed through
+ resolve_tag_names (process_schedules does it), with:
+ - tag_name (str): Name of the tag to read. A tag without one (a tag_id the
+ tag database cannot resolve) is left out; process_slots notifies it
- aggr_func (str, optional): Aggregation function for data collection (default: 'lts')
Common values: 'lts' (last), 'avg' (average), 'min', 'max', 'sum'
- data_range (list[int], optional): Valid data range [min, max] (default: [-100, 100])
@@ -278,6 +282,8 @@ def scouter(config: dict[str, Any]):
tags = {}
for tag in config['read_tags']:
+ if not tag.get('tag_name'):
+ continue
tags[tag['tag_name']] = {
'aggr_func': tag.get('aggr_func', 'lts'),
'data_range': tag.get('data_range', [-100, 100]),
@@ -415,11 +421,10 @@ def predictions_batch(config: dict[str, Any]):
Args:
config (dict[str, Any]): Pipeline configuration containing:
- query (str): SQL query to retrieve input data for predictions
- - write_tags (list[dict]): List of OPC tag configurations for write-back with:
- - server_id (str): ID of the target OPC server
- - type (str): Tag type - 'prediction' (model output) or 'confidence' (prediction confidence)
- - addr (str): OPC tag address/path
- - data_type (str, optional): OPC data type (default: 'float')
+ - opc_output_config (dict, optional): The OPC write-back configuration, built by
+ process_schedules from the pipeline's write_tags ([{tag_id}]) through the tag
+ database (resolve_write_tags): {connector_id: {prediction_tags |
+ confidence_tags: {address: {data_type}}}} (default: {})
- datetime_columns (list[str], optional): Column names to parse as datetime (default: [])
- path_priority (list[str], optional): Filter policy execution order (default: ["STOP", "CONTINUE", "REPEAT"])
- input_filters (list[dict], optional): Input data validation filters
@@ -435,23 +440,6 @@ def predictions_batch(config: dict[str, Any]):
dict[str, Any]: Complete predictions batch configuration with OPC output mappings,
PI Web API output configuration, multi-stage filters, SQL query, and retention policies
"""
- tags: dict[str, Any] = {}
- for tag in config.get('write_tags', []):
- if tag['server_id'] not in tags:
- tags[tag['server_id']] = {}
-
- tag_type = tag['type']
-
- if tag_type == 'prediction' or tag_type == 'confidence':
- tag_type_str = f'{tag_type}_tags'
-
- if tag_type_str not in tags[tag['server_id']]:
- tags[tag['server_id']][tag_type_str] = {}
-
- tags[tag['server_id']][tag_type_str][tag['addr']] = {
- 'data_type': tag.get('data_type', 'float'),
- }
-
path_priority = process_path_priority(
config.get('path_priority', ['STOP', 'CONTINUE', 'REPEAT'])
)
@@ -465,7 +453,7 @@ def predictions_batch(config: dict[str, Any]):
'save_transform': config.get('save_transform', True),
'transform_table_name': 'transformed_data',
'retention_time': config.get('model_retention_minutes', 60) * 60,
- 'opc_output_config': tags,
+ 'opc_output_config': config.get('opc_output_config', {}),
'pi_web_api_output_config': config.get('pi_web_api_output_config', {}),
'input_filters': overlap_filter_config(
{'EMPTY_DATA': {'policy': 'STOP', 'config': {}}}, config.get('input_filters', [])
@@ -484,112 +472,3 @@ def predictions_batch(config: dict[str, Any]):
'path_priority': path_priority,
'predictions_storage_policy': config.get('predictions_storage_policy', 'lts:1'),
}
-
-
-def gather_read_tags(pipelines: list[dict[str, Any]]) -> dict[str, Any]:
- """
- Gather all read tags from scouter pipeline configurations.
-
- Collects all read tags from scouter-type pipelines and organizes them by
- server_id and tag_address, tracking which Kafka topics each tag is associated with.
- This function is used during slot configuration to aggregate tags across multiple
- scouter pipelines for efficient OPC server slot allocation.
-
- Args:
- pipelines (list[dict[str, Any]]): List of pipeline configurations to process.
- Only pipelines with workflow_type 'scouter' are processed. Each scouter
- pipeline should contain a 'read_tags' list with tag configurations.
-
- Returns:
- dict[str, Any]: Dictionary of read tags keyed by "server_id:tag_address",
- where each entry contains:
- - All original tag configuration fields
- - topics (list[str]): List of Kafka topic names associated with this tag
- (format: 'raw_{schedule_name}')
- """
-
- tags = {}
-
- # Get all read tags from pipelines
- for pipeline in pipelines:
- if pipeline['workflow_type'] != 'scouter':
- continue
-
- for tag in pipeline.get('read_tags', []):
- tag_string = f'{tag["server_id"]}:{tag["tag_address"]}'
- if tag_string not in tags:
- tags[tag_string] = {**tag, 'topics': []}
-
- tags[tag_string]['topics'].append(f'raw_{pipeline["schedule_name"]}')
-
- return tags
-
-
-def build_tag_config(
- tags: list[dict[str, Any]], opc_servers: dict[str, Any]
-) -> tuple[dict[str, Any], list]:
- """
- Build tag configuration for a specific slot and OPC server.
-
- Organizes tags by OPC server name and calculates the minimum subscription period
- based on tag frequencies. Validates that all server IDs exist in the OPC
- servers configuration. The subscription period is set to half of the minimum
- tag frequency to ensure efficient data collection.
-
- Args:
- tags (list[dict[str, Any]]): List of tag configurations containing:
- - server_id (str): ID of the OPC server
- - tag_address (str): Address/path of the OPC tag
- - frequency (int): Tag read frequency in milliseconds
- - Additional tag-specific configuration fields
- opc_servers (dict[str, Any]): Dictionary of OPC server configurations keyed by server_id.
- Each server configuration should contain:
- - server_name (str): Human-readable server name
- - url (str): OPC server URL
- - uri (str): OPC server URI
- - cert_path (str, optional): Certificate file path
- - private_key_path (str, optional): Private key file path
- - server_cert_path (str, optional): Server certificate file path
-
- Returns:
- tuple[dict[str, Any], list]: A tuple containing:
- - Slot configuration dictionary organized by server_name, where each server
- contains connection details, tags dictionary, and subscription_period_ms
- - List of server IDs (str) that were not found in opc_servers configuration
- """
-
- slot_config = {}
- notifications = []
-
- for tag in tags:
- server_id = tag['server_id']
-
- if server_id not in opc_servers:
- notifications.append(server_id)
- continue
-
- server_name = opc_servers[server_id]['server_name']
- if server_name not in slot_config:
- slot_config[server_name] = {
- 'server_id': server_id,
- 'name': server_name,
- 'url': opc_servers[server_id]['url'],
- 'server_uri': opc_servers[server_id]['uri'],
- 'cert_path': opc_servers[server_id].get('cert_path', None),
- 'private_key_path': opc_servers[server_id].get('private_key_path', None),
- 'server_cert_path': opc_servers[server_id].get('server_cert_path', None),
- 'tags': {},
- }
-
- slot_config[server_name]['tags'][tag['tag_address']] = {
- **tag,
- }
-
- for server_name in slot_config:
- frequencies = [int(x['frequency']) for x in slot_config[server_name]['tags'].values()]
-
- min_frequency = min(frequencies) if frequencies else 1000
-
- slot_config[server_name]['subscription_period_ms'] = min_frequency / 2
-
- return slot_config, notifications
diff --git a/orchestrator/utils/tag_database.py b/orchestrator/utils/tag_database.py
new file mode 100644
index 0000000..9b22126
--- /dev/null
+++ b/orchestrator/utils/tag_database.py
@@ -0,0 +1,310 @@
+"""
+Tag database (connectors + connector_tags): which tags the pipelines reference, the
+indexes of what was loaded, and the resolution of the OPC UA read and write tags.
+"""
+
+from typing import Any
+
+# Fields a tag database document needs to take part in a slot or a write config.
+# Documents missing any of them are left out of the index, so their tags are notified
+# as not found instead of failing the whole run.
+CONNECTOR_TAG_FIELDS = ('tag_id', 'connector_id', 'address', 'tag_name')
+
+
+CONNECTOR_FIELDS = ('connector_id', 'name', 'url')
+
+
+# security fields of an OPC UA connector: the source of its OPC security (D4)
+SECURITY_FIELDS = ('mode', 'policy')
+
+
+# Fields of a client certificate the slot carries, and of each object it names in MinIO
+CERTIFICATE_FIELDS = ('certificate_id', 'application_uri')
+CERTIFICATE_OBJECTS = ('cert', 'private_key')
+STORED_OBJECT_FIELDS = ('bucket', 'object_name')
+
+
+# connector_tags categories each use accepts
+READ_CATEGORIES = ('Read',)
+WRITE_CATEGORIES = ('Write', 'Prediction Confidence')
+
+
+# opc_output_config block each write category goes to
+WRITE_TAG_GROUPS = {'Write': 'prediction_tags', 'Prediction Confidence': 'confidence_tags'}
+
+
+# data_type values the OPC Gateway write engine accepts (contract C2)
+OPC_DATA_TYPES = ('float', 'double', 'real', 'int', 'integer', 'bool', 'boolean', 'str', 'string')
+DEFAULT_DATA_TYPE = 'float'
+
+
+# Workflow types whose write_tags reference the tag database
+WRITE_WORKFLOW_TYPES = ('predictions_batch', 'xgboost_predictions_batch')
+
+
+def is_complete(document: dict[str, Any], fields: tuple[str, ...]) -> bool:
+ return all(document.get(field) not in (None, '') for field in fields)
+
+
+def tag_id_of(tag: Any) -> int | None:
+ """
+ The tag's tag_id, or None when it is not an integer.
+
+ The connector_tags validator stores tag_id as int, so a pipeline references a tag
+ by the same int: text ('7'), bool, null or a missing tag_id is no reference.
+ """
+ tag_id = tag.get('tag_id') if isinstance(tag, dict) else None
+ return tag_id if isinstance(tag_id, int) and not isinstance(tag_id, bool) else None
+
+
+def _is_active(document: dict[str, Any]) -> bool:
+ """A document without status (an override that does not project it) counts as active."""
+ return document.get('status') in (None, 'Active')
+
+
+def pipeline_tag_refs(pipeline: dict[str, Any]) -> list[Any]:
+ """
+ The tag references of a pipeline: the read_tags of a scouter and the write_tags of a
+ predictions_batch. Entries are returned as they are (they may lack a tag_id).
+ """
+ workflow_type = pipeline.get('workflow_type')
+ if workflow_type == 'scouter':
+ return list(pipeline.get('read_tags') or [])
+ if workflow_type in WRITE_WORKFLOW_TYPES:
+ return list(pipeline.get('write_tags') or [])
+ return []
+
+
+def referenced_tag_ids(pipelines: list[dict[str, Any]]) -> list[int]:
+ """
+ The tag_ids the pipelines reference (scouter read_tags and predictions_batch
+ write_tags), for the connector_tags query.
+
+ Sorted, so the query is the same on every run. Runs in the workflow, so it never
+ raises on malformed references.
+
+ Args:
+ pipelines (list[dict[str, Any]]): Pipeline configurations
+
+ Returns:
+ list[int]: The referenced tag_ids (empty when no tag carries a tag_id)
+ """
+ ids = {tag_id_of(tag) for pipeline in pipelines for tag in pipeline_tag_refs(pipeline)}
+ return sorted(tag_id for tag_id in ids if tag_id is not None)
+
+
+def filter_by_tag_ids(query: dict[str, Any], tag_ids: list[int]) -> dict[str, Any]:
+ """
+ Restrict a connector_tags aggregation to the referenced tag_ids.
+
+ Args:
+ query (dict[str, Any]): The connector_tags query ({'collection', 'aggregation'})
+ tag_ids (list[int]): The ids from referenced_tag_ids
+
+ Returns:
+ dict[str, Any]: A new query whose aggregation starts with a tag_id $match. The
+ aggregation list is new too: aggregate_documents_in_mongodb appends a stage
+ to the list it gets.
+ """
+ return {
+ **query,
+ 'aggregation': [{'$match': {'tag_id': {'$in': tag_ids}}}, *query['aggregation']],
+ }
+
+
+def index_tag_documents(connector_tags: list[dict[str, Any]]) -> dict[int, dict[str, Any]]:
+ """
+ Every loaded connector_tags document keyed by tag_id, whatever its status, category
+ or completeness.
+ """
+ indexed = {}
+ for tag in connector_tags:
+ tag_id = tag_id_of(tag)
+ if tag_id is not None:
+ indexed[tag_id] = tag
+ return indexed
+
+
+def index_connector_tags(
+ connector_tags: list[dict[str, Any]], categories: tuple[str, ...] = READ_CATEGORIES
+) -> dict[int, dict[str, Any]]:
+ """
+ Index the tag database's usable connector_tags by tag_id.
+
+ Documents that are not active, have another category, or lack an int tag_id,
+ connector_id, address or tag_name are left out.
+
+ Args:
+ connector_tags (list[dict[str, Any]]): connector_tags documents (SIENTIAPDE-1569)
+ categories (tuple[str, ...]): Accepted categories (READ_CATEGORIES for the slot
+ and the scouter names, WRITE_CATEGORIES for the write tags). A document
+ without category is accepted.
+
+ Returns:
+ dict[int, dict[str, Any]]: connector_tags documents keyed by tag_id
+ """
+ return {
+ tag_id: tag
+ for tag_id, tag in index_tag_documents(connector_tags).items()
+ if is_complete(tag, CONNECTOR_TAG_FIELDS)
+ and _is_active(tag)
+ and tag.get('category') in (None, *categories)
+ }
+
+
+def index_connectors(connectors: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
+ """
+ Index the tag database's connectors by connector_id (as text).
+
+ Documents that are not active or lack connector_id, name, url, security.mode or
+ security.policy are left out.
+
+ Args:
+ connectors (list[dict[str, Any]]): connectors documents (SIENTIAPDE-1569)
+
+ Returns:
+ dict[str, dict[str, Any]]: connectors documents keyed by str(connector_id)
+ """
+ return {
+ str(connector['connector_id']): connector
+ for connector in connectors
+ if is_complete(connector, CONNECTOR_FIELDS)
+ and _has_security(connector)
+ and _is_active(connector)
+ }
+
+
+def _has_security(connector: dict[str, Any]) -> bool:
+ security = connector.get('security')
+ return isinstance(security, dict) and is_complete(security, SECURITY_FIELDS)
+
+
+def client_certificate_id(connector: dict[str, Any]) -> Any:
+ """The connector's security.client_certificate_id (None when it has none)."""
+ security = connector.get('security')
+ return security.get('client_certificate_id') if isinstance(security, dict) else None
+
+
+def _is_usable_certificate(certificate: Any) -> bool:
+ return (
+ isinstance(certificate, dict)
+ and is_complete(certificate, CERTIFICATE_FIELDS)
+ and all(
+ isinstance(certificate.get(name), dict)
+ and is_complete(certificate[name], STORED_OBJECT_FIELDS)
+ for name in CERTIFICATE_OBJECTS
+ )
+ )
+
+
+def resolve_certificate(connector: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ The client certificate a connector goes into the slot with, or None (D5).
+
+ connectors_query's $lookup puts in `certificates` the opc_certificates document the
+ connector's security.client_certificate_id points to, only when it is active and of
+ kind client. None means no certificate: the gateway connects without security. A
+ connector with a client_certificate_id and no certificate is most likely a
+ registration error, reported by the caller (see client_certificate_id).
+
+ Args:
+ connector (dict[str, Any]): A connectors document from connectors_query
+
+ Returns:
+ dict[str, Any] | None: certificate_id, application_uri, and cert and private_key
+ as {bucket, object_name}; None without a complete certificate
+ """
+ for certificate in connector.get('certificates') or []:
+ if _is_usable_certificate(certificate):
+ return {
+ **{field: certificate[field] for field in CERTIFICATE_FIELDS},
+ **{
+ name: {field: certificate[name][field] for field in STORED_OBJECT_FIELDS}
+ for name in CERTIFICATE_OBJECTS
+ },
+ }
+ return None
+
+
+def resolve_tag_names(
+ read_tags: list[dict[str, Any]], tags_by_id: dict[int, dict[str, Any]]
+) -> list[dict[str, Any]]:
+ """
+ Fill in the tag_name of each read tag of a scouter pipeline.
+
+ The name is the pipeline's own tag_name when it has one, else the connector_tags
+ name of its tag_id. The scouter's model_tags (scouter()) and the slot's names[]
+ (gather_read_tags()) both take the name from here, so the name the gateway writes
+ is always the one the scouter looks for (SIENTIAPDE-1974, SC1).
+
+ Args:
+ read_tags (list[dict[str, Any]]): The pipeline's read_tags
+ tags_by_id (dict[int, dict[str, Any]]): connector_tags indexed by index_connector_tags
+
+ Returns:
+ list[dict[str, Any]]: The read tags, each with tag_name set when it could be
+ resolved. A tag_id missing from tags_by_id leaves the tag without a name.
+ """
+ resolved = []
+ for tag in read_tags:
+ name = tag.get('tag_name')
+ tag_id = tag_id_of(tag)
+ if not name and tag_id is not None:
+ name = tags_by_id.get(tag_id, {}).get('tag_name')
+ resolved.append({**tag, 'tag_name': name} if name else tag)
+ return resolved
+
+
+def resolve_write_tags(
+ write_tags: list[Any],
+ tags_by_id: dict[int, dict[str, Any]],
+ connectors: dict[str, dict[str, Any]],
+) -> tuple[dict[str, Any], dict[str, set[str]]]:
+ """
+ Build the opc_output_config of a predictions_batch from its write_tags.
+
+ Each write tag is only a tag_id: the tag database gives the rest. The category says
+ where the value goes ('Write' → prediction_tags, 'Prediction Confidence' →
+ confidence_tags), the connector_id is the server the OPC Gateway writes to (it
+ resolves it in connectors), and the address and data_type come from connector_tags
+ (data_type defaults to 'float').
+
+ Args:
+ write_tags (list[Any]): The pipeline's write_tags ([{tag_id}])
+ tags_by_id (dict[int, dict[str, Any]]): Write connector_tags indexed by
+ index_connector_tags(..., WRITE_CATEGORIES)
+ connectors (dict[str, dict[str, Any]]): Active OPC UA connectors indexed by
+ index_connectors
+
+ Returns:
+ tuple: the opc_output_config ({connector_id: {prediction_tags | confidence_tags:
+ {address: {data_type}}}}) and what was left out, by kind: 'tags' (tag_id not
+ an active, complete Write / Prediction Confidence tag), 'connectors'
+ (connector_id) and 'invalid' (an entry without an int tag_id, or a tag whose
+ data_type the gateway does not accept)
+ """
+ issues: dict[str, set[str]] = {'tags': set(), 'connectors': set(), 'invalid': set()}
+ output_config: dict[str, Any] = {}
+ for entry in write_tags:
+ tag_id = tag_id_of(entry)
+ if tag_id is None:
+ issues['invalid'].add(str(entry))
+ continue
+ connector_tag = tags_by_id.get(tag_id)
+ # The category is what tells a prediction from a confidence: it cannot be missing
+ if connector_tag is None or connector_tag.get('category') not in WRITE_TAG_GROUPS:
+ issues['tags'].add(str(tag_id))
+ continue
+ connector_id = str(connector_tag['connector_id'])
+ if connector_id not in connectors:
+ issues['connectors'].add(connector_id)
+ continue
+ data_type = connector_tag.get('data_type') or DEFAULT_DATA_TYPE
+ if data_type not in OPC_DATA_TYPES:
+ issues['invalid'].add(str(tag_id))
+ continue
+ group = WRITE_TAG_GROUPS[connector_tag['category']]
+ output_config.setdefault(connector_id, {}).setdefault(group, {})[
+ connector_tag['address']
+ ] = {'data_type': data_type}
+ return output_config, issues
diff --git a/orchestrator/worker/worker.py b/orchestrator/worker/worker.py
index 3c9d9ba..2d7254c 100644
--- a/orchestrator/worker/worker.py
+++ b/orchestrator/worker/worker.py
@@ -5,6 +5,7 @@ with workflow.unsafe.imports_passed_through():
import asyncio
import os
import sys
+ import traceback
from prometheus_client import start_http_server
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
@@ -192,13 +193,12 @@ async def main():
try:
await asyncio.gather(*handlers)
except BaseException as e:
- logger.error(f'An unhandled exception occurred: {e}', exc_info=True)
+ # sientia_do's Logger.error takes only a message: the traceback goes in it
+ logger.error(f'An unhandled exception occurred: {e}\n{traceback.format_exc()}')
exit_code = 1
finally:
- if notification_handler:
- notification_handler.shutdown()
- if activities:
- activities.shutdown()
+ notification_handler.shutdown()
+ activities.shutdown()
metrics.APP_UP.labels(pod_id=POD_ID).set(0)
sys.exit(exit_code)
diff --git a/orchestrator/workflows/orchestrator.py b/orchestrator/workflows/orchestrator.py
index a0fe690..0b5f839 100644
--- a/orchestrator/workflows/orchestrator.py
+++ b/orchestrator/workflows/orchestrator.py
@@ -7,6 +7,90 @@ with workflow.unsafe.imports_passed_through():
from sientia_do.temporal.policies import retry_policy
from orchestrator.activities.activities import Activities
+ from orchestrator.utils.tag_database import (
+ filter_by_tag_ids,
+ referenced_tag_ids,
+ )
+
+# MongoDB aggregation stages
+MATCH = '$match'
+PROJECT = '$project'
+LOOKUP = '$lookup'
+
+# Tag database (SIENTIAPDE-1569) read by the slot v2 and the schedules (SIENTIAPDE-1974).
+# The workflow input may override these aggregations (connectors_query /
+# connector_tags_query); the workflow puts a tag_id $match in front of
+# connector_tags_query. Every category and status of the referenced tags is loaded: the
+# slot uses the active Read tags, predictions_batch the active Write / Prediction
+# Confidence ones (status and category are filtered in the code). Only the fields used
+# are projected: other attributes may hold BSON types a Temporal payload cannot carry
+# (so the certificate dates are left out).
+# Each connector brings, in `certificates`, the opc_certificates document its
+# security.client_certificate_id points to, only when it is active and of kind client
+# (D5); resolve_certificate turns it into the slot's certificate.
+DEFAULT_CONNECTORS_QUERY = {
+ 'collection': 'connectors',
+ 'aggregation': [
+ {MATCH: {'type': 'opc_ua', 'status': 'Active'}},
+ {
+ LOOKUP: {
+ 'from': 'opc_certificates',
+ 'let': {'certificate_id': '$security.client_certificate_id'},
+ 'pipeline': [
+ {
+ MATCH: {
+ '$expr': {'$eq': ['$certificate_id', '$$certificate_id']},
+ 'status': 'active',
+ 'kind': 'client',
+ }
+ },
+ {
+ PROJECT: {
+ '_id': 0,
+ 'certificate_id': 1,
+ 'application_uri': 1,
+ 'cert.bucket': 1,
+ 'cert.object_name': 1,
+ 'private_key.bucket': 1,
+ 'private_key.object_name': 1,
+ }
+ },
+ ],
+ 'as': 'certificates',
+ }
+ },
+ {
+ PROJECT: {
+ '_id': 0,
+ 'connector_id': 1,
+ 'name': 1,
+ 'url': 1,
+ 'subscription_period_ms': 1,
+ 'security.mode': 1,
+ 'security.policy': 1,
+ 'security.client_certificate_id': 1,
+ 'certificates': 1,
+ }
+ },
+ ],
+}
+DEFAULT_CONNECTOR_TAGS_QUERY = {
+ 'collection': 'connector_tags',
+ 'aggregation': [
+ {
+ PROJECT: {
+ '_id': 0,
+ 'tag_id': 1,
+ 'connector_id': 1,
+ 'address': 1,
+ 'tag_name': 1,
+ 'category': 1,
+ 'data_type': 1,
+ 'status': 1,
+ }
+ },
+ ],
+}
@workflow.defn(name='orchestrator')
@@ -20,14 +104,79 @@ class Orchestrator:
and Redis infrastructure.
"""
+ async def _load_tag_database(
+ self,
+ input_data: dict[str, Any],
+ metadata: dict[str, Any],
+ pipelines: list[dict[str, Any]],
+ ) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ """
+ Load the tag database (SIENTIAPDE-1974): the only source of the OPC servers and
+ tags of the read and write tags.
+
+ Only the referenced tags are loaded, with the fields used, so the payload stays
+ bounded by the pipelines. Nothing is loaded when no scouter read tag nor
+ predictions_batch write tag has a tag_id.
+
+ Args:
+ input_data (dict[str, Any]): Workflow input (optional connectors_query and
+ connector_tags_query)
+ metadata (dict[str, Any]): Activity metadata ({'metadata': {...}})
+ pipelines (list[dict[str, Any]]): The loaded pipeline configurations
+
+ Returns:
+ tuple[list[dict[str, Any]], list[dict[str, Any]]]: connectors and connector_tags
+ """
+ tag_ids = referenced_tag_ids(pipelines)
+ if not tag_ids:
+ return [], []
+
+ return await self._load_connectors(
+ metadata,
+ input_data.get('connectors_query', DEFAULT_CONNECTORS_QUERY),
+ input_data.get('connector_tags_query', DEFAULT_CONNECTOR_TAGS_QUERY),
+ tag_ids,
+ )
+
+ async def _load_connectors(
+ self,
+ metadata: dict[str, Any],
+ connectors_query: dict[str, Any],
+ connector_tags_query: dict[str, Any],
+ tag_ids: list[Any],
+ ) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ """Run a connectors query and a connector_tags query restricted to tag_ids, in parallel."""
+ connectors_handler = workflow.start_local_activity_method(
+ Activities.aggregate_documents_in_mongodb,
+ {
+ **metadata,
+ # A fresh aggregation list: the activity appends a stage to the one it gets
+ 'query': {**connectors_query, 'aggregation': list(connectors_query['aggregation'])},
+ },
+ retry_policy=retry_policy,
+ start_to_close_timeout=timedelta(seconds=60),
+ )
+
+ connector_tags_handler = workflow.start_local_activity_method(
+ Activities.aggregate_documents_in_mongodb,
+ {
+ **metadata,
+ 'query': filter_by_tag_ids(connector_tags_query, tag_ids),
+ },
+ retry_policy=retry_policy,
+ start_to_close_timeout=timedelta(seconds=60),
+ )
+
+ return await connectors_handler, await connector_tags_handler
+
@workflow.run
async def run(self, input_data: dict[str, Any]):
"""
Execute the orchestration workflow for pipeline and slot management.
- This workflow retrieves pipeline configurations and OPC server data,
- processes schedules and slot configurations, and deploys them to
- the appropriate services. It handles creation, updates, and deletion
+ This workflow retrieves pipeline configurations and the tag database
+ (connectors and connector_tags), processes schedules and slot configurations,
+ and deploys them to the appropriate services. It handles creation, updates, and deletion
of schedules and slots based on current system state.
Args:
@@ -35,7 +184,11 @@ class Orchestrator:
Required fields:
- schedule_name (str): Name of the orchestration schedule
- pipelines_query (dict[str, Any]): MongoDB query for pipeline configurations
- - opc_servers_query (dict[str, Any]): MongoDB query for OPC server data
+ Optional fields:
+ - connectors_query (dict[str, Any]): MongoDB aggregation for the tag
+ database connectors (default: DEFAULT_CONNECTORS_QUERY)
+ - connector_tags_query (dict[str, Any]): MongoDB aggregation for the tag
+ database tags (default: DEFAULT_CONNECTOR_TAGS_QUERY)
Returns:
None: Workflow completes without return value
@@ -66,13 +219,6 @@ class Orchestrator:
start_to_close_timeout=timedelta(seconds=60),
)
- opc_servers_handler = workflow.start_local_activity_method(
- Activities.find_documents_in_mongodb,
- {**metadata, 'query': input_data['opc_servers_query']},
- retry_policy=retry_policy,
- start_to_close_timeout=timedelta(seconds=60),
- )
-
orchestrated_schedules_handler = workflow.start_local_activity_method(
Activities.find_documents_in_mongodb,
{
@@ -103,9 +249,13 @@ class Orchestrator:
)
pipeline_config = await pipeline_config_handler
+
+ connectors, connector_tags = await self._load_tag_database(
+ input_data, metadata, pipeline_config
+ )
+
orchestrated_schedules = await orchestrated_schedules_handler
current_slot_config = await current_slot_config_handler
- opc_servers = await opc_servers_handler
active_ingestors = await active_ingestors_handler
formatted_orchestrated_schedules_handler = workflow.start_local_activity_method(
@@ -117,7 +267,12 @@ class Orchestrator:
schedules_config_handler = workflow.start_local_activity_method(
Activities.process_schedules,
- {**metadata, 'pipelines': pipeline_config},
+ {
+ **metadata,
+ 'pipelines': pipeline_config,
+ 'connectors': connectors,
+ 'connector_tags': connector_tags,
+ },
retry_policy=retry_policy,
start_to_close_timeout=timedelta(seconds=60),
)
@@ -126,7 +281,8 @@ class Orchestrator:
Activities.process_slots,
{
**metadata,
- 'opc_servers': opc_servers,
+ 'connectors': connectors,
+ 'connector_tags': connector_tags,
'active_ingestors': active_ingestors,
'pipelines': pipeline_config,
},
diff --git a/pipelines_sample.json b/pipelines_sample.json
index a9441d3..73fd28d 100644
--- a/pipelines_sample.json
+++ b/pipelines_sample.json
@@ -16,25 +16,22 @@
],
"read_tags": [
{
- "server_id": "1",
+ "tag_id": 1,
"tag_name": "Counter",
- "tag_address": "ns=2;s=GatewayProSYS.ProSYS.Simulation.Counter",
"aggr_func": "lts",
"data_range": [1, 30],
"frequency": 30000
},
{
- "server_id": "1",
+ "tag_id": 2,
"tag_name": "Square",
- "tag_address": "ns=2;s=GatewayProSYS.ProSYS.Simulation.Square",
"aggr_func": "lts",
"data_range": [-2, 2],
"frequency": 30000
},
{
- "server_id": "1",
+ "tag_id": 3,
"tag_name": "Rollout",
- "tag_address": "ns=2;s=GatewayProSYS.ProSYS.Simulation.Rollout",
"aggr_func": "lts",
"data_range": [-2, 2],
"frequency": 30000
diff --git a/tests/orchestrator/activities/test_email.py b/tests/orchestrator/activities/test_email.py
index 23ec052..e74f5a9 100644
--- a/tests/orchestrator/activities/test_email.py
+++ b/tests/orchestrator/activities/test_email.py
@@ -85,7 +85,9 @@ def test_close(sientia_monitoring_mock, email):
email.close()
email.server.quit.assert_called_once()
- sientia_monitoring_mock.shutdown.assert_called_once()
+ # Count only this instance: an Email left by another test may be garbage-collected
+ # (Email.__del__ -> close) while the patch is active
+ assert sientia_monitoring_mock.shutdown.call_args_list.count(call(email)) == 1
def test_build_email_html(email):
@@ -324,3 +326,37 @@ def test_send_email(mimemultipart, mimetext, email):
)
assert email.try_send_email.call_count == 2
+
+
+@patch('orchestrator.activities.email.EmailBuilder')
+@patch('orchestrator.activities.email.smtplib')
+def test___init___without_smtp_server(smtplib, email_builder):
+ email = Email(
+ sender_email='test@test.com',
+ sender_password='test',
+ smtp_server=None,
+ smtp_port=587,
+ logger=MagicMock(),
+ notification_handler=MagicMock(),
+ metrics_controller=AsyncMock(),
+ )
+
+ smtplib.SMTP.assert_not_called()
+ assert not hasattr(email, 'server')
+
+
+@patch('orchestrator.activities.email.smtplib.SMTP')
+def test_try_send_email_reconnect_without_password(smtp, email):
+ email.sender_password = None
+ email.server.sendmail = MagicMock(side_effect=SMTPServerDisconnected('test'))
+ email.server.quit = MagicMock()
+
+ msg = MagicMock()
+
+ email.try_send_email(msg, 'test')
+
+ smtp.return_value.starttls.assert_not_called()
+ smtp.return_value.login.assert_not_called()
+ smtp.return_value.sendmail.assert_called_once_with(
+ 'test@test.com', 'test', msg.as_string.return_value
+ )
diff --git a/tests/orchestrator/activities/test_formatters.py b/tests/orchestrator/activities/test_formatters.py
index 320852c..d479379 100644
--- a/tests/orchestrator/activities/test_formatters.py
+++ b/tests/orchestrator/activities/test_formatters.py
@@ -5,7 +5,14 @@ from pandas import DataFrame
from pytest import fixture
from sientia_do.notifications.models import NotificationLevel
-from orchestrator.activities.formatters import Formatters
+from orchestrator.activities.formatters import Formatters, schedule_config_hash
+from orchestrator.utils.pipeline_configs import predictions_batch
+from tests.orchestrator.utils.tag_database_fixtures import (
+ CERTIFICATE,
+ NO_SECURITY,
+ SECURE,
+ secure_connector_fields,
+)
@fixture
@@ -140,7 +147,10 @@ def test_process_schedules(formatters):
}
mock_scouter.assert_called_once_with(input_data['pipelines'][0])
- mock_predictions_batch.assert_called_once_with(input_data['pipelines'][1])
+ # A predictions_batch gets its opc_output_config from its (here absent) write_tags
+ mock_predictions_batch.assert_called_once_with(
+ {**input_data['pipelines'][1], 'opc_output_config': {}}
+ )
mock_minimal_retrain.assert_called_once_with(input_data['pipelines'][2])
mock_drift.assert_called_once_with(input_data['pipelines'][3])
mock_simple_metrics.assert_called_once_with(input_data['pipelines'][4])
@@ -218,115 +228,408 @@ def test_process_schedules_with_invalid_workflow_type(formatters):
mock_scouter.assert_any_call(pipelines[2])
-@patch(
- 'orchestrator.activities.formatters.gather_read_tags',
- return_value={
- '1:test_tag_address': {
- 'server_id': '1',
- 'server_name': 'test_server_name',
- 'tag_address': 'test_tag_address',
- 'topics': ['raw_test_schedule'],
- 'frequency': 1000,
- },
- '2:test_tag_address2': {
- 'server_id': '2',
- 'server_name': 'test_server_name2',
- 'tag_address': 'test_tag_address2',
- 'topics': ['raw_test_schedule2'],
- 'frequency': 1000,
- },
- '2:test_tag_address3': {
- 'server_id': '2',
- 'server_name': 'test_server_name2',
- 'tag_address': 'test_tag_address3',
- 'topics': ['raw_test_schedule2'],
- 'frequency': 1000,
- },
+def _scouter_pipeline(schedule_name, model_id, read_tags):
+ return {
+ 'workflow_type': 'scouter',
+ 'schedule_name': schedule_name,
+ 'model_id': model_id,
+ 'model': {'name': f'model {model_id}'},
+ 'read_tags': read_tags,
+ }
+
+
+slot_connectors = [
+ {
+ 'connector_id': 'kepserver',
+ 'name': 'kepserver',
+ 'url': 'opc.tcp://kep:49320',
+ 'security': NO_SECURITY,
+ 'status': 'Active',
},
-)
-@patch('orchestrator.activities.formatters.build_tag_config')
-def test_process_slots(mock_build_tag_config, mock_gather_read_tags, formatters):
+]
+
+
+def _read_tag_doc(tag_id, address, tag_name, connector_id='kepserver'):
+ return {
+ 'tag_id': tag_id,
+ 'connector_id': connector_id,
+ 'address': address,
+ 'tag_name': tag_name,
+ 'category': 'Read',
+ 'status': 'Active',
+ }
+
+
+slot_connector_tags = [
+ _read_tag_doc(1, 'ns=2;s=A', 'a'),
+ _read_tag_doc(2, 'ns=2;s=B', 'b'),
+ _read_tag_doc(3, 'ns=2;s=C', 'c'),
+]
+
+
+def _slot_input(pipelines, active_ingestors, **extra):
+ return {
+ 'connectors': slot_connectors,
+ 'connector_tags': slot_connector_tags,
+ 'active_ingestors': active_ingestors,
+ 'pipelines': pipelines,
+ **metadata,
+ **extra,
+ }
+
+
+def test_process_slots(formatters):
+ """Slot v2 split by address: the names of an address never land in different slots"""
+ pipelines = [
+ _scouter_pipeline(
+ 'sin', 'm1', [{'tag_id': 1, 'frequency': 1000}, {'tag_id': 2, 'frequency': 1000}]
+ ),
+ _scouter_pipeline(
+ 'cos',
+ 'm2',
+ [{'tag_id': 1, 'tag_name': 'x', 'frequency': 500}, {'tag_id': 3, 'frequency': 4000}],
+ ),
+ ]
+
+ result = formatters.process_slots(
+ _slot_input(pipelines, ['heartbeat:ingestor:1', 'heartbeat:ingestor:2'])
+ )
+
+ server = {
+ 'server_id': 'kepserver',
+ 'name': 'kepserver',
+ 'url': 'opc.tcp://kep:49320',
+ 'security': NO_SECURITY,
+ 'certificate': None,
+ }
+ assert result == {
+ '1': {
+ 'kepserver': {
+ **server,
+ 'subscription_period_ms': 250,
+ 'tags': {
+ 'ns=2;s=A': {
+ 'frequency': 500,
+ 'active_read': {'enabled': False},
+ 'names': [
+ {
+ 'collection': 'raw_cos',
+ 'tag_name': 'x',
+ 'model_id': 'm2',
+ 'model_name': 'model m2',
+ },
+ {
+ 'collection': 'raw_sin',
+ 'tag_name': 'a',
+ 'model_id': 'm1',
+ 'model_name': 'model m1',
+ },
+ ],
+ },
+ 'ns=2;s=B': {
+ 'frequency': 1000,
+ 'active_read': {'enabled': False},
+ 'names': [
+ {
+ 'collection': 'raw_sin',
+ 'tag_name': 'b',
+ 'model_id': 'm1',
+ 'model_name': 'model m1',
+ },
+ ],
+ },
+ },
+ }
+ },
+ '2': {
+ 'kepserver': {
+ **server,
+ 'subscription_period_ms': 2000,
+ 'tags': {
+ 'ns=2;s=C': {
+ 'frequency': 4000,
+ 'active_read': {'enabled': False},
+ 'names': [
+ {
+ 'collection': 'raw_cos',
+ 'tag_name': 'c',
+ 'model_id': 'm2',
+ 'model_name': 'model m2',
+ },
+ ],
+ },
+ },
+ }
+ },
+ }
+ formatters.send_notification.assert_not_called()
+
+
+def test_process_slots_without_ingestors_builds_one_slot(formatters):
+ pipelines = [_scouter_pipeline('sin', 'm1', [{'tag_id': 1, 'frequency': 1000}])]
+
+ result = formatters.process_slots(_slot_input(pipelines, []))
+
+ assert list(result) == ['1']
+ assert list(result['1']['kepserver']['tags']) == ['ns=2;s=A']
+
+
+def test_process_slots_more_ingestors_than_addresses(formatters):
+ """Extra ingestors get empty slots, as before"""
+ pipelines = [_scouter_pipeline('sin', 'm1', [{'tag_id': 1, 'frequency': 1000}])]
+
+ result = formatters.process_slots(_slot_input(pipelines, ['i1', 'i2', 'i3']))
+
+ assert list(result) == ['1', '2', '3']
+ assert list(result['1']['kepserver']['tags']) == ['ns=2;s=A']
+ assert result['2'] == {}
+ assert result['3'] == {}
+
+
+def test_process_slots_with_tag_database(formatters):
+ """A read tag takes address, name and server from the tag database"""
+ pipelines = [
+ _scouter_pipeline('sin', 'm1', [{'tag_id': 7, 'frequency': 1000}]),
+ ]
input_data = {
- 'opc_servers': [
- {'id': '1', 'server_name': 'test_server_name', 'url': 'test_url', 'uri': 'test_uri'},
- {'id': '2', 'server_name': 'test_server_name2', 'url': 'test_url2', 'uri': 'test_uri2'},
+ 'connectors': [
+ {
+ 'connector_id': 3,
+ 'type': 'opc_ua',
+ 'name': 'plant',
+ 'url': 'opc.tcp://plant:4840',
+ 'subscription_period_ms': 1500,
+ 'status': 'Active',
+ **secure_connector_fields(),
+ }
],
- 'active_ingestors': ['test_active_ingestor1', 'test_active_ingestor2'],
- 'pipelines': 'test_gather_read_tags',
+ 'connector_tags': [
+ {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1001', 'tag_name': 'Pressure'}
+ ],
+ 'active_ingestors': ['i1'],
+ 'pipelines': pipelines,
**metadata,
}
- expected_opc_servers = {
- '1': {
- 'id': '1',
- 'server_name': 'test_server_name',
- 'url': 'test_url',
- 'uri': 'test_uri',
- },
- '2': {
- 'id': '2',
- 'server_name': 'test_server_name2',
- 'url': 'test_url2',
- 'uri': 'test_uri2',
- },
- }
+ result = formatters.process_slots(input_data)
- slot_mock = {
- 'test_server_name': {
- 'server_id': '1',
- 'name': 'test_server_name',
- 'url': 'test_url',
- 'server_uri': 'test_uri',
- 'cert_path': None,
- 'private_key_path': None,
- 'server_cert_path': None,
+ assert result == {
+ '1': {
+ 'plant': {
+ 'server_id': '3',
+ 'name': 'plant',
+ 'url': 'opc.tcp://plant:4840',
+ 'subscription_period_ms': 1500,
+ 'security': SECURE,
+ 'certificate': CERTIFICATE,
+ 'tags': {
+ 'ns=3;i=1001': {
+ 'frequency': 1000,
+ 'active_read': {'enabled': False},
+ 'names': [
+ {
+ 'collection': 'raw_sin',
+ 'tag_name': 'Pressure',
+ 'model_id': 'm1',
+ 'model_name': 'model m1',
+ }
+ ],
+ }
+ },
+ }
}
}
- mock_build_tag_config.return_value = (slot_mock, ['2'])
- result = formatters.process_slots(input_data)
+def test_process_slots_notifies_unresolved_tags_once(formatters):
+ """Each kind of unresolved read tag is notified once per run, not once per slot"""
+ pipelines = [
+ _scouter_pipeline(
+ 'sin',
+ 'm1',
+ [
+ {'tag_id': 1, 'frequency': 1000},
+ {'tag_name': 'b', 'frequency': 1000},
+ {'tag_id': 404, 'frequency': 1000},
+ {'tag_id': 8, 'frequency': 1000},
+ {'tag_id': 3},
+ ],
+ ),
+ ]
+ connector_tags = [*slot_connector_tags, _read_tag_doc(8, 'ns=3;i=8', 'Level', 'other')]
- tags = list(mock_gather_read_tags.return_value.values())
-
- mock_gather_read_tags.assert_called_once_with(input_data['pipelines'])
-
- mock_build_tag_config.assert_has_calls(
- [
- call(tags[:2], expected_opc_servers),
- call(tags[2:], expected_opc_servers),
- ]
+ result = formatters.process_slots(
+ _slot_input(pipelines, ['i1', 'i2'], connector_tags=connector_tags)
)
+ assert list(result['1']['kepserver']['tags']) == ['ns=2;s=A']
+ assert result['2'] == {}
formatters.send_notification.assert_has_calls(
[
call(
metadata=metadata['metadata'],
- notification_id='ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION',
- message='Servers 2 not found in opc_servers',
+ notification_id='ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+ message='Tags 404 not found among the active, complete Read tags of connector_tags',
block='orchestrator',
level=NotificationLevel.ERROR,
),
call(
metadata=metadata['metadata'],
- notification_id='ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION',
- message='Servers 2 not found in opc_servers',
+ notification_id='ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+ message='Connectors other not found, not active or incomplete in connectors',
+ block='orchestrator',
+ level=NotificationLevel.ERROR,
+ ),
+ call(
+ metadata=metadata['metadata'],
+ notification_id='ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION',
+ message=(
+ 'Read tags sin/3, sin/b have no integer tag_id, no integer frequency or an '
+ 'active_read without an integer cycle_multiple; or their pipeline lacks '
+ 'schedule_name, model_id or model.name'
+ ),
block='orchestrator',
level=NotificationLevel.ERROR,
),
]
)
+ assert formatters.send_notification.call_count == 3
- assert result == {
- '1': slot_mock,
- '2': slot_mock,
+
+def test_process_slots_warns_about_a_missing_certificate(formatters):
+ """A client_certificate_id without an active client certificate: the server goes into
+ the slot without a certificate, and a warning names the connector (D5)"""
+ connectors = [
+ {**slot_connectors[0], **secure_connector_fields(certificate_id=9, certificates=[])},
+ {
+ **slot_connectors[0],
+ 'connector_id': 'plant',
+ 'name': 'plant',
+ **secure_connector_fields(),
+ },
+ ]
+ connector_tags = [*slot_connector_tags, _read_tag_doc(7, 'ns=3;i=7', 'P', 'plant')]
+ pipelines = [
+ _scouter_pipeline('sin', 'm1', [{'tag_id': 1, 'frequency': 1000}]),
+ _scouter_pipeline('cos', 'm2', [{'tag_id': 7, 'frequency': 1000}]),
+ ]
+
+ result = formatters.process_slots(
+ _slot_input(pipelines, [], connectors=connectors, connector_tags=connector_tags)
+ )
+
+ assert result['1']['kepserver']['certificate'] is None
+ assert result['1']['kepserver']['security'] == SECURE
+ assert result['1']['plant']['certificate'] == CERTIFICATE
+ formatters.send_notification.assert_called_once_with(
+ metadata=metadata['metadata'],
+ notification_id='ORCHESTRATOR_CERTIFICATE_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+ message='Connectors kepserver have a security.client_certificate_id without an active '
+ 'client certificate in opc_certificates: they go into the slot without a certificate, '
+ 'so without security',
+ block='orchestrator',
+ level=NotificationLevel.WARNING,
+ )
+
+
+def test_process_slots_survives_malformed_tag_database(formatters):
+ """An incomplete connector or tag leaves only its own read tags out of the slot"""
+ pipelines = [
+ _scouter_pipeline(
+ 'sin',
+ 'm1',
+ [
+ {'tag_id': 1, 'frequency': 1000},
+ {'tag_id': 7, 'frequency': 1000},
+ {'tag_id': 8, 'frequency': 1000},
+ ],
+ ),
+ ]
+ connectors = [*slot_connectors, {'connector_id': 3, 'name': 'plant', 'security': NO_SECURITY}]
+ connector_tags = [
+ *slot_connector_tags,
+ {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1', 'tag_name': 'P'},
+ {'tag_id': 8, 'connector_id': 3, 'tag_name': 'no address'},
+ ]
+
+ result = formatters.process_slots(
+ _slot_input(pipelines, [], connectors=connectors, connector_tags=connector_tags)
+ )
+
+ assert list(result['1']) == ['kepserver']
+ notified = {c.kwargs['notification_id'] for c in formatters.send_notification.call_args_list}
+ assert notified == {
+ 'ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+ 'ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION',
}
+def test_process_slots_is_stable_across_pipeline_order(formatters):
+ """The same pipelines in another order give the same slots (O6)"""
+ pipelines = [
+ _scouter_pipeline('sin', 'm1', [{'tag_id': 1, 'frequency': 1000}]),
+ _scouter_pipeline(
+ 'cos',
+ 'm2',
+ [{'tag_id': 2, 'frequency': 1000}, {'tag_id': 1, 'tag_name': 'x', 'frequency': 1000}],
+ ),
+ ]
+
+ def run(order):
+ return formatters.process_slots(_slot_input(order, ['i1', 'i2']))
+
+ assert json.dumps(run(pipelines)) == json.dumps(run(list(reversed(pipelines))))
+
+
+def test_process_schedules_names_scouter_tags_like_the_slot(formatters):
+ """model_tags and the slot's names[] take the tag name from the same source (SC1)"""
+ pipeline = {
+ **_scouter_pipeline('sin', 'm1', [{'tag_id': 7, 'frequency': 1000, 'aggr_func': 'avg'}]),
+ 'updated_at': '2021-01-01',
+ }
+ connector_tags = [
+ {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1001', 'tag_name': 'Pressure'}
+ ]
+ connectors = [
+ {
+ 'connector_id': 3,
+ 'name': 'plant',
+ 'url': 'opc.tcp://plant:4840',
+ 'security': NO_SECURITY,
+ }
+ ]
+
+ schedules = formatters.process_schedules(
+ {'pipelines': [pipeline], 'connector_tags': connector_tags, **metadata}
+ )
+ slots = formatters.process_slots(
+ {
+ 'connectors': connectors,
+ 'connector_tags': connector_tags,
+ 'active_ingestors': [],
+ 'pipelines': [pipeline],
+ **metadata,
+ }
+ )
+
+ model_tags = schedules['scouter']['sin']['model_tags']
+ names = slots['1']['plant']['tags']['ns=3;i=1001']['names']
+ assert list(model_tags) == ['Pressure']
+ assert model_tags['Pressure']['aggr_func'] == 'avg'
+ assert [name['tag_name'] for name in names] == ['Pressure']
+ assert names[0]['collection'] == schedules['scouter']['sin']['topic']
+
+
def test_format_schedule_config(formatters):
input_data = {
'schedule_config': [
- {'namespace': 'test_namespace1', 'schedule_name': 'test1', 'updated_at': '2021-01-01'},
+ {
+ 'namespace': 'test_namespace1',
+ 'schedule_name': 'test1',
+ 'updated_at': '2021-01-01',
+ 'config_hash': 'abc',
+ },
+ # Deployed before config_hash existed
{'namespace': 'test_namespace2', 'schedule_name': 'test2', 'updated_at': '2021-01-02'},
],
**metadata,
@@ -335,33 +638,34 @@ def test_format_schedule_config(formatters):
result = formatters.format_schedule_config(input_data)
assert result == {
- 'test_namespace1': {'test1': '2021-01-01'},
- 'test_namespace2': {'test2': '2021-01-02'},
+ 'test_namespace1': {'test1': {'updated_at': '2021-01-01', 'config_hash': 'abc'}},
+ 'test_namespace2': {'test2': {'updated_at': '2021-01-02', 'config_hash': None}},
}
def test_create_schedule_config(formatters):
+ to_create = {'frequency': 60, 'data': {'test': 'test'}, 'updated_at': '2021-01-03'}
+ to_update = {'frequency': 60, 'data': {'test': 'test2'}, 'updated_at': '2021-01-04'}
+ unchanged = {'frequency': 60, 'data': {'test': 'test3'}, 'updated_at': '2021-01-01'}
input_data = {
'current_schedule_config': {
'scouter': {
- 'test_schedule_name_to_delete': '2021-01-01',
- 'test_schedule_name_to_update': '2021-01-02',
+ 'test_schedule_name_to_delete': {'updated_at': '2021-01-01', 'config_hash': 'x'},
+ 'test_schedule_name_to_update': {
+ 'updated_at': '2021-01-02',
+ 'config_hash': schedule_config_hash(to_update),
+ },
+ 'test_schedule_name_unchanged': {
+ 'updated_at': '2021-01-02',
+ 'config_hash': schedule_config_hash(unchanged),
+ },
}
},
'schedule_config': {
- 'laborious': {
- 'test_schedule_name_to_create': {
- 'frequency': 60,
- 'data': {'test': 'test'},
- 'updated_at': '2021-01-03',
- }
- },
+ 'laborious': {'test_schedule_name_to_create': to_create},
'scouter': {
- 'test_schedule_name_to_update': {
- 'frequency': 60,
- 'data': {'test': 'test2'},
- 'updated_at': '2021-01-04',
- }
+ 'test_schedule_name_to_update': to_update,
+ 'test_schedule_name_unchanged': unchanged,
},
},
}
@@ -372,9 +676,8 @@ def test_create_schedule_config(formatters):
'to_create': {
'laborious': {
'test_schedule_name_to_create': {
- 'frequency': 60,
- 'data': {'test': 'test'},
- 'updated_at': '2021-01-03',
+ **to_create,
+ 'config_hash': schedule_config_hash(to_create),
}
},
'scouter': {},
@@ -382,9 +685,8 @@ def test_create_schedule_config(formatters):
'to_update': {
'scouter': {
'test_schedule_name_to_update': {
- 'frequency': 60,
- 'data': {'test': 'test2'},
- 'updated_at': '2021-01-04',
+ **to_update,
+ 'config_hash': schedule_config_hash(to_update),
}
},
'laborious': {},
@@ -393,6 +695,56 @@ def test_create_schedule_config(formatters):
}
+def test_schedule_config_hash_ignores_updated_at_and_key_order():
+ schedule = {'frequency': 60, 'data': {'a': 1, 'b': 2}, 'updated_at': '2021-01-01'}
+ same = {'data': {'b': 2, 'a': 1}, 'updated_at': '2021-02-02', 'frequency': 60}
+
+ assert schedule_config_hash(schedule) == schedule_config_hash(same)
+ assert schedule_config_hash(schedule) == schedule_config_hash(
+ {**schedule, 'config_hash': 'stale'}
+ )
+ assert schedule_config_hash(schedule) != schedule_config_hash({**schedule, 'frequency': 30})
+
+
+def test_create_schedule_config_updates_on_config_hash_change(formatters):
+ """Same updated_at, different content (e.g. a pipeline edited by hand): redeploy"""
+ schedule = {'frequency': 60, 'data': {'test': 'new'}, 'updated_at': '2021-01-01'}
+
+ result = formatters.create_schedule_config(
+ {
+ 'current_schedule_config': {
+ 'scouter': {
+ 's': {
+ 'updated_at': '2021-01-02',
+ 'config_hash': schedule_config_hash({**schedule, 'data': {'test': 'old'}}),
+ }
+ }
+ },
+ 'schedule_config': {'scouter': {'s': schedule}, 'laborious': {}},
+ }
+ )
+
+ assert set(result['to_update']['scouter']) == {'s'}
+
+
+def test_create_schedule_config_updates_on_newer_updated_at(formatters):
+ """Bumping updated_at forces a redeploy even when nothing else changed"""
+ schedule = {'frequency': 60, 'updated_at': '2021-01-05'}
+
+ result = formatters.create_schedule_config(
+ {
+ 'current_schedule_config': {
+ 'scouter': {
+ 's': {'updated_at': '2021-01-02', 'config_hash': schedule_config_hash(schedule)}
+ }
+ },
+ 'schedule_config': {'scouter': {'s': schedule}, 'laborious': {}},
+ }
+ )
+
+ assert set(result['to_update']['scouter']) == {'s'}
+
+
def test_create_slot_config(formatters):
input_data = {
'current_slot_config': {
@@ -410,6 +762,31 @@ def test_create_slot_config(formatters):
}
+def test_create_slot_config_skips_unchanged_slots(formatters):
+ """Only new or changed slots are written (O6); current slots come keyed by Redis key"""
+ slot = {'kepserver': {'server_id': '1', 'tags': {'ns=2;s=A': {'names': []}}}}
+ changed = {'kepserver': {'server_id': '1', 'tags': {'ns=2;s=B': {'names': []}}}}
+ input_data = {
+ 'current_slot_config': {
+ 'slot:opc_tags:1': slot,
+ 'slot:opc_tags:2': changed,
+ },
+ 'slot_config': {'1': slot, '2': slot, '3': {}},
+ }
+
+ result = formatters.create_slot_config(input_data)
+
+ assert result == {'to_delete': [], 'to_insert': {'2': slot, '3': {}}}
+
+
+def test_create_slot_config_without_current_slots(formatters):
+ slot_config = {'1': {'kepserver': {'tags': {}}}, '2': {}}
+
+ result = formatters.create_slot_config({'current_slot_config': {}, 'slot_config': slot_config})
+
+ assert result == {'to_delete': [], 'to_insert': slot_config}
+
+
def test_send_success_report(formatters):
formatters.send_success_report(
metadata=metadata,
@@ -1059,7 +1436,9 @@ def test_create_schedule_config_warns_on_frequency_alert_below_floor(formatters)
def test_create_schedule_config_warns_on_frequency_alert_below_floor_on_update(formatters):
input_data = _schedule_config_with_frequency_alert('30s')
- input_data['current_schedule_config'] = {'laborious': {'m1-simple_metrics-60m': '2026-10-06'}}
+ input_data['current_schedule_config'] = {
+ 'laborious': {'m1-simple_metrics-60m': {'updated_at': '2026-10-06', 'config_hash': None}}
+ }
result = formatters.create_schedule_config(input_data)
@@ -1111,8 +1490,395 @@ def test_create_schedule_config_warns_on_invalid_chunk_period(formatters):
def test_create_schedule_config_no_chunk_period_warning_when_unchanged(formatters):
input_data = _drift_schedule_config('hour')
- input_data['current_schedule_config'] = {'laborious': {'m1-drift-1m': '2026-10-27'}}
+ schedule = input_data['schedule_config']['laborious']['m1-drift-1m']
+ input_data['current_schedule_config'] = {
+ 'laborious': {
+ 'm1-drift-1m': {
+ 'updated_at': '2026-10-27',
+ 'config_hash': schedule_config_hash(schedule),
+ }
+ }
+ }
formatters.create_schedule_config(input_data)
formatters.send_notification.assert_not_called()
+
+
+# Write tags through the tag database (SIENTIAPDE-1974)
+
+PLANT_CONNECTOR = {
+ 'connector_id': 'plant',
+ 'name': 'plant',
+ 'url': 'opc.tcp://plant:4840',
+ 'security': NO_SECURITY,
+ 'status': 'Active',
+}
+
+
+def _write_tag(tag_id, address, category='Write', **extra):
+ return {
+ 'tag_id': tag_id,
+ 'connector_id': 'plant',
+ 'address': address,
+ 'tag_name': f'tag-{tag_id}',
+ 'category': category,
+ 'status': 'Active',
+ **extra,
+ }
+
+
+def _batch_pipeline(schedule_name, write_tags, updated_at='2026-10-01 00:00:00.000+0000'):
+ return {
+ 'workflow_type': 'predictions_batch',
+ 'schedule_name': schedule_name,
+ 'model_id': 'model_id',
+ 'model': {'name': 'model_name'},
+ 'query': 'SELECT 1',
+ 'write_tags': write_tags,
+ 'updated_at': updated_at,
+ }
+
+
+def test_process_schedules_resolves_write_tags_through_tag_database(formatters):
+ batch = _batch_pipeline('batch', [{'tag_id': 77}, {'tag_id': 78}])
+
+ result = formatters.process_schedules(
+ {
+ 'pipelines': [batch],
+ 'connectors': [PLANT_CONNECTOR],
+ 'connector_tags': [
+ _write_tag(77, 'ns=2;s=P1'),
+ _write_tag(78, 'ns=2;s=C1', category='Prediction Confidence', data_type='int'),
+ ],
+ **metadata,
+ }
+ )
+
+ schedule = result['laborious']['batch']
+ assert schedule['opc_output_config'] == {
+ 'plant': {
+ 'prediction_tags': {'ns=2;s=P1': {'data_type': 'float'}},
+ 'confidence_tags': {'ns=2;s=C1': {'data_type': 'int'}},
+ }
+ }
+ assert 'write_tags' not in schedule
+ formatters.send_notification.assert_not_called()
+
+
+def test_process_schedules_notifies_unresolved_write_tags_once(formatters):
+ pipelines = [
+ _batch_pipeline('a', [{'tag_id': 404}, {'tag_id': 7}, {'tag_id': 9}]),
+ _batch_pipeline('b', [{'tag_name': 'L'}]),
+ ]
+
+ result = formatters.process_schedules(
+ {
+ 'pipelines': pipelines,
+ 'connectors': [PLANT_CONNECTOR],
+ 'connector_tags': [
+ _write_tag(7, 'ns=2;s=R', category='Read'),
+ {**_write_tag(9, 'ns=2;s=X'), 'connector_id': 'gone'},
+ ],
+ **metadata,
+ }
+ )
+
+ # Unresolved tags are left out; the pipelines are still scheduled
+ assert result['laborious']['a']['opc_output_config'] == {}
+ assert result['laborious']['b']['opc_output_config'] == {}
+ formatters.send_notification.assert_has_calls(
+ [
+ call(
+ metadata=metadata['metadata'],
+ notification_id='ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION',
+ message=(
+ 'Write tags a/404, a/7 not found among the active, complete Write and '
+ 'Prediction Confidence tags of connector_tags'
+ ),
+ block='orchestrator',
+ level=NotificationLevel.ERROR,
+ ),
+ call(
+ metadata=metadata['metadata'],
+ notification_id=(
+ 'ORCHESTRATOR_WRITE_TAG_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION'
+ ),
+ message='Connectors a/gone of write tags not found, not active or incomplete in connectors',
+ block='orchestrator',
+ level=NotificationLevel.ERROR,
+ ),
+ call(
+ metadata=metadata['metadata'],
+ notification_id='ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION',
+ message=(
+ "Write tags b/{'tag_name': 'L'} have no integer tag_id or a data_type the OPC "
+ 'Gateway does not accept'
+ ),
+ block='orchestrator',
+ level=NotificationLevel.ERROR,
+ ),
+ ]
+ )
+ assert formatters.send_notification.call_count == 3
+
+
+def test_tag_database_changes_redeploy_through_config_hash(formatters):
+ """
+ A tag changed or deleted in the tag database (without bumping any updated_at)
+ changes the resolved schedule, so its config_hash, and redeploys the pipelines that
+ use it. A change that does not reach the schedule (e.g. a description) does not.
+ """
+ pipelines = [
+ _batch_pipeline('moved', [{'tag_id': 77}]),
+ _batch_pipeline('deleted', [{'tag_id': 78}]),
+ _batch_pipeline('same', [{'tag_id': 79}]),
+ {
+ 'workflow_type': 'scouter',
+ 'schedule_name': 'renamed',
+ 'model_id': 'm1',
+ 'model': {'name': 'model m1'},
+ 'read_tags': [{'tag_id': 7, 'frequency': 1000}],
+ 'updated_at': '2026-10-01 00:00:00.000+0000',
+ },
+ ]
+ before = [
+ _write_tag(77, 'ns=2;s=P1'),
+ _write_tag(78, 'ns=2;s=P2'),
+ _write_tag(79, 'ns=2;s=P3', description='old'),
+ _write_tag(7, 'ns=3;i=1001', category='Read'),
+ ]
+ after = [
+ _write_tag(77, 'ns=2;s=P1-moved'),
+ _write_tag(79, 'ns=2;s=P3', description='new'),
+ {**_write_tag(7, 'ns=3;i=1001', category='Read'), 'tag_name': 'renamed-tag'},
+ ]
+
+ def process(connector_tags):
+ return formatters.process_schedules(
+ {
+ 'pipelines': pipelines,
+ 'connectors': [PLANT_CONNECTOR],
+ 'connector_tags': connector_tags,
+ **metadata,
+ }
+ )
+
+ deployed = process(before)
+ # Last orchestrated after every pipeline's updated_at, with the hash of `before`
+ current = {
+ namespace: {
+ name: {
+ 'updated_at': '2026-10-05 00:00:00.000+0000',
+ 'config_hash': schedule_config_hash(schedule),
+ }
+ for name, schedule in schedules.items()
+ }
+ for namespace, schedules in deployed.items()
+ }
+
+ unchanged = formatters.create_schedule_config(
+ {'current_schedule_config': current, 'schedule_config': process(before), **metadata}
+ )
+ actions = formatters.create_schedule_config(
+ {'current_schedule_config': current, 'schedule_config': process(after), **metadata}
+ )
+
+ assert unchanged['to_update'] == {'scouter': {}, 'laborious': {}}
+ assert set(actions['to_update']['laborious']) == {'moved', 'deleted'}
+ assert set(actions['to_update']['scouter']) == {'renamed'}
+ assert actions['to_update']['laborious']['deleted']['opc_output_config'] == {}
+
+
+def test_process_schedules_without_updated_at_redeploys_every_run(formatters):
+ batch = _batch_pipeline('batch', [{'tag_id': 77}])
+ del batch['updated_at']
+
+ with patch(
+ 'orchestrator.activities.formatters.now',
+ return_value=MagicMock(strftime=MagicMock(return_value='2026-10-28 12:00:00.000+0000')),
+ ):
+ result = formatters.process_schedules(
+ {
+ 'pipelines': [batch],
+ 'connectors': [PLANT_CONNECTOR],
+ 'connector_tags': [
+ _write_tag(77, 'ns=2;s=P1', updated_at='2026-10-10 00:00:00.000+0000')
+ ],
+ **metadata,
+ }
+ )
+
+ assert result['laborious']['batch']['updated_at'] == '2026-10-28 12:00:00.000+0000'
+
+
+def test_predictions_batch_schedule_matches_the_builder(formatters):
+ """process_schedules only adds the resolved opc_output_config to the builder's input"""
+ batch = _batch_pipeline('batch', [{'tag_id': 77}])
+
+ result = formatters.process_schedules(
+ {
+ 'pipelines': [batch],
+ 'connectors': [PLANT_CONNECTOR],
+ 'connector_tags': [_write_tag(77, 'ns=2;s=P1')],
+ **metadata,
+ }
+ )
+
+ resolved = {
+ **batch,
+ 'opc_output_config': {'plant': {'prediction_tags': {'ns=2;s=P1': {'data_type': 'float'}}}},
+ }
+ assert json.dumps(result['laborious']['batch'], sort_keys=True) == json.dumps(
+ {**predictions_batch(resolved), 'updated_at': batch['updated_at']}, sort_keys=True
+ )
+
+
+def test_format_schedule_config_groups_schedules_of_one_namespace(formatters):
+ result = formatters.format_schedule_config(
+ {
+ 'schedule_config': [
+ {'namespace': 'laborious', 'schedule_name': 'a', 'updated_at': '2021-01-01'},
+ {'namespace': 'laborious', 'schedule_name': 'b', 'updated_at': '2021-01-02'},
+ ],
+ **metadata,
+ }
+ )
+
+ assert result == {
+ 'laborious': {
+ 'a': {'updated_at': '2021-01-01', 'config_hash': None},
+ 'b': {'updated_at': '2021-01-02', 'config_hash': None},
+ }
+ }
+
+
+def test_report_schedule_orchestration_success_only_and_empty_types(formatters):
+ formatters.send_success_report = MagicMock()
+ formatters.send_error_report = MagicMock()
+
+ formatters.report_schedule_orchestration(
+ {
+ **metadata,
+ 'created_schedules': [
+ {'namespace': 'laborious', 'schedule_name': 'a', 'success': True}
+ ],
+ 'updated_schedules': [],
+ 'deleted_schedules': [],
+ }
+ )
+
+ formatters.send_success_report.assert_called_once()
+ assert formatters.send_success_report.call_args.kwargs['notification_id'] == (
+ 'REPORT_ORCHESTRATION_CREATED_SCHEDULES'
+ )
+ formatters.send_error_report.assert_not_called()
+
+
+def test_report_schedule_orchestration_errors_only(formatters):
+ formatters.send_success_report = MagicMock()
+ formatters.send_error_report = MagicMock()
+
+ formatters.report_schedule_orchestration(
+ {
+ **metadata,
+ 'created_schedules': [],
+ 'updated_schedules': [],
+ 'deleted_schedules': [
+ {
+ 'namespace': 'scouter',
+ 'schedule_name': 'b',
+ 'success': False,
+ 'message': 'boom',
+ 'attachment': None,
+ }
+ ],
+ }
+ )
+
+ formatters.send_success_report.assert_not_called()
+ formatters.send_error_report.assert_called_once()
+ assert formatters.send_error_report.call_args.kwargs['notification_id'] == (
+ 'REPORT_ORCHESTRATION_DELETED_SCHEDULES_ERROR'
+ )
+
+
+def test_report_slot_orchestration_success_only_inserts(formatters):
+ formatters.send_success_report = MagicMock()
+ formatters.send_error_report = MagicMock()
+
+ formatters.report_slot_orchestration(
+ {**metadata, 'inserted_slots': {'1': {'success': True}}, 'deleted_slots': {}}
+ )
+
+ formatters.send_success_report.assert_called_once_with(
+ metadata=metadata['metadata'],
+ message='Inserted slots: \n 1',
+ notification_id='REPORT_ORCHESTRATION_INSERTED_SLOTS',
+ )
+ formatters.send_error_report.assert_not_called()
+
+
+def test_report_slot_orchestration_errors_only_deletes(formatters):
+ formatters.send_success_report = MagicMock()
+ formatters.send_error_report = MagicMock()
+ deleted = {'2': {'success': False, 'error': 'boom'}}
+
+ formatters.report_slot_orchestration(
+ {**metadata, 'inserted_slots': {}, 'deleted_slots': deleted}
+ )
+
+ formatters.send_success_report.assert_not_called()
+ formatters.send_error_report.assert_called_once_with(
+ metadata=metadata['metadata'],
+ message='Failed to delete slots: \n 2',
+ notification_id='REPORT_ORCHESTRATION_DELETED_SLOTS',
+ attachment=deleted,
+ )
+
+
+def test_report_slot_orchestration_errors_only_inserts_success_only_deletes(formatters):
+ formatters.send_success_report = MagicMock()
+ formatters.send_error_report = MagicMock()
+
+ formatters.report_slot_orchestration(
+ {
+ **metadata,
+ 'inserted_slots': {'1': {'success': False, 'error': 'boom'}},
+ 'deleted_slots': {'2': {'success': True}},
+ }
+ )
+
+ assert [c.kwargs['notification_id'] for c in formatters.send_error_report.call_args_list] == [
+ 'REPORT_ORCHESTRATION_INSERTED_SLOTS'
+ ]
+ assert [c.kwargs['notification_id'] for c in formatters.send_success_report.call_args_list] == [
+ 'REPORT_ORCHESTRATION_DELETED_SLOTS'
+ ]
+
+
+def test_format_log_report_same_notification_twice_in_one_group(formatters):
+ notification = {
+ 'notification_id': 'n1',
+ 'trigger': 't1',
+ 'timestamp': '2021-01-01',
+ 'message': 'm',
+ 'level': 'ERROR',
+ 'block': 'b',
+ 'pipeline': 'p',
+ 'project': 'pr',
+ 'model_name': 'mn',
+ 'model_id': 'mi',
+ }
+
+ result = formatters.format_log_report(
+ {
+ **metadata,
+ 'receiver_groups': {
+ 'g1': {'notifications': [notification, dict(notification)], 'status': 'sent'}
+ },
+ 'mail_type': 'alerts',
+ }
+ )
+
+ assert [row['groups'] for row in DataFrame(result).to_dict('records')] == [['g1']]
diff --git a/tests/orchestrator/activities/test_mongo_db.py b/tests/orchestrator/activities/test_mongo_db.py
index 6feda41..205ff2a 100644
--- a/tests/orchestrator/activities/test_mongo_db.py
+++ b/tests/orchestrator/activities/test_mongo_db.py
@@ -1,14 +1,14 @@
from datetime import UTC, datetime
-from unittest.mock import ANY, AsyncMock, MagicMock, patch
+from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
-from pytest import fixture, mark
+import pytest
from sientia_do.notifications.models import NotificationLevel
from sientia_do.temporal.constants import DATETIME_FORMAT_MS_WITH_TZ
from orchestrator.activities.mongo_db import MongoDB
-@fixture
+@pytest.fixture
@patch('orchestrator.activities.mongo_db.MongoDBRepository')
def mongo_db(mongo_mock):
mongo = MongoDB(
@@ -192,6 +192,30 @@ def test_aggregate_documents_in_mongodb_success(mongo_db):
)
+def test_aggregate_documents_in_mongodb_formats_only_datetimes(mongo_db):
+ """A timestamp field stored as text, or missing, is left as it is"""
+ mongo_db.mongo_db_repository.aggregate = MagicMock(
+ return_value=[
+ {'tag_id': 1, 'updated_at': datetime(2026, 10, 1, 12, 0, 0)},
+ {'tag_id': 2, 'updated_at': '2026-10-02 00:00:00.000+0000'},
+ {'tag_id': 3},
+ ]
+ )
+
+ result = mongo_db.aggregate_documents_in_mongodb(
+ {
+ 'query': {'collection': 'connector_tags', 'aggregation': [{'$match': {}}]},
+ 'timestamp_fields': ['updated_at'],
+ }
+ )
+
+ assert result == [
+ {'tag_id': 1, 'updated_at': '2026-10-01 12:00:00.000000+0000'},
+ {'tag_id': 2, 'updated_at': '2026-10-02 00:00:00.000+0000'},
+ {'tag_id': 3},
+ ]
+
+
def test_aggregate_documents_in_mongodb_failure(mongo_db):
input_data = {
'collection': 'test_collection',
@@ -246,56 +270,59 @@ def test_aggregate_documents_in_mongodb_missing_aggregation(mongo_db):
def test_update_pipelines_timestamps_success(now_mock, mongo_db):
input_data = {
'updated_pipelines': [
- {'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
+ {'schedule_name': 'test1', 'namespace': 'test1', 'success': True, 'config_hash': 'h1'},
+ # Updated by a run started before config_hash
{'schedule_name': 'test2', 'namespace': 'test2', 'success': True},
]
}
- mongo_db.mongo_db_repository.update_many = MagicMock(return_value=MagicMock())
+ mongo_db.mongo_db_repository.update_one = MagicMock(return_value=1)
mongo_db.update_pipelines_timestamps(input_data)
- mongo_db.mongo_db_repository.update_many.assert_called_once_with(
- 'orchestrated_schedules',
- {
- '$or': [
- {'schedule_name': 'test1', 'namespace': 'test1'},
- {'schedule_name': 'test2', 'namespace': 'test2'},
- ]
- },
- {'$set': {'updated_at': now_mock.return_value}},
- {},
- )
+ assert mongo_db.mongo_db_repository.update_one.call_args_list == [
+ call(
+ 'orchestrated_schedules',
+ {'schedule_name': 'test1', 'namespace': 'test1'},
+ {'$set': {'updated_at': now_mock.return_value, 'config_hash': 'h1'}},
+ {},
+ ),
+ call(
+ 'orchestrated_schedules',
+ {'schedule_name': 'test2', 'namespace': 'test2'},
+ {'$set': {'updated_at': now_mock.return_value, 'config_hash': None}},
+ {},
+ ),
+ ]
@patch('orchestrator.activities.mongo_db.now')
-@mark.parametrize(
+@pytest.mark.parametrize(
'pipelines', [[], [{'schedule_name': 'test1', 'namespace': 'test1', 'success': False}]]
)
def test_update_pipelines_timestamps_all_failed_stamps_nothing(now_mock, mongo_db, pipelines):
input_data = {'updated_pipelines': pipelines}
- mongo_db.mongo_db_repository.update_many = MagicMock()
+ mongo_db.mongo_db_repository.update_one = MagicMock()
mongo_db.update_pipelines_timestamps(input_data)
- # An empty filter would match and stamp every tracked schedule, so the failed one
- # would never be retried.
- mongo_db.mongo_db_repository.update_many.assert_not_called()
+ # A failed update is not stamped, so it is retried on the next tick
+ mongo_db.mongo_db_repository.update_one.assert_not_called()
@patch('orchestrator.activities.mongo_db.now')
def test_update_pipelines_timestamps_mixed_stamps_only_successes(now_mock, mongo_db):
input_data = {
'updated_pipelines': [
- {'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
- {'schedule_name': 'test2', 'namespace': 'test2', 'success': False},
+ {'schedule_name': 'test1', 'namespace': 'test1', 'success': True, 'config_hash': 'h1'},
+ {'schedule_name': 'test2', 'namespace': 'test2', 'success': False, 'config_hash': 'h2'},
]
}
- mongo_db.mongo_db_repository.update_many = MagicMock()
+ mongo_db.mongo_db_repository.update_one = MagicMock()
mongo_db.update_pipelines_timestamps(input_data)
- mongo_db.mongo_db_repository.update_many.assert_called_once_with(
+ mongo_db.mongo_db_repository.update_one.assert_called_once_with(
'orchestrated_schedules',
- {'$or': [{'schedule_name': 'test1', 'namespace': 'test1'}]},
- {'$set': {'updated_at': now_mock.return_value}},
+ {'schedule_name': 'test1', 'namespace': 'test1'},
+ {'$set': {'updated_at': now_mock.return_value, 'config_hash': 'h1'}},
{},
)
@@ -310,15 +337,17 @@ def test_update_pipelines_timestamps_failure(now_mock, mongo_db):
**metadata,
}
- mongo_db.mongo_db_repository.update_many.side_effect = Exception('Error')
+ # The first update fails: the second is still tried, and only the failure is notified
+ mongo_db.mongo_db_repository.update_one.side_effect = [Exception('Error'), 1]
try:
mongo_db.update_pipelines_timestamps(input_data)
- except Exception as e:
- assert str(e) == 'Error'
+ except RuntimeError as e:
+ assert str(e) == 'Failed to update 1 of 2 pipelines timestamps'
+ assert mongo_db.mongo_db_repository.update_one.call_count == 2
mongo_db.send_notification.assert_called_once_with(
metadata=metadata['metadata'],
notification_id='MONGODB_UPDATE_PIPELINES_ERROR',
- message='Failed to update pipelines timestamps: Error',
+ message='Failed to update pipelines timestamps: test1/test1: Error',
block='update_pipelines_timestamps',
level=NotificationLevel.ERROR,
attachment_content=ANY,
@@ -328,11 +357,37 @@ def test_update_pipelines_timestamps_failure(now_mock, mongo_db):
raise AssertionError('Expected an exception to be raised')
+@patch('orchestrator.activities.mongo_db.now')
+def test_update_pipelines_timestamps_failures_notified_together(now_mock, mongo_db):
+ input_data = {
+ 'updated_pipelines': [
+ {'schedule_name': 'test1', 'namespace': 'ns', 'success': True},
+ {'schedule_name': 'test2', 'namespace': 'ns', 'success': True},
+ {'schedule_name': 'test3', 'namespace': 'ns', 'success': True},
+ ],
+ **metadata,
+ }
+ mongo_db.mongo_db_repository.update_one.side_effect = [
+ Exception('down'),
+ 1,
+ Exception('timeout'),
+ ]
+
+ with pytest.raises(RuntimeError, match='Failed to update 2 of 3 pipelines timestamps'):
+ mongo_db.update_pipelines_timestamps(input_data)
+
+ assert mongo_db.mongo_db_repository.update_one.call_count == 3
+ mongo_db.send_notification.assert_called_once()
+ assert mongo_db.send_notification.call_args.kwargs['message'] == (
+ 'Failed to update pipelines timestamps: ns/test1: down; ns/test3: timeout'
+ )
+
+
@patch('orchestrator.activities.mongo_db.now')
def test_create_pipelines_timestamps_success(now_mock, mongo_db):
input_data = {
'created_pipelines': [
- {'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
+ {'schedule_name': 'test1', 'namespace': 'test1', 'success': True, 'config_hash': 'h1'},
{'schedule_name': 'test2', 'namespace': 'test2', 'success': True},
]
}
@@ -341,8 +396,18 @@ def test_create_pipelines_timestamps_success(now_mock, mongo_db):
mongo_db.mongo_db_repository.insert_many.assert_called_once_with(
'orchestrated_schedules',
[
- {'schedule_name': 'test1', 'namespace': 'test1', 'updated_at': now_mock.return_value},
- {'schedule_name': 'test2', 'namespace': 'test2', 'updated_at': now_mock.return_value},
+ {
+ 'schedule_name': 'test1',
+ 'namespace': 'test1',
+ 'updated_at': now_mock.return_value,
+ 'config_hash': 'h1',
+ },
+ {
+ 'schedule_name': 'test2',
+ 'namespace': 'test2',
+ 'updated_at': now_mock.return_value,
+ 'config_hash': None,
+ },
],
{},
)
@@ -397,7 +462,7 @@ def test_delete_pipelines_timestamps_success(now_mock, mongo_db):
)
-@mark.parametrize(
+@pytest.mark.parametrize(
'pipelines', [[], [{'schedule_name': 'test1', 'namespace': 'test1', 'success': False}]]
)
def test_delete_pipelines_timestamps_all_failed_deletes_nothing(mongo_db, pipelines):
@@ -704,3 +769,48 @@ def test_load_latest_data_error(mongo_db):
level=NotificationLevel.ERROR,
attachment_content=ANY,
)
+
+
+def test_find_documents_in_mongodb_leaves_documents_without_timestamp_field(mongo_db):
+ mongo_db.mongo_db_repository.find = MagicMock(return_value=[{'name': 'no-timestamp'}])
+
+ result = mongo_db.find_documents_in_mongodb(
+ {'query': {'collection': 'test_collection'}, 'timestamp_fields': ['timestamp']}
+ )
+
+ assert result == [{'name': 'no-timestamp'}]
+
+
+def test_aggregate_documents_in_mongodb_leaves_documents_without_timestamp_field(mongo_db):
+ mongo_db.mongo_db_repository.aggregate = MagicMock(return_value=[{'name': 'no-timestamp'}])
+
+ result = mongo_db.aggregate_documents_in_mongodb(
+ {
+ 'query': {'collection': 'test_collection', 'aggregation': [{'$match': {}}]},
+ 'timestamp_fields': ['timestamp'],
+ }
+ )
+
+ assert result == [{'name': 'no-timestamp'}]
+
+
+@patch('orchestrator.activities.mongo_db.now')
+def test_create_pipelines_timestamps_all_failed_inserts_nothing(_now_mock, mongo_db):
+ mongo_db.mongo_db_repository.insert_many = MagicMock()
+
+ mongo_db.create_pipelines_timestamps(
+ {'created_pipelines': [{'schedule_name': 'a', 'namespace': 'n', 'success': False}]}
+ )
+
+ mongo_db.mongo_db_repository.insert_many.assert_not_called()
+
+
+def test_create_collection_with_ttl_index_skips_pipelines_without_topic(mongo_db):
+ database = mongo_db.mongo_db_repository.database
+ database.list_collection_names = MagicMock(return_value=[])
+
+ mongo_db.create_collection_with_ttl_index(
+ {**metadata, 'pipelines': {'pi-scouter': {'model_tags': {}}, 'held': {'held': True}}}
+ )
+
+ database.create_collection.assert_not_called()
diff --git a/tests/orchestrator/activities/test_temporal_manager.py b/tests/orchestrator/activities/test_temporal_manager.py
index f3c44ed..7e53e3d 100644
--- a/tests/orchestrator/activities/test_temporal_manager.py
+++ b/tests/orchestrator/activities/test_temporal_manager.py
@@ -145,6 +145,7 @@ async def test_create_schedule(
'data': {'test': 'test'},
'execution_timeout_seconds': 100,
'task_timeout_seconds': 100,
+ 'config_hash': 'h1',
},
'test-schedule-invalid-frequency': {
'model_id': 2,
@@ -291,24 +292,29 @@ async def test_create_schedule(
]
)
+ # config_hash goes to the report (orchestrated_schedules), not to the workflow
+ assert 'config_hash' not in input_data['schedules']['scouter']['test-schedule']
assert report == [
{
'schedule_name': 'test-schedule',
'namespace': 'scouter',
'success': True,
'message': 'Schedule created successfully',
+ 'config_hash': 'h1',
},
{
'schedule_name': 'test-schedule-invalid-frequency',
'namespace': 'scouter',
'success': False,
'message': 'Invalid frequency',
+ 'config_hash': None,
},
{
'schedule_name': 'test-schedule-laborious',
'namespace': 'laborious',
'success': True,
'message': 'Schedule created successfully',
+ 'config_hash': None,
},
]
@@ -355,6 +361,7 @@ async def test_update_schedules(
'workflow_type': 'scouter',
'frequency': '1m',
'data': {'test': 'test'},
+ 'config_hash': 'h1',
},
'test-schedule_no_handler': {
'workflow_type': 'scouter',
@@ -400,24 +407,28 @@ async def test_update_schedules(
handler_scouter.update.assert_called_once()
handler_laborious.update.assert_called_once()
+ assert 'config_hash' not in input_data['schedules']['scouter']['test-schedule']
assert report == [
{
'schedule_name': 'test-schedule',
'namespace': 'scouter',
'success': True,
'message': 'Schedule updated successfully',
+ 'config_hash': 'h1',
},
{
'schedule_name': 'test-schedule_no_handler',
'namespace': 'scouter',
'success': False,
'message': 'Schedule test-schedule_no_handler not found',
+ 'config_hash': None,
},
{
'schedule_name': 'test-schedule-laborious',
'namespace': 'laborious',
'success': True,
'message': 'Schedule updated successfully',
+ 'config_hash': None,
},
]
@@ -723,3 +734,23 @@ async def test_create_schedules_drift_keeps_default_skip_overlap(
mock_schedule.assert_called_once()
policy = mock_schedule.call_args.kwargs.get('policy')
assert policy is None or policy.overlap == ScheduleOverlapPolicy.SKIP
+
+
+@mark.asyncio
+async def test_normalize_schedules_keeps_schedules_known_in_mongo(temporal_manager):
+ """An orchestrated schedule still tracked in orchestrated_schedules is not deleted"""
+ input_data = {
+ 'orchestrated_schedules': {
+ 'scouter': {'test-schedule-id': '2021-01-01'},
+ 'laborious': {'test-schedule-id': '2021-01-01'},
+ },
+ **metadata,
+ }
+ for client in temporal_manager.temporal_clients.values():
+ client.list_schedules = AsyncMock(return_value=async_iter())
+ client.get_schedule_handle = MagicMock()
+
+ await temporal_manager.normalize_schedules(input_data)
+
+ for client in temporal_manager.temporal_clients.values():
+ client.get_schedule_handle.assert_not_called()
diff --git a/tests/orchestrator/utils/tag_database_fixtures.py b/tests/orchestrator/utils/tag_database_fixtures.py
new file mode 100644
index 0000000..80b6303
--- /dev/null
+++ b/tests/orchestrator/utils/tag_database_fixtures.py
@@ -0,0 +1,82 @@
+"""Pipeline and tag database documents shared by the utils tests."""
+
+
+def scouter_pipeline_doc(schedule_name, model_id, read_tags, workflow_type='scouter'):
+ return {
+ 'workflow_type': workflow_type,
+ 'schedule_name': schedule_name,
+ 'model_id': model_id,
+ 'model': {'name': f'model {model_id}'},
+ 'read_tags': read_tags,
+ }
+
+
+TAGS_BY_ID = {
+ 7: {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1001', 'tag_name': 'Pressure'},
+ 8: {'tag_id': 8, 'connector_id': 5, 'address': 'ns=3;i=1002', 'tag_name': 'Level'},
+}
+
+
+# Connector security and client certificates (SIENTIAPDE-1974)
+
+NO_SECURITY = {'mode': 'None', 'policy': 'None'}
+SECURE = {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}
+
+
+# A client certificate as connectors_query's $lookup brings it (certificates[])
+def certificate_doc(certificate_id=1, name='sientia-platform', **extra):
+ return {
+ 'certificate_id': certificate_id,
+ 'application_uri': 'urn:sientia:opc-ingestor',
+ 'cert': {'bucket': 'opc-certificates', 'object_name': f'client/{name}/cert.pem'},
+ 'private_key': {'bucket': 'opc-certificates', 'object_name': f'client/{name}/key.pem'},
+ **extra,
+ }
+
+
+# The slot's certificate for certificate_doc(): the same fields
+CERTIFICATE = certificate_doc()
+
+
+def secure_connector_fields(certificate_id=1, certificates=None):
+ """security and certificates of a connector with a client certificate."""
+ return {
+ 'security': {**SECURE, 'client_certificate_id': certificate_id},
+ 'certificates': [certificate_doc(certificate_id)] if certificates is None else certificates,
+ }
+
+
+# Write tags (predictions_batch write_tags through the tag database)
+
+OPC_CONNECTORS = {
+ 'plant': {
+ 'connector_id': 'plant',
+ 'name': 'plant',
+ 'url': 'opc.tcp://plant:4840',
+ 'security': NO_SECURITY,
+ },
+}
+
+
+def write_tag_doc(tag_id, address, category='Write', connector_id='plant', **extra):
+ return {
+ 'tag_id': tag_id,
+ 'connector_id': connector_id,
+ 'address': address,
+ 'tag_name': f'tag-{tag_id}',
+ 'category': category,
+ 'status': 'Active',
+ **extra,
+ }
+
+
+def batch_pipeline_doc(schedule_name, write_tags, **fields):
+ return {
+ 'workflow_type': 'predictions_batch',
+ 'schedule_name': schedule_name,
+ 'model_id': 'model_id',
+ 'model': {'name': 'model_name'},
+ 'query': 'SELECT 1',
+ 'write_tags': write_tags,
+ **fields,
+ }
diff --git a/tests/orchestrator/utils/test_opc_slots.py b/tests/orchestrator/utils/test_opc_slots.py
new file mode 100644
index 0000000..f32afbb
--- /dev/null
+++ b/tests/orchestrator/utils/test_opc_slots.py
@@ -0,0 +1,450 @@
+import pytest
+
+from orchestrator.utils.opc_slots import (
+ build_tag_config,
+ gather_read_tags,
+)
+from tests.orchestrator.utils.tag_database_fixtures import (
+ CERTIFICATE,
+ NO_SECURITY,
+ SECURE,
+ TAGS_BY_ID,
+ certificate_doc,
+ scouter_pipeline_doc,
+ secure_connector_fields,
+)
+
+NO_ISSUES = {'tags': set(), 'connectors': set(), 'invalid': set(), 'certificates': set()}
+
+
+def _name(schedule_name, tag_name, model_id):
+ return {
+ 'collection': f'raw_{schedule_name}',
+ 'tag_name': tag_name,
+ 'model_id': model_id,
+ 'model_name': f'model {model_id}',
+ }
+
+
+def _read(tag_id, frequency=1000, **extra):
+ return {'tag_id': tag_id, 'frequency': frequency, **extra}
+
+
+def _connector(connector_id, name, **extra):
+ return {
+ 'connector_id': connector_id,
+ 'name': name,
+ 'url': f'opc.tcp://{name}:4840',
+ 'security': NO_SECURITY,
+ **extra,
+ }
+
+
+def _db_tag(tag_id, connector_id, address, tag_name):
+ return {
+ 'tag_id': tag_id,
+ 'connector_id': connector_id,
+ 'address': address,
+ 'tag_name': tag_name,
+ }
+
+
+connectors = {
+ '1': _connector(1, 'kepserver'),
+ '3': _connector(3, 'plant', subscription_period_ms=1500, **secure_connector_fields()),
+}
+
+
+tags_by_id = {
+ **TAGS_BY_ID,
+ 10: _db_tag(10, 1, 'ns=2;s=A', 'A'),
+ 11: _db_tag(11, 1, 'ns=2;s=B', 'B'),
+ 12: _db_tag(12, 1, 'ns=2;s=A', 'A2'),
+ 13: _db_tag(13, 3, 'ns=2;s=A', 'A on plant'),
+ 14: _db_tag(14, 1, 'ns=2;s=C', 'C'),
+}
+
+
+def _gather(pipelines, connector_index=None, tag_index=None):
+ return gather_read_tags(
+ pipelines,
+ connectors if connector_index is None else connector_index,
+ tags_by_id if tag_index is None else tag_index,
+ )
+
+
+def test_gather_read_tags_multi_name():
+ """One address read by two pipelines with different names: one entry, two names"""
+ pipelines = [
+ scouter_pipeline_doc('sin', 'm1', [_read(10, 15000, tag_name='Counter')]),
+ scouter_pipeline_doc('cos', 'm2', [_read(10, 5000, tag_name='Contador')]),
+ ]
+
+ entries, issues = _gather(pipelines)
+
+ assert len(entries) == 1
+ assert entries[0]['address'] == 'ns=2;s=A'
+ assert entries[0]['frequency'] == 5000
+ assert entries[0]['names'] == [_name('cos', 'Contador', 'm2'), _name('sin', 'Counter', 'm1')]
+ assert issues == NO_ISSUES
+
+
+def test_gather_read_tags_same_name_in_two_pipelines():
+ """The same name read by two pipelines still goes to both collections"""
+ pipelines = [
+ scouter_pipeline_doc('sin', 'm1', [_read(10)]),
+ scouter_pipeline_doc('cos', 'm2', [_read(10)]),
+ ]
+
+ entries, _issues = _gather(pipelines)
+
+ assert entries[0]['names'] == [_name('cos', 'A', 'm2'), _name('sin', 'A', 'm1')]
+
+
+def test_gather_read_tags_duplicate_tag_in_one_pipeline():
+ entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(10), _read(10)])])
+
+ assert entries[0]['names'] == [_name('sin', 'A', 'm1')]
+
+
+def test_gather_read_tags_two_tag_ids_on_one_address():
+ """Two connector_tags on the same node: one entry, one name per tag"""
+ entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(10), _read(12)])])
+
+ assert len(entries) == 1
+ assert entries[0]['names'] == [_name('sin', 'A', 'm1'), _name('sin', 'A2', 'm1')]
+
+
+def test_gather_read_tags_same_address_on_two_servers():
+ entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(13), _read(10)])])
+
+ assert [(e['server']['server_id'], e['address']) for e in entries] == [
+ ('1', 'ns=2;s=A'),
+ ('3', 'ns=2;s=A'),
+ ]
+
+
+def test_gather_read_tags_with_tag_database():
+ """Address, name and server come from connector_tags + connectors"""
+ entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(7)])])
+
+ assert entries == [
+ {
+ 'server': {
+ 'server_id': '3',
+ 'name': 'plant',
+ 'url': 'opc.tcp://plant:4840',
+ 'subscription_period_ms': 1500,
+ 'security': SECURE,
+ 'certificate': CERTIFICATE,
+ 'key': 'plant',
+ },
+ 'address': 'ns=3;i=1001',
+ 'frequency': 1000,
+ 'active_read': {'enabled': False},
+ 'names': [_name('sin', 'Pressure', 'm1')],
+ }
+ ]
+
+
+def test_gather_read_tags_pipeline_name_wins_over_tag_database():
+ entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(7, tag_name='P1')])])
+
+ assert entries[0]['names'] == [_name('sin', 'P1', 'm1')]
+
+
+def test_gather_read_tags_unknown_tag_id():
+ entries, issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(404)])])
+
+ assert entries == []
+ assert issues == {**NO_ISSUES, 'tags': {'404'}}
+
+
+def test_gather_read_tags_inactive_connector():
+ """Connector 5 is not among the active connectors: its tag is left out"""
+ entries, issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(8)])])
+
+ assert entries == []
+ assert issues == {**NO_ISSUES, 'connectors': {'5'}}
+
+
+def test_gather_read_tags_without_tag_id():
+ """connectors is the only source of the servers: a read tag needs a tag_id"""
+ pipelines = [
+ scouter_pipeline_doc(
+ 'sin',
+ 'm1',
+ [
+ _read(10),
+ {'tag_name': 'b', 'frequency': 1000},
+ {'tag_id': '', 'tag_name': 'c', 'frequency': 1000},
+ {'tag_id': None, 'tag_name': 'd', 'frequency': 1000},
+ # tag_id is int in connector_tags: text is no reference
+ {'tag_id': '11', 'frequency': 1000},
+ ],
+ )
+ ]
+
+ entries, issues = _gather(pipelines)
+
+ assert [e['address'] for e in entries] == ['ns=2;s=A']
+ assert issues == {**NO_ISSUES, 'invalid': {'sin/b', 'sin/c', 'sin/d', 'sin/11'}}
+
+
+def test_gather_read_tags_server_name_clash():
+ """Two servers with the same name get distinct slot keys, stable by server_id"""
+ connector_index = {**connectors, '2': _connector(2, 'kepserver')}
+ tag_index = {**tags_by_id, 20: _db_tag(20, 2, 'ns=2;s=A', 'A')}
+ pipelines = [scouter_pipeline_doc('sin', 'm1', [_read(20), _read(11)])]
+
+ entries, _issues = _gather(pipelines, connector_index, tag_index)
+
+ assert [e['server']['key'] for e in entries] == ['kepserver', 'kepserver:2']
+ assert [e['server']['name'] for e in entries] == ['kepserver', 'kepserver']
+
+
+def test_gather_read_tags_active_read():
+ """Active read: on if any pipeline asks, with the lowest cycle_multiple (D3)"""
+ pipelines = [
+ scouter_pipeline_doc('p1', 'm1', [_read(10)]),
+ scouter_pipeline_doc(
+ 'p2', 'm2', [_read(10, active_read={'enabled': True, 'cycle_multiple': 4})]
+ ),
+ scouter_pipeline_doc(
+ 'p3', 'm3', [_read(10, active_read={'enabled': True, 'cycle_multiple': 2})]
+ ),
+ scouter_pipeline_doc(
+ 'p4', 'm4', [_read(11, active_read={'enabled': False, 'cycle_multiple': 1})]
+ ),
+ scouter_pipeline_doc('p5', 'm5', [_read(14, active_read={'enabled': True})]),
+ ]
+
+ entries, _issues = _gather(pipelines)
+
+ assert [e['active_read'] for e in entries] == [
+ {'enabled': True, 'cycle_multiple': 2},
+ {'enabled': False},
+ {'enabled': True, 'cycle_multiple': 1},
+ ]
+
+
+def test_gather_read_tags_filters_non_scouter_workflows():
+ """Only scouter pipelines feed the slots"""
+ pipelines = [
+ scouter_pipeline_doc('sin', 'm1', [_read(10)]),
+ scouter_pipeline_doc('pred', 'm2', [_read(11)], workflow_type='predictions_batch'),
+ scouter_pipeline_doc('retrain', 'm3', [_read(14)], workflow_type='minimal_retrain'),
+ scouter_pipeline_doc('drift', 'm4', [_read(7)], workflow_type='drift'),
+ ]
+
+ entries, _issues = _gather(pipelines)
+
+ assert [e['address'] for e in entries] == ['ns=2;s=A']
+
+
+def test_gather_read_tags_without_read_tags():
+ pipeline = scouter_pipeline_doc('sin', 'm1', [])
+ del pipeline['read_tags']
+
+ assert _gather([pipeline]) == ([], NO_ISSUES)
+
+
+def test_build_tag_config():
+ """Slot v2 (contract C1): security, certificate and subscription period per server"""
+ pipelines = [
+ scouter_pipeline_doc('sin', 'm1', [_read(10, 1000), _read(11, '300'), _read(7, 100)])
+ ]
+ entries, _issues = _gather(pipelines)
+
+ result = build_tag_config(entries)
+
+ assert result == {
+ 'kepserver': {
+ 'server_id': '1',
+ 'name': 'kepserver',
+ 'url': 'opc.tcp://kepserver:4840',
+ # No client certificate: no security (D5)
+ 'security': NO_SECURITY,
+ 'certificate': None,
+ # No subscription_period_ms on the connector: half its lowest tag frequency
+ 'subscription_period_ms': 150,
+ 'tags': {
+ 'ns=2;s=A': {
+ 'frequency': 1000,
+ 'active_read': {'enabled': False},
+ 'names': [_name('sin', 'A', 'm1')],
+ },
+ 'ns=2;s=B': {
+ 'frequency': 300,
+ 'active_read': {'enabled': False},
+ 'names': [_name('sin', 'B', 'm1')],
+ },
+ },
+ },
+ 'plant': {
+ 'server_id': '3',
+ 'name': 'plant',
+ 'url': 'opc.tcp://plant:4840',
+ 'subscription_period_ms': 1500,
+ 'security': SECURE,
+ 'certificate': CERTIFICATE,
+ 'tags': {
+ 'ns=3;i=1001': {
+ 'frequency': 100,
+ 'active_read': {'enabled': False},
+ 'names': [_name('sin', 'Pressure', 'm1')],
+ },
+ },
+ },
+ }
+
+
+def test_build_tag_config_empty():
+ assert build_tag_config([]) == {}
+
+
+def test_gather_read_tags_invalid_read_tags():
+ """Unusable read tags are reported and skipped; the others still make the slot"""
+ pipelines = [
+ scouter_pipeline_doc('sin', 'm1', [_read(10), {'tag_id': 11}, _read(7, frequency='fast')])
+ ]
+
+ entries, issues = _gather(pipelines)
+
+ assert [e['address'] for e in entries] == ['ns=2;s=A']
+ assert issues['invalid'] == {'sin/11', 'sin/7'}
+
+
+def test_gather_read_tags_frequency_type_does_not_depend_on_order():
+ """15000 and '15000' give the same int frequency in any pipeline order (O6)"""
+ as_int = scouter_pipeline_doc('sin', 'm1', [_read(10, 15000)])
+ as_text = scouter_pipeline_doc('cos', 'm2', [_read(10, '15000')])
+
+ first, _ = _gather([as_int, as_text])
+ second, _ = _gather([as_text, as_int])
+
+ assert first[0]['frequency'] == second[0]['frequency'] == 15000
+ assert isinstance(first[0]['frequency'], int)
+ assert isinstance(second[0]['frequency'], int)
+
+
+def test_gather_read_tags_server_key_never_collides():
+ """A server literally named 'x:3' does not share its key with server 3"""
+ connector_index = {
+ '1': _connector(1, 'x:3'),
+ '2': _connector(2, 'x'),
+ '3': _connector(3, 'x'),
+ }
+ tag_index = {
+ 1: _db_tag(1, 1, 'ns=2;s=A', 'a'),
+ 2: _db_tag(2, 2, 'ns=2;s=B', 'b'),
+ 3: _db_tag(3, 3, 'ns=2;s=C', 'c'),
+ }
+ pipelines = [scouter_pipeline_doc('sin', 'm1', [_read(1), _read(2), _read(3)])]
+
+ entries, _issues = _gather(pipelines, connector_index, tag_index)
+ slot = build_tag_config(entries)
+
+ keys = [e['server']['key'] for e in entries]
+ assert len(set(keys)) == 3
+ assert {server['server_id'] for server in slot.values()} == {'1', '2', '3'}
+ assert all(len(server['tags']) == 1 for server in slot.values())
+
+
+def test_gather_read_tags_invalid_cycle_multiple_does_not_fail_the_slot():
+ """A bad active_read is reported as invalid instead of failing every pipeline's slot"""
+ pipelines = [
+ scouter_pipeline_doc(
+ 'sin',
+ 'm1',
+ [
+ _read(10),
+ _read(11, active_read={'enabled': True, 'cycle_multiple': 'x'}),
+ _read(7, active_read={'enabled': True, 'cycle_multiple': None}),
+ # Not a dict: treated as no active read
+ _read(14, active_read=True),
+ ],
+ )
+ ]
+
+ entries, issues = _gather(pipelines)
+
+ assert [e['address'] for e in entries] == ['ns=2;s=A', 'ns=2;s=C']
+ assert entries[1]['active_read'] == {'enabled': False}
+ assert issues['invalid'] == {'sin/11', 'sin/7'}
+
+
+def test_gather_read_tags_skips_incomplete_pipelines():
+ """A pipeline without schedule_name, model_id or model.name is reported, not raised"""
+ no_model = scouter_pipeline_doc('no-model', 'm1', [_read(10)])
+ del no_model['model']
+ no_name = scouter_pipeline_doc('no-name', 'm2', [_read(10)])
+ no_name['model'] = {}
+ no_model_id = scouter_pipeline_doc('no-id', None, [_read(10)])
+ no_schedule = scouter_pipeline_doc(None, 'm4', [_read(10)])
+ ok = scouter_pipeline_doc('ok', 'm5', [_read(11)])
+
+ entries, issues = _gather([no_model, no_name, no_model_id, no_schedule, ok])
+
+ assert [e['address'] for e in entries] == ['ns=2;s=B']
+ assert issues['invalid'] == {'no-model', 'no-name', 'no-id', 'None'}
+
+
+def _plant_with(**fields):
+ return {'1': connectors['1'], '3': {**connectors['3'], **fields}}
+
+
+@pytest.mark.parametrize(
+ 'fields',
+ [
+ # $lookup found nothing: missing id, inactive or not a client certificate
+ pytest.param(secure_connector_fields(certificates=[]), id='not-found'),
+ pytest.param(
+ secure_connector_fields(certificates=[certificate_doc(application_uri='')]),
+ id='no-application-uri',
+ ),
+ pytest.param(
+ secure_connector_fields(
+ certificates=[{**certificate_doc(), 'private_key': {'bucket': 'b'}}]
+ ),
+ id='key-without-object',
+ ),
+ ],
+)
+def test_gather_read_tags_registered_certificate_missing(fields):
+ """An id without a usable certificate: in the slot without a certificate, with a warning (D5)"""
+ entries, issues = _gather(
+ [scouter_pipeline_doc('sin', 'm1', [_read(7)])], _plant_with(**fields)
+ )
+
+ assert entries[0]['server']['certificate'] is None
+ assert entries[0]['server']['security'] == SECURE
+ assert issues == {**NO_ISSUES, 'certificates': {'3'}}
+
+
+@pytest.mark.parametrize('certificate_id', [None, 'absent'])
+def test_gather_read_tags_without_certificate_id(certificate_id):
+ """No client_certificate_id: no certificate and nothing to report (D5)"""
+ security = {**NO_SECURITY, 'client_certificate_id': certificate_id}
+ if certificate_id == 'absent':
+ security = NO_SECURITY
+ entries, issues = _gather(
+ [scouter_pipeline_doc('sin', 'm1', [_read(7)])],
+ _plant_with(security=security, certificates=[]),
+ )
+
+ assert entries[0]['server']['certificate'] is None
+ assert issues == NO_ISSUES
+
+
+def test_gather_read_tags_reports_a_certificate_once_per_connector():
+ entries, issues = _gather(
+ [
+ scouter_pipeline_doc('sin', 'm1', [_read(7)]),
+ scouter_pipeline_doc('cos', 'm2', [_read(7)]),
+ ],
+ _plant_with(**secure_connector_fields(certificates=[])),
+ )
+
+ assert len(entries) == 1
+ assert issues['certificates'] == {'3'}
diff --git a/tests/orchestrator/utils/test_orchestrator_functions.py b/tests/orchestrator/utils/test_pipeline_configs.py
similarity index 73%
rename from tests/orchestrator/utils/test_orchestrator_functions.py
rename to tests/orchestrator/utils/test_pipeline_configs.py
index bda5c74..bd08283 100644
--- a/tests/orchestrator/utils/test_orchestrator_functions.py
+++ b/tests/orchestrator/utils/test_pipeline_configs.py
@@ -2,14 +2,12 @@ from unittest.mock import call, patch
from pytest import mark
-from orchestrator.utils.orchestrator_functions import (
+from orchestrator.utils.pipeline_configs import (
base_scouter,
- build_tag_config,
chunk_period_warning,
common_config,
drift,
frequency_alert_warning,
- gather_read_tags,
minimal_retrain,
overlap_filter_config,
pi_web_api_scouter,
@@ -345,11 +343,11 @@ def test_process_path_priority():
@patch(
- 'orchestrator.utils.orchestrator_functions.overlap_filter_config',
+ 'orchestrator.utils.pipeline_configs.overlap_filter_config',
return_value={'test_filter_name': {'policy': 'test_policy', 'config': {}}},
)
@patch(
- 'orchestrator.utils.orchestrator_functions.process_path_priority',
+ 'orchestrator.utils.pipeline_configs.process_path_priority',
return_value=['STOP', 'CONTINUE', 'REPEAT'],
)
def test_predictions_batch(mock_process_path_priority, mock_overlap_filter_config):
@@ -359,10 +357,14 @@ def test_predictions_batch(mock_process_path_priority, mock_overlap_filter_confi
'model_id': 'test_model_id',
'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
'query': 'test_query',
- 'write_tags': [
- {'server_id': 'test_server_id', 'type': 'prediction', 'addr': 'test_addr'},
- {'server_id': 'test_server_id', 'type': 'confidence', 'addr': 'test_addr'},
- ],
+ 'write_tags': [{'tag_id': 77}, {'tag_id': 78}],
+ # Resolved from write_tags by process_schedules
+ 'opc_output_config': {
+ 'test_server_id': {
+ 'prediction_tags': {'test_addr': {'data_type': 'float'}},
+ 'confidence_tags': {'test_addr': {'data_type': 'float'}},
+ }
+ },
'pi_web_api_output_config': {
'endpoint': 'test_endpoint',
'prediction_tags': {
@@ -445,249 +447,17 @@ def test_predictions_batch(mock_process_path_priority, mock_overlap_filter_confi
assert result == expected
-def test_gather_read_tags():
- pipelines = [
- {
- 'workflow_type': 'scouter',
- 'schedule_name': 'test_schedule',
- 'read_tags': [
- {
- 'server_id': '1',
- 'server_name': 'test_server_name',
- 'tag_address': 'test_tag_address',
- }
- ],
- },
- {
- 'workflow_type': 'scouter',
- 'schedule_name': 'test_schedule2',
- 'read_tags': [
- {
- 'server_id': '2',
- 'server_name': 'test_server_name2',
- 'tag_address': 'test_tag_address2',
- },
- {
- 'server_id': '2',
- 'server_name': 'test_server_name2',
- 'tag_address': 'test_tag_address3',
- },
- ],
- },
- ]
-
- result = gather_read_tags(pipelines)
-
- expected = {
- '1:test_tag_address': {
- 'server_id': '1',
- 'server_name': 'test_server_name',
- 'tag_address': 'test_tag_address',
- 'topics': ['raw_test_schedule'],
- },
- '2:test_tag_address2': {
- 'server_id': '2',
- 'server_name': 'test_server_name2',
- 'tag_address': 'test_tag_address2',
- 'topics': ['raw_test_schedule2'],
- },
- '2:test_tag_address3': {
- 'server_id': '2',
- 'server_name': 'test_server_name2',
- 'tag_address': 'test_tag_address3',
- 'topics': ['raw_test_schedule2'],
- },
+def test_scouter_skips_tags_without_name():
+ """A tag_id the tag database cannot resolve has no name and stays out of model_tags"""
+ config = {
+ 'workflow_type': 'scouter',
+ 'schedule_name': 'sin',
+ 'model_id': 'm1',
+ 'model': {'name': 'model m1'},
+ 'read_tags': [{'tag_name': 'a'}, {'tag_id': 404}],
}
- assert result == expected
-
-
-def test_gather_read_tags_filters_non_scouter_workflows():
- """Test that gather_read_tags only processes scouter workflow types and ignores others"""
- pipelines = [
- {
- 'workflow_type': 'scouter',
- 'schedule_name': 'test_scouter_schedule',
- 'read_tags': [
- {
- 'server_id': '1',
- 'server_name': 'test_server_name',
- 'tag_address': 'test_tag_address',
- }
- ],
- },
- {
- 'workflow_type': 'predictions_batch',
- 'schedule_name': 'test_predictions_schedule',
- 'read_tags': [
- {
- 'server_id': '2',
- 'server_name': 'test_server_name2',
- 'tag_address': 'test_tag_address_predictions',
- }
- ],
- },
- {
- 'workflow_type': 'minimal_retrain',
- 'schedule_name': 'test_retrain_schedule',
- 'read_tags': [
- {
- 'server_id': '3',
- 'server_name': 'test_server_name3',
- 'tag_address': 'test_tag_address_retrain',
- }
- ],
- },
- {
- 'workflow_type': 'pi_web_api_scouter',
- 'schedule_name': 'test_pi_web_api_schedule',
- 'read_tags': [
- {
- 'server_id': '4',
- 'server_name': 'test_server_name4',
- 'tag_address': 'test_tag_address_pi_web_api',
- }
- ],
- },
- {
- 'workflow_type': 'drift',
- 'schedule_name': 'test_drift_schedule',
- 'read_tags': [
- {
- 'server_id': '5',
- 'server_name': 'test_server_name5',
- 'tag_address': 'test_tag_address_drift',
- }
- ],
- },
- {
- 'workflow_type': 'scouter',
- 'schedule_name': 'test_scouter_schedule2',
- 'read_tags': [
- {
- 'server_id': '1',
- 'server_name': 'test_server_name',
- 'tag_address': 'test_tag_address2',
- }
- ],
- },
- ]
-
- result = gather_read_tags(pipelines)
-
- # Only scouter workflow types should be included
- expected = {
- '1:test_tag_address': {
- 'server_id': '1',
- 'server_name': 'test_server_name',
- 'tag_address': 'test_tag_address',
- 'topics': ['raw_test_scouter_schedule'],
- },
- '1:test_tag_address2': {
- 'server_id': '1',
- 'server_name': 'test_server_name',
- 'tag_address': 'test_tag_address2',
- 'topics': ['raw_test_scouter_schedule2'],
- },
- }
-
- assert result == expected
- # Ensure non-scouter pipelines are not included
- assert '2:test_tag_address_predictions' not in result
- assert '3:test_tag_address_retrain' not in result
- assert '4:test_tag_address_pi_web_api' not in result
- assert '5:test_tag_address_drift' not in result
-
-
-def test_build_tag_config():
- tags = [
- {
- 'server_id': '1',
- 'server_name': 'test_server_name',
- 'tag_address': 'test_tag_address',
- 'frequency': 1000,
- },
- {
- 'server_id': '2',
- 'server_name': 'test_server_name2',
- 'tag_address': 'test_tag_address2',
- 'frequency': 1000,
- },
- {
- 'server_id': '2',
- 'server_name': 'test_server_name2',
- 'tag_address': 'test_tag_address3',
- 'frequency': 300,
- },
- {
- 'server_id': '3',
- 'server_name': 'test_server_name3',
- 'tag_address': 'test_tag_address4',
- 'frequency': 200,
- },
- ]
-
- opc_servers = {
- '1': {
- 'server_name': 'test_server_name',
- 'url': 'test_url',
- 'uri': 'test_uri',
- },
- '2': {
- 'server_name': 'test_server_name2',
- 'url': 'test_url2',
- 'uri': 'test_uri2',
- },
- }
-
- result = build_tag_config(tags, opc_servers)
-
- expected = {
- 'test_server_name': {
- 'server_id': '1',
- 'name': 'test_server_name',
- 'url': 'test_url',
- 'server_uri': 'test_uri',
- 'cert_path': None,
- 'private_key_path': None,
- 'server_cert_path': None,
- 'subscription_period_ms': 500,
- 'tags': {
- 'test_tag_address': {
- 'server_id': '1',
- 'server_name': 'test_server_name',
- 'tag_address': 'test_tag_address',
- 'frequency': 1000,
- },
- },
- },
- 'test_server_name2': {
- 'server_id': '2',
- 'name': 'test_server_name2',
- 'url': 'test_url2',
- 'server_uri': 'test_uri2',
- 'cert_path': None,
- 'private_key_path': None,
- 'server_cert_path': None,
- 'subscription_period_ms': 150,
- 'tags': {
- 'test_tag_address2': {
- 'server_id': '2',
- 'server_name': 'test_server_name2',
- 'tag_address': 'test_tag_address2',
- 'frequency': 1000,
- },
- 'test_tag_address3': {
- 'server_id': '2',
- 'server_name': 'test_server_name2',
- 'tag_address': 'test_tag_address3',
- 'frequency': 300,
- },
- },
- },
- }
-
- assert result == (expected, ['3'])
+ assert list(scouter(config)['model_tags']) == ['a']
def test_base_scouter():
@@ -912,3 +682,61 @@ def test_pi_web_api_scouter_with_no_timeout():
)
def test_frequency_alert_warning(frequency_alert, warns):
assert (frequency_alert_warning(frequency_alert) is not None) is warns
+
+
+def test_pi_web_api_scouter_caps_api_timeout_at_frequency_only_when_larger():
+ config = {
+ 'workflow_type': 'pi_web_api_scouter',
+ 'schedule_name': 'pi_schedule',
+ 'model_id': 'model_id',
+ 'model': {'name': 'model_name'},
+ 'frequency': '1m',
+ 'read_tags': {'TI-101': {'webid': 'F1DP-X'}},
+ 'pi_web_api_config': {
+ 'endpoint': '/streamsets/recorded',
+ 'max_count': 5,
+ 'api_timeout': 10,
+ },
+ }
+
+ result = pi_web_api_scouter(config)
+
+ assert result['model_tags'] == {
+ 'TI-101': {'webid': 'F1DP-X', 'aggr_func': 'lts', 'data_range': [-100, 100]}
+ }
+ assert result['pi_web_api_query'] == {
+ 'endpoint': '/streamsets/recorded',
+ 'period': '*-1d',
+ 'max_count': 5,
+ 'api_timeout': 10,
+ }
+
+
+def test_predictions_batch_copies_the_resolved_opc_output_config():
+ opc_output_config = {'plant': {'prediction_tags': {'ns=2;s=P1': {'data_type': 'int'}}}}
+ config = {
+ 'workflow_type': 'predictions_batch',
+ 'schedule_name': 'batch',
+ 'model_id': 'model_id',
+ 'model': {'name': 'model_name'},
+ 'query': 'SELECT 1',
+ 'write_tags': [{'tag_id': 77}],
+ 'opc_output_config': opc_output_config,
+ }
+
+ result = predictions_batch(config)
+
+ assert result['opc_output_config'] == opc_output_config
+ assert 'write_tags' not in result
+
+
+def test_predictions_batch_without_write_tags_has_no_opc_output():
+ config = {
+ 'workflow_type': 'predictions_batch',
+ 'schedule_name': 'batch',
+ 'model_id': 'model_id',
+ 'model': {'name': 'model_name'},
+ 'query': 'SELECT 1',
+ }
+
+ assert predictions_batch(config)['opc_output_config'] == {}
diff --git a/tests/orchestrator/utils/test_tag_database.py b/tests/orchestrator/utils/test_tag_database.py
new file mode 100644
index 0000000..05ce73e
--- /dev/null
+++ b/tests/orchestrator/utils/test_tag_database.py
@@ -0,0 +1,279 @@
+import pytest
+
+from orchestrator.utils.tag_database import (
+ WRITE_CATEGORIES,
+ client_certificate_id,
+ filter_by_tag_ids,
+ index_connector_tags,
+ index_connectors,
+ index_tag_documents,
+ referenced_tag_ids,
+ resolve_certificate,
+ resolve_tag_names,
+ resolve_write_tags,
+ tag_id_of,
+)
+from tests.orchestrator.utils.tag_database_fixtures import (
+ CERTIFICATE,
+ NO_SECURITY,
+ OPC_CONNECTORS,
+ TAGS_BY_ID,
+ batch_pipeline_doc,
+ certificate_doc,
+ scouter_pipeline_doc,
+ secure_connector_fields,
+ write_tag_doc,
+)
+
+
+def test_index_connector_tags():
+ tags = [
+ {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1', 'tag_name': 'a'},
+ # tag_id is int in connector_tags: a text id is no tag
+ {'tag_id': '8', 'connector_id': 3, 'address': 'ns=3;i=2', 'tag_name': 'b'},
+ ]
+
+ assert index_connector_tags(tags) == {7: tags[0]}
+
+
+def test_resolve_tag_names():
+ read_tags = [
+ {'tag_id': 7},
+ {'tag_id': 7, 'tag_name': 'own name'},
+ {'tag_id': '7'},
+ {'tag_id': 404},
+ {'tag_name': 'no tag_id'},
+ ]
+
+ assert resolve_tag_names(read_tags, TAGS_BY_ID) == [
+ {'tag_id': 7, 'tag_name': 'Pressure'},
+ {'tag_id': 7, 'tag_name': 'own name'},
+ {'tag_id': '7'},
+ {'tag_id': 404},
+ {'tag_name': 'no tag_id'},
+ ]
+
+
+def test_referenced_tag_ids():
+ """Int ids only, scouter pipelines only; text, empty or null tag_id ignored"""
+ pipelines = [
+ scouter_pipeline_doc(
+ 'sin', 'm1', [{'tag_id': 7}, {'tag_id': '12'}, {'tag_id': ''}, {'tag_id': None}]
+ ),
+ scouter_pipeline_doc(
+ 'cos', 'm2', [{'tag_id': 'abc'}, {'tag_name': 'a', 'frequency': 1000}]
+ ),
+ scouter_pipeline_doc('pred', 'm3', [{'tag_id': 99}], workflow_type='predictions_batch'),
+ ]
+
+ # Sorted, so the query is the same on every run
+ assert referenced_tag_ids(pipelines) == [7]
+
+
+def test_referenced_tag_ids_without_tag_ids():
+ pipeline = scouter_pipeline_doc('sin', 'm1', [{'tag_name': 'a', 'frequency': 1000}])
+ no_read_tags = scouter_pipeline_doc('cos', 'm2', [])
+ del no_read_tags['read_tags']
+
+ assert referenced_tag_ids([pipeline, no_read_tags]) == []
+
+
+def test_filter_by_tag_ids():
+ """A tag_id $match goes in front, on a new aggregation list"""
+ query = {'collection': 'connector_tags', 'aggregation': [{'$project': {'_id': 0}}]}
+
+ result = filter_by_tag_ids(query, [7, 8])
+
+ assert result == {
+ 'collection': 'connector_tags',
+ 'aggregation': [{'$match': {'tag_id': {'$in': [7, 8]}}}, {'$project': {'_id': 0}}],
+ }
+ assert result['aggregation'] is not query['aggregation']
+ assert query['aggregation'] == [{'$project': {'_id': 0}}]
+
+
+def test_index_skips_incomplete_documents():
+ """A malformed tag database document is left out instead of failing the run"""
+ tags = [
+ {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1', 'tag_name': 'a'},
+ {'connector_id': 3, 'address': 'ns=3;i=2', 'tag_name': 'no id'},
+ {'tag_id': 9, 'connector_id': 3, 'address': '', 'tag_name': 'no address'},
+ ]
+ connector_list = [
+ {'connector_id': 3, 'name': 'plant', 'url': 'opc.tcp://p:4840', 'security': NO_SECURITY},
+ {'connector_id': 4, 'name': 'no url', 'security': NO_SECURITY},
+ {'name': 'no id', 'url': 'opc.tcp://x:4840', 'security': NO_SECURITY},
+ # security.mode and security.policy are the source of the OPC security (D4)
+ {'connector_id': 5, 'name': 'no security', 'url': 'opc.tcp://x:4840'},
+ {
+ 'connector_id': 6,
+ 'name': 'no mode',
+ 'url': 'opc.tcp://x:4840',
+ 'security': {'policy': 'None'},
+ },
+ {
+ 'connector_id': 7,
+ 'name': 'no policy',
+ 'url': 'opc.tcp://x:4840',
+ 'security': {'mode': 'None'},
+ },
+ ]
+
+ assert list(index_connector_tags(tags)) == [7]
+ assert list(index_connectors(connector_list)) == ['3']
+
+
+def test_tag_id_of_accepts_only_int():
+ """connector_tags stores tag_id as int: anything else is no reference"""
+ assert tag_id_of({'tag_id': 7}) == 7
+ assert tag_id_of({'tag_id': '7'}) is None
+ assert tag_id_of({'tag_id': 'abc'}) is None
+ assert tag_id_of({'tag_id': 7.0}) is None
+ assert tag_id_of({'tag_id': True}) is None
+ assert tag_id_of({'tag_id': ''}) is None
+ assert tag_id_of({'tag_id': None}) is None
+ assert tag_id_of({}) is None
+ assert tag_id_of('ns=2;s=raw') is None
+
+
+def test_referenced_tag_ids_includes_write_tags():
+ pipelines = [
+ scouter_pipeline_doc('sin', 'm1', [{'tag_id': 7}]),
+ batch_pipeline_doc('pred', [{'tag_id': 77}, {'tag_id': 78}, {'server_id': '1'}]),
+ batch_pipeline_doc('xgb', [{'tag_id': 79}], workflow_type='xgboost_predictions_batch'),
+ batch_pipeline_doc('retrain', [{'tag_id': 99}], workflow_type='minimal_retrain'),
+ ]
+
+ assert referenced_tag_ids(pipelines) == [7, 77, 78, 79]
+
+
+def test_index_connector_tags_filters_status_and_category():
+ tags = [
+ write_tag_doc(1, 'ns=2;s=R', category='Read'),
+ write_tag_doc(2, 'ns=2;s=W'),
+ write_tag_doc(3, 'ns=2;s=C', category='Prediction Confidence'),
+ write_tag_doc(4, 'ns=2;s=I', status='Inactive'),
+ # An override that projects neither status nor category
+ {'tag_id': 5, 'connector_id': 'plant', 'address': 'ns=2;s=N', 'tag_name': 'n'},
+ ]
+
+ assert list(index_connector_tags(tags)) == [1, 5]
+ assert list(index_connector_tags(tags, WRITE_CATEGORIES)) == [2, 3, 5]
+ # Every document, whatever its status or category
+ assert list(index_tag_documents(tags)) == [1, 2, 3, 4, 5]
+
+
+def test_index_connectors_leaves_out_inactive():
+ connectors = [
+ {**OPC_CONNECTORS['plant'], 'status': 'Active'},
+ {**OPC_CONNECTORS['plant'], 'connector_id': 'old', 'status': 'Inactive'},
+ ]
+
+ assert list(index_connectors(connectors)) == ['plant']
+
+
+def test_resolve_write_tags():
+ tags = index_connector_tags(
+ [
+ write_tag_doc(77, 'ns=2;s=P1'),
+ write_tag_doc(78, 'ns=2;s=C1', category='Prediction Confidence', data_type='double'),
+ write_tag_doc(79, 'ns=2;s=P2', connector_id='other', data_type='int'),
+ ],
+ WRITE_CATEGORIES,
+ )
+ connectors = {**OPC_CONNECTORS, 'other': {**OPC_CONNECTORS['plant'], 'connector_id': 'other'}}
+
+ output_config, issues = resolve_write_tags(
+ [{'tag_id': 77}, {'tag_id': 78}, {'tag_id': 79}], tags, connectors
+ )
+
+ assert output_config == {
+ 'plant': {
+ 'prediction_tags': {'ns=2;s=P1': {'data_type': 'float'}},
+ 'confidence_tags': {'ns=2;s=C1': {'data_type': 'double'}},
+ },
+ 'other': {'prediction_tags': {'ns=2;s=P2': {'data_type': 'int'}}},
+ }
+ assert not any(issues.values())
+
+
+def test_resolve_write_tags_issues():
+ tags = index_connector_tags(
+ [
+ write_tag_doc(1, 'ns=2;s=R', category='Read'),
+ write_tag_doc(2, 'ns=2;s=X', connector_id='gone'),
+ write_tag_doc(3, 'ns=2;s=B', data_type='Double'),
+ write_tag_doc(4, 'ns=2;s=I', status='Inactive'),
+ # No category: a write tag cannot tell prediction from confidence
+ {'tag_id': 5, 'connector_id': 'plant', 'address': 'ns=2;s=N', 'tag_name': 'n'},
+ write_tag_doc(6, 'ns=2;s=OK'),
+ ],
+ WRITE_CATEGORIES,
+ )
+ write_tags = [
+ {'tag_id': 1},
+ {'tag_id': 2},
+ {'tag_id': 3},
+ {'tag_id': 4},
+ {'tag_id': 5},
+ {'tag_id': 404},
+ # Every write tag is a tag_id
+ {'tag_name': 'L'},
+ {'tag_id': None},
+ # tag_id is int: a text id is invalid
+ {'tag_id': '6'},
+ 'ns=2;s=raw',
+ {'tag_id': 6},
+ ]
+
+ output_config, issues = resolve_write_tags(write_tags, tags, OPC_CONNECTORS)
+
+ assert output_config == {'plant': {'prediction_tags': {'ns=2;s=OK': {'data_type': 'float'}}}}
+ assert issues['tags'] == {'1', '4', '5', '404'}
+ assert issues['connectors'] == {'gone'}
+ assert issues['invalid'] == {
+ '3',
+ "{'tag_name': 'L'}",
+ "{'tag_id': '6'}",
+ "{'tag_id': None}",
+ 'ns=2;s=raw',
+ }
+
+
+def test_resolve_certificate():
+ """Only the fields the gateway needs: dates and other fields stay out of the slot"""
+ connector = {
+ **OPC_CONNECTORS['plant'],
+ **secure_connector_fields(
+ certificates=[certificate_doc(thumbprint_sha1='ab', not_after=None)]
+ ),
+ }
+
+ assert resolve_certificate(connector) == CERTIFICATE
+ assert client_certificate_id(connector) == 1
+
+
+@pytest.mark.parametrize(
+ 'certificates',
+ [
+ pytest.param(None, id='no-lookup'),
+ pytest.param([], id='not-found'),
+ pytest.param([certificate_doc(application_uri=None)], id='no-application-uri'),
+ pytest.param([{**certificate_doc(), 'cert': None}], id='no-cert'),
+ pytest.param(
+ [{**certificate_doc(), 'private_key': {'object_name': 'k'}}], id='key-no-bucket'
+ ),
+ pytest.param(['not a document'], id='not-a-document'),
+ ],
+)
+def test_resolve_certificate_without_a_usable_certificate(certificates):
+ connector = {**OPC_CONNECTORS['plant'], **secure_connector_fields()}
+ connector['certificates'] = certificates
+
+ assert resolve_certificate(connector) is None
+
+
+def test_client_certificate_id_without_one():
+ assert client_certificate_id(OPC_CONNECTORS['plant']) is None
+ assert client_certificate_id({'connector_id': 'x'}) is None
+ assert client_certificate_id({'security': None}) is None
diff --git a/tests/orchestrator/worker/test_worker.py b/tests/orchestrator/worker/test_worker.py
index d23cbb7..ced676c 100644
--- a/tests/orchestrator/worker/test_worker.py
+++ b/tests/orchestrator/worker/test_worker.py
@@ -3,7 +3,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
from pytest import mark
-from orchestrator.worker.worker import main
+from orchestrator.worker.worker import main, start_prometheus_server
from orchestrator.workflows.alerts import Alerts
from orchestrator.workflows.orchestrator import Orchestrator
from orchestrator.workflows.reports import Reports
@@ -47,3 +47,43 @@ async def test_main_starts_three_workers_for_expected_workflows(
assert main_workflows == [Orchestrator, Alerts, Reports]
assert worker_mock.run.call_count == 3
+
+
+@mark.asyncio
+@patch('orchestrator.worker.worker.Runtime')
+@patch('orchestrator.worker.worker.sys')
+@patch('orchestrator.worker.worker.start_http_server')
+@patch('orchestrator.worker.worker.NotificationHandler')
+@patch('orchestrator.worker.worker.Activities')
+@patch('orchestrator.worker.worker.prepare_worker')
+@patch('orchestrator.worker.worker.client.Client.connect', new_callable=AsyncMock)
+async def test_main_exits_with_error_when_a_worker_fails(
+ _connect_mock,
+ prepare_worker_mock,
+ activities_mock,
+ notification_handler_mock,
+ _start_http_server,
+ sys_mock,
+ _runtime_mock,
+):
+ """A worker failure shuts everything down and exits with code 1."""
+ activities_instance = activities_mock.return_value
+ activities_instance.connect_to_temporal = AsyncMock()
+
+ worker_mock = MagicMock()
+ worker_mock.run = AsyncMock(side_effect=RuntimeError('worker crashed'))
+ prepare_worker_mock.return_value = worker_mock
+
+ await main()
+
+ notification_handler_mock.return_value.shutdown.assert_called_once()
+ activities_instance.shutdown.assert_called_once()
+ sys_mock.exit.assert_called_once_with(1)
+
+
+@patch('orchestrator.worker.worker.os._exit')
+@patch('orchestrator.worker.worker.start_http_server', side_effect=OSError('port in use'))
+def test_start_prometheus_server_exits_on_failure(_start_http_server, exit_mock):
+ start_prometheus_server()
+
+ exit_mock.assert_called_once_with(1)
diff --git a/tests/orchestrator/workflows/histories/orchestrator_tag_database.json b/tests/orchestrator/workflows/histories/orchestrator_tag_database.json
new file mode 100644
index 0000000..bfb0aac
--- /dev/null
+++ b/tests/orchestrator/workflows/histories/orchestrator_tag_database.json
@@ -0,0 +1,1583 @@
+{
+ "events": [
+ {
+ "eventId": "1",
+ "eventTime": "2026-10-29T13:13:12.477828782Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_STARTED",
+ "taskId": "1048587",
+ "workflowExecutionStartedEventAttributes": {
+ "workflowType": {
+ "name": "orchestrator"
+ },
+ "taskQueue": {
+ "name": "orchestrator-test-queue",
+ "kind": "TASK_QUEUE_KIND_NORMAL"
+ },
+ "input": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJwaXBlbGluZXNfcXVlcnkiOnsiYWdncmVnYXRpb24iOlt7IiRtYXRjaCI6eyJhY3RpdmUiOnRydWV9fV0sImNvbGxlY3Rpb24iOiJwaXBlbGluZXMifSwic2NoZWR1bGVfbmFtZSI6Im9yY2hlc3RyYXRvci1lMmUtdGFnLWRhdGFiYXNlIn0="
+ }
+ ]
+ },
+ "workflowTaskTimeout": "10s",
+ "originalExecutionRunId": "01a187cb-081d-7ca0-a74e-7c386b89c4d7",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "firstExecutionRunId": "01a187cb-081d-7ca0-a74e-7c386b89c4d7",
+ "attempt": 1,
+ "firstWorkflowTaskBackoff": "0s",
+ "workflowId": "orchestrator-tagdb-history-c27d057245d4",
+ "priority": {}
+ }
+ },
+ {
+ "eventId": "2",
+ "eventTime": "2026-10-29T13:13:12.477987913Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
+ "taskId": "1048588",
+ "workflowTaskScheduledEventAttributes": {
+ "taskQueue": {
+ "name": "orchestrator-test-queue",
+ "kind": "TASK_QUEUE_KIND_NORMAL"
+ },
+ "startToCloseTimeout": "10s",
+ "attempt": 1
+ }
+ },
+ {
+ "eventId": "3",
+ "eventTime": "2026-10-29T13:13:12.502468791Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
+ "taskId": "1048593",
+ "workflowTaskStartedEventAttributes": {
+ "scheduledEventId": "2",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "7bb5d28f-ab9e-4a33-a820-02f5d4500c25",
+ "historySizeBytes": "468",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "4",
+ "eventTime": "2026-10-29T13:13:12.658641824Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
+ "taskId": "1048599",
+ "workflowTaskCompletedEventAttributes": {
+ "scheduledEventId": "2",
+ "startedEventId": "3",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ },
+ "sdkMetadata": {
+ "coreUsedFlags": [
+ 1,
+ 2,
+ 3
+ ],
+ "sdkName": "temporal-python",
+ "sdkVersion": "1.27.2"
+ },
+ "meteringMetadata": {}
+ }
+ },
+ {
+ "eventId": "5",
+ "eventTime": "2026-10-29T13:13:12.658932991Z",
+ "eventType": "EVENT_TYPE_MARKER_RECORDED",
+ "taskId": "1048600",
+ "markerRecordedEventAttributes": {
+ "markerName": "core_local_activity",
+ "details": {
+ "data": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJzZXEiOjEsImF0dGVtcHQiOjEsImFjdGl2aXR5X2lkIjoiMSIsImFjdGl2aXR5X3R5cGUiOiJhZ2dyZWdhdGVfZG9jdW1lbnRzX2luX21vbmdvZGIiLCJjb21wbGV0ZV90aW1lIjp7InNlY29uZHMiOjE3OTMyNzk1OTIsIm5hbm9zIjo1MTI0OTkwOTN9LCJiYWNrb2ZmIjpudWxsLCJvcmlnaW5hbF9zY2hlZHVsZV90aW1lIjp7InNlY29uZHMiOjE3OTMyNzk1OTIsIm5hbm9zIjo1NTkyMjA1MzR9fQ=="
+ }
+ ]
+ },
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "W3siYWN0aXZlIjp0cnVlLCJmcmVxdWVuY3kiOiIxbSIsIm1vZGVsIjp7Im5hbWUiOiJNb2RlbCBtb2RlbC1vcGMifSwibW9kZWxfaWQiOiJtb2RlbC1vcGMiLCJyZWFkX3RhZ3MiOlt7ImFnZ3JfZnVuYyI6ImF2ZyIsImZyZXF1ZW5jeSI6MTAwMCwidGFnX2lkIjo3fV0sInNjaGVkdWxlX25hbWUiOiJvcGMtdGFnZGIiLCJ1cGRhdGVkX2F0IjoiMjAyNi0xMC0yNyAxMzoxMzoxMi4zOTEwMDArMDAwMCIsIndvcmtmbG93X3R5cGUiOiJzY291dGVyIn0seyJhY3RpdmUiOnRydWUsImZyZXF1ZW5jeSI6IjFtIiwibW9kZWwiOnsibmFtZSI6Ik1vZGVsIG1vZGVsLW9wYyJ9LCJtb2RlbF9pZCI6Im1vZGVsLW9wYyIsInF1ZXJ5IjoiU0VMRUNUIDEiLCJzY2hlZHVsZV9uYW1lIjoicHJlZC10YWdkYiIsInVwZGF0ZWRfYXQiOiIyMDI2LTEwLTI3IDEzOjEzOjEyLjM5MTAwMCswMDAwIiwid29ya2Zsb3dfdHlwZSI6InByZWRpY3Rpb25zX2JhdGNoIiwid3JpdGVfdGFncyI6W3sidGFnX2lkIjoxMDF9LHsidGFnX2lkIjoxMDJ9LHsidGFnX2lkIjo3fSx7InRhZ19pZCI6MTAzfSx7InRhZ19pZCI6NDA0fSx7InRhZ19uYW1lIjoiUFJFRC1OTy1JRCJ9XX1d"
+ }
+ ]
+ }
+ },
+ "workflowTaskCompletedEventId": "4"
+ }
+ },
+ {
+ "eventId": "6",
+ "eventTime": "2026-10-29T13:13:12.658993409Z",
+ "eventType": "EVENT_TYPE_MARKER_RECORDED",
+ "taskId": "1048601",
+ "markerRecordedEventAttributes": {
+ "markerName": "core_local_activity",
+ "details": {
+ "data": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJzZXEiOjIsImF0dGVtcHQiOjEsImFjdGl2aXR5X2lkIjoiMiIsImFjdGl2aXR5X3R5cGUiOiJmaW5kX2RvY3VtZW50c19pbl9tb25nb2RiIiwiY29tcGxldGVfdGltZSI6eyJzZWNvbmRzIjoxNzkzMjc5NTkyLCJuYW5vcyI6NTEzNzQ3Nzg4fSwiYmFja29mZiI6bnVsbCwib3JpZ2luYWxfc2NoZWR1bGVfdGltZSI6eyJzZWNvbmRzIjoxNzkzMjc5NTkyLCJuYW5vcyI6NTU5MjQ1Njg1fX0="
+ }
+ ]
+ },
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "W10="
+ }
+ ]
+ }
+ },
+ "workflowTaskCompletedEventId": "4"
+ }
+ },
+ {
+ "eventId": "7",
+ "eventTime": "2026-10-29T13:13:12.659003855Z",
+ "eventType": "EVENT_TYPE_MARKER_RECORDED",
+ "taskId": "1048602",
+ "markerRecordedEventAttributes": {
+ "markerName": "core_local_activity",
+ "details": {
+ "data": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJzZXEiOjMsImF0dGVtcHQiOjEsImFjdGl2aXR5X2lkIjoiMyIsImFjdGl2aXR5X3R5cGUiOiJsb2FkX29wY19zbG90cyIsImNvbXBsZXRlX3RpbWUiOnsic2Vjb25kcyI6MTc5MzI3OTU5MiwibmFub3MiOjUwOTUwNTg0OH0sImJhY2tvZmYiOm51bGwsIm9yaWdpbmFsX3NjaGVkdWxlX3RpbWUiOnsic2Vjb25kcyI6MTc5MzI3OTU5MiwibmFub3MiOjU1OTI1MDMwNH19"
+ }
+ ]
+ },
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "e30="
+ }
+ ]
+ }
+ },
+ "workflowTaskCompletedEventId": "4"
+ }
+ },
+ {
+ "eventId": "8",
+ "eventTime": "2026-10-29T13:13:12.659014273Z",
+ "eventType": "EVENT_TYPE_MARKER_RECORDED",
+ "taskId": "1048603",
+ "markerRecordedEventAttributes": {
+ "markerName": "core_local_activity",
+ "details": {
+ "data": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJzZXEiOjQsImF0dGVtcHQiOjEsImFjdGl2aXR5X2lkIjoiNCIsImFjdGl2aXR5X3R5cGUiOiJsb2FkX2FjdGl2ZV9pbmdlc3RvcnMiLCJjb21wbGV0ZV90aW1lIjp7InNlY29uZHMiOjE3OTMyNzk1OTIsIm5hbm9zIjo1MDkwMjQ5MjV9LCJiYWNrb2ZmIjpudWxsLCJvcmlnaW5hbF9zY2hlZHVsZV90aW1lIjp7InNlY29uZHMiOjE3OTMyNzk1OTIsIm5hbm9zIjo1NTkyNjI5MTB9fQ=="
+ }
+ ]
+ },
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "WyJoZWFydGJlYXQ6aW5nZXN0b3I6MSJd"
+ }
+ ]
+ }
+ },
+ "workflowTaskCompletedEventId": "4"
+ }
+ },
+ {
+ "eventId": "9",
+ "eventTime": "2026-10-29T13:13:12.659024082Z",
+ "eventType": "EVENT_TYPE_MARKER_RECORDED",
+ "taskId": "1048604",
+ "markerRecordedEventAttributes": {
+ "markerName": "core_local_activity",
+ "details": {
+ "data": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJzZXEiOjUsImF0dGVtcHQiOjEsImFjdGl2aXR5X2lkIjoiNSIsImFjdGl2aXR5X3R5cGUiOiJhZ2dyZWdhdGVfZG9jdW1lbnRzX2luX21vbmdvZGIiLCJjb21wbGV0ZV90aW1lIjp7InNlY29uZHMiOjE3OTMyNzk1OTIsIm5hbm9zIjo1MTk0MDY4NDN9LCJiYWNrb2ZmIjpudWxsLCJvcmlnaW5hbF9zY2hlZHVsZV90aW1lIjp7InNlY29uZHMiOjE3OTMyNzk1OTIsIm5hbm9zIjo1ODUwMTc0MDZ9fQ=="
+ }
+ ]
+ },
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "W3siY29ubmVjdG9yX2lkIjoicGxhbnQiLCJuYW1lIjoicGxhbnQiLCJzZXJ2ZXJfdXJpIjoidXJuOnBsYW50Iiwic3Vic2NyaXB0aW9uX3BlcmlvZF9tcyI6NTAwLCJ1cmwiOiJvcGMudGNwOi8vcGxhbnQ6NDg0MCIsInVzZV9zZWN1cml0eSI6ZmFsc2V9XQ=="
+ }
+ ]
+ }
+ },
+ "workflowTaskCompletedEventId": "4"
+ }
+ },
+ {
+ "eventId": "10",
+ "eventTime": "2026-10-29T13:13:12.659032932Z",
+ "eventType": "EVENT_TYPE_MARKER_RECORDED",
+ "taskId": "1048605",
+ "markerRecordedEventAttributes": {
+ "markerName": "core_local_activity",
+ "details": {
+ "data": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJzZXEiOjYsImF0dGVtcHQiOjEsImFjdGl2aXR5X2lkIjoiNiIsImFjdGl2aXR5X3R5cGUiOiJhZ2dyZWdhdGVfZG9jdW1lbnRzX2luX21vbmdvZGIiLCJjb21wbGV0ZV90aW1lIjp7InNlY29uZHMiOjE3OTMyNzk1OTIsIm5hbm9zIjo1MTk2NTEwNDd9LCJiYWNrb2ZmIjpudWxsLCJvcmlnaW5hbF9zY2hlZHVsZV90aW1lIjp7InNlY29uZHMiOjE3OTMyNzk1OTIsIm5hbm9zIjo1ODUwMjUyNDl9fQ=="
+ }
+ ]
+ },
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "W3siYWRkcmVzcyI6Im5zPTM7aT0xMDAxIiwiY2F0ZWdvcnkiOiJSZWFkIiwiY29ubmVjdG9yX2lkIjoicGxhbnQiLCJzdGF0dXMiOiJBY3RpdmUiLCJ0YWdfaWQiOjcsInRhZ19uYW1lIjoiUHJlc3N1cmUifSx7ImFkZHJlc3MiOiJucz0zO2k9MjAwMSIsImNhdGVnb3J5IjoiV3JpdGUiLCJjb25uZWN0b3JfaWQiOiJwbGFudCIsImRhdGFfdHlwZSI6ImRvdWJsZSIsInN0YXR1cyI6IkFjdGl2ZSIsInRhZ19pZCI6MTAxLCJ0YWdfbmFtZSI6IlBSRUQtMTAxIn0seyJhZGRyZXNzIjoibnM9MztpPTIwMDIiLCJjYXRlZ29yeSI6IlByZWRpY3Rpb24gQ29uZmlkZW5jZSIsImNvbm5lY3Rvcl9pZCI6InBsYW50Iiwic3RhdHVzIjoiQWN0aXZlIiwidGFnX2lkIjoxMDIsInRhZ19uYW1lIjoiQ09ORi0xMDEifSx7ImFkZHJlc3MiOiJucz0zO2k9MjAwMyIsImNhdGVnb3J5IjoiV3JpdGUiLCJjb25uZWN0b3JfaWQiOiJwbGFudCIsInN0YXR1cyI6IkluYWN0aXZlIiwidGFnX2lkIjoxMDMsInRhZ19uYW1lIjoiUFJFRC1PTEQifV0="
+ }
+ ]
+ }
+ },
+ "workflowTaskCompletedEventId": "4"
+ }
+ },
+ {
+ "eventId": "11",
+ "eventTime": "2026-10-29T13:13:12.659041082Z",
+ "eventType": "EVENT_TYPE_MARKER_RECORDED",
+ "taskId": "1048606",
+ "markerRecordedEventAttributes": {
+ "markerName": "core_local_activity",
+ "details": {
+ "data": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJzZXEiOjcsImF0dGVtcHQiOjEsImFjdGl2aXR5X2lkIjoiNyIsImFjdGl2aXR5X3R5cGUiOiJmb3JtYXRfc2NoZWR1bGVfY29uZmlnIiwiY29tcGxldGVfdGltZSI6eyJzZWNvbmRzIjoxNzkzMjc5NTkyLCJuYW5vcyI6NTIyNjUyMDA0fSwiYmFja29mZiI6bnVsbCwib3JpZ2luYWxfc2NoZWR1bGVfdGltZSI6eyJzZWNvbmRzIjoxNzkzMjc5NTkyLCJuYW5vcyI6NjAwNzQwMjM2fX0="
+ }
+ ]
+ },
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "e30="
+ }
+ ]
+ }
+ },
+ "workflowTaskCompletedEventId": "4"
+ }
+ },
+ {
+ "eventId": "12",
+ "eventTime": "2026-10-29T13:13:12.659049234Z",
+ "eventType": "EVENT_TYPE_MARKER_RECORDED",
+ "taskId": "1048607",
+ "markerRecordedEventAttributes": {
+ "markerName": "core_local_activity",
+ "details": {
+ "data": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJzZXEiOjksImF0dGVtcHQiOjEsImFjdGl2aXR5X2lkIjoiOSIsImFjdGl2aXR5X3R5cGUiOiJwcm9jZXNzX3Nsb3RzIiwiY29tcGxldGVfdGltZSI6eyJzZWNvbmRzIjoxNzkzMjc5NTkyLCJuYW5vcyI6NTIyOTc2MjYxfSwiYmFja29mZiI6bnVsbCwib3JpZ2luYWxfc2NoZWR1bGVfdGltZSI6eyJzZWNvbmRzIjoxNzkzMjc5NTkyLCJuYW5vcyI6NjAwNzg1NTQ0fX0="
+ }
+ ]
+ },
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyIxIjp7InBsYW50Ijp7Im5hbWUiOiJwbGFudCIsInNlcnZlcl9pZCI6InBsYW50Iiwic2VydmVyX3VyaSI6InVybjpwbGFudCIsInN1YnNjcmlwdGlvbl9wZXJpb2RfbXMiOjUwMCwidGFncyI6eyJucz0zO2k9MTAwMSI6eyJhY3RpdmVfcmVhZCI6eyJlbmFibGVkIjpmYWxzZX0sImZyZXF1ZW5jeSI6MTAwMCwibmFtZXMiOlt7ImNvbGxlY3Rpb24iOiJyYXdfb3BjLXRhZ2RiIiwibW9kZWxfaWQiOiJtb2RlbC1vcGMiLCJtb2RlbF9uYW1lIjoiTW9kZWwgbW9kZWwtb3BjIiwidGFnX25hbWUiOiJQcmVzc3VyZSJ9XX19LCJ1cmwiOiJvcGMudGNwOi8vcGxhbnQ6NDg0MCIsInVzZV9zZWN1cml0eSI6ZmFsc2V9fX0="
+ }
+ ]
+ }
+ },
+ "workflowTaskCompletedEventId": "4"
+ }
+ },
+ {
+ "eventId": "13",
+ "eventTime": "2026-10-29T13:13:12.659058653Z",
+ "eventType": "EVENT_TYPE_MARKER_RECORDED",
+ "taskId": "1048608",
+ "markerRecordedEventAttributes": {
+ "markerName": "core_local_activity",
+ "details": {
+ "data": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJzZXEiOjgsImF0dGVtcHQiOjEsImFjdGl2aXR5X2lkIjoiOCIsImFjdGl2aXR5X3R5cGUiOiJwcm9jZXNzX3NjaGVkdWxlcyIsImNvbXBsZXRlX3RpbWUiOnsic2Vjb25kcyI6MTc5MzI3OTU5MiwibmFub3MiOjU1ODAwODU4NX0sImJhY2tvZmYiOm51bGwsIm9yaWdpbmFsX3NjaGVkdWxlX3RpbWUiOnsic2Vjb25kcyI6MTc5MzI3OTU5MiwibmFub3MiOjYwMDc3Nzc5NX19"
+ }
+ ]
+ },
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJsYWJvcmlvdXMiOnsicHJlZC10YWdkYiI6eyJkYXRldGltZV9jb2x1bW5zIjpbXSwiZXhlY3V0aW9uX3RpbWVvdXRfc2Vjb25kcyI6MzAwLCJmcmVxdWVuY3kiOiIxbSIsImlucHV0X2ZpbHRlcnMiOnsiRU1QVFlfREFUQSI6eyJjb25maWciOnt9LCJwb2xpY3kiOiJTVE9QIn19LCJtYXhfcmV0cnlfcG9saWN5IjoxLCJtbGZsb3dfcHJlZGljdF9maWx0ZXJzIjp7IkFQSV9FUlJPUiI6eyJjb25maWciOnt9LCJwb2xpY3kiOiJTVE9QIn19LCJtbGZsb3dfdHJhbnNmb3JtX2ZpbHRlcnMiOnsiQVBJX0VSUk9SIjp7ImNvbmZpZyI6e30sInBvbGljeSI6IlNUT1AifSwiRU1QVFlfREFUQSI6eyJjb25maWciOnt9LCJwb2xpY3kiOiJTVE9QIn19LCJtb2RlbF9jb25maWciOnt9LCJtb2RlbF9pZCI6Im1vZGVsLW9wYyIsIm1vZGVsX25hbWUiOiJNb2RlbCBtb2RlbC1vcGMiLCJvZmZzZXQiOiIwbSIsIm9uX2NvbmZsaWN0IjoiZXJyb3IiLCJvcGNfb3V0cHV0X2NvbmZpZyI6eyJwbGFudCI6eyJjb25maWRlbmNlX3RhZ3MiOnsibnM9MztpPTIwMDIiOnsiZGF0YV90eXBlIjoiZmxvYXQifX0sInByZWRpY3Rpb25fdGFncyI6eyJucz0zO2k9MjAwMSI6eyJkYXRhX3R5cGUiOiJkb3VibGUifX19fSwicGF0aF9wcmlvcml0eSI6WyJTVE9QIiwiQ09OVElOVUUiLCJSRVBFQVQiXSwicGlfd2ViX2FwaV9vdXRwdXRfY29uZmlnIjp7fSwicHJlZGljdGlvbnNfc3RvcmFnZV9wb2xpY3kiOiJsdHM6MSIsInF1ZXJ5IjoiU0VMRUNUIDEiLCJyZXRlbnRpb25fdGltZSI6MzYwMCwicnVudGltZSI6ImxlZ2FjeSIsInNhdmVfdHJhbnNmb3JtIjp0cnVlLCJzY2hlZHVsZV9uYW1lIjoicHJlZC10YWdkYiIsInNjaGVtYSI6InNpZW50aWFfZGF0YSIsInRhYmxlX25hbWUiOiJwcmVkaWN0aW9ucyIsInRhc2tfdGltZW91dF9zZWNvbmRzIjozMDAsInRyYW5zZm9ybV90YWJsZV9uYW1lIjoidHJhbnNmb3JtZWRfZGF0YSIsInVwZGF0ZWRfYXQiOiIyMDI2LTEwLTI3IDEzOjEzOjEyLjM5MTAwMCswMDAwIiwid29ya2Zsb3dfdHlwZSI6InByZWRpY3Rpb25zX2JhdGNoIn19LCJzY291dGVyIjp7Im9wYy10YWdkYiI6eyJkZWJ1Z19kYXRhX3BhY2thZ2UiOmZhbHNlLCJleGVjdXRpb25fdGltZW91dF9zZWNvbmRzIjozMDAsImZpbGxfbWlzc2luZ190YWdzIjpmYWxzZSwiZmlsdGVycyI6e30sImZyZXF1ZW5jeSI6IjFtIiwibWF4X3JldHJ5X3BvbGljeSI6MSwibW9kZWxfY29uZmlnIjp7fSwibW9kZWxfaWQiOiJtb2RlbC1vcGMiLCJtb2RlbF9uYW1lIjoiTW9kZWwgbW9kZWwtb3BjIiwibW9kZWxfdGFncyI6eyJQcmVzc3VyZSI6eyJhZ2dyX2Z1bmMiOiJhdmciLCJkYXRhX3JhbmdlIjpbLTEwMCwxMDBdfX0sIm9mZnNldCI6IjBtIiwib25fY29uZmxpY3QiOiJlcnJvciIsInJldGVudGlvbl90aW1lIjozNjAwLCJydW50aW1lIjoibGVnYWN5Iiwic2NoZWR1bGVfbmFtZSI6Im9wYy10YWdkYiIsInNjaGVtYSI6InNpZW50aWFfZGF0YSIsInRhYmxlX25hbWUiOiJsYWJvcmlvdXNfZGF0YSIsInRhc2tfdGltZW91dF9zZWNvbmRzIjozMDAsInRvcGljIjoicmF3X29wYy10YWdkYiIsInRyaWdnZXJfbGFib3Jpb3VzIjpmYWxzZSwidXBkYXRlZF9hdCI6IjIwMjYtMTAtMjcgMTM6MTM6MTIuMzkxMDAwKzAwMDAiLCJ3b3JrZmxvd190eXBlIjoic2NvdXRlciJ9fX0="
+ }
+ ]
+ }
+ },
+ "workflowTaskCompletedEventId": "4"
+ }
+ },
+ {
+ "eventId": "14",
+ "eventTime": "2026-10-29T13:13:12.659132860Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
+ "taskId": "1048609",
+ "activityTaskScheduledEventAttributes": {
+ "activityId": "12",
+ "activityType": {
+ "name": "normalize_schedules"
+ },
+ "taskQueue": {
+ "name": "orchestrator-test-queue",
+ "kind": "TASK_QUEUE_KIND_NORMAL"
+ },
+ "header": {},
+ "input": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJtZXRhZGF0YSI6eyJtb2RlbF9pZCI6Ii0iLCJtb2RlbF9uYW1lIjoiLSIsInNjaGVkdWxlX25hbWUiOiJvcmNoZXN0cmF0b3ItZTJlLXRhZy1kYXRhYmFzZSIsIndvcmtmbG93X25hbWUiOiJvcmNoZXN0cmF0b3IifSwib3JjaGVzdHJhdGVkX3NjaGVkdWxlcyI6e319"
+ }
+ ]
+ },
+ "scheduleToCloseTimeout": "0s",
+ "scheduleToStartTimeout": "0s",
+ "startToCloseTimeout": "60s",
+ "heartbeatTimeout": "0s",
+ "workflowTaskCompletedEventId": "4",
+ "retryPolicy": {
+ "initialInterval": "1s",
+ "backoffCoefficient": 2.0,
+ "maximumInterval": "60s",
+ "maximumAttempts": 1
+ },
+ "useWorkflowBuildId": true,
+ "priority": {}
+ }
+ },
+ {
+ "eventId": "15",
+ "eventTime": "2026-10-29T13:13:12.659650616Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
+ "taskId": "1048610",
+ "activityTaskScheduledEventAttributes": {
+ "activityId": "13",
+ "activityType": {
+ "name": "create_collection_with_ttl_index"
+ },
+ "taskQueue": {
+ "name": "orchestrator-test-queue",
+ "kind": "TASK_QUEUE_KIND_NORMAL"
+ },
+ "header": {},
+ "input": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJtZXRhZGF0YSI6eyJtb2RlbF9pZCI6Ii0iLCJtb2RlbF9uYW1lIjoiLSIsInNjaGVkdWxlX25hbWUiOiJvcmNoZXN0cmF0b3ItZTJlLXRhZy1kYXRhYmFzZSIsIndvcmtmbG93X25hbWUiOiJvcmNoZXN0cmF0b3IifSwicGlwZWxpbmVzIjp7Im9wYy10YWdkYiI6eyJkZWJ1Z19kYXRhX3BhY2thZ2UiOmZhbHNlLCJleGVjdXRpb25fdGltZW91dF9zZWNvbmRzIjozMDAsImZpbGxfbWlzc2luZ190YWdzIjpmYWxzZSwiZmlsdGVycyI6e30sImZyZXF1ZW5jeSI6IjFtIiwibWF4X3JldHJ5X3BvbGljeSI6MSwibW9kZWxfY29uZmlnIjp7fSwibW9kZWxfaWQiOiJtb2RlbC1vcGMiLCJtb2RlbF9uYW1lIjoiTW9kZWwgbW9kZWwtb3BjIiwibW9kZWxfdGFncyI6eyJQcmVzc3VyZSI6eyJhZ2dyX2Z1bmMiOiJhdmciLCJkYXRhX3JhbmdlIjpbLTEwMCwxMDBdfX0sIm9mZnNldCI6IjBtIiwib25fY29uZmxpY3QiOiJlcnJvciIsInJldGVudGlvbl90aW1lIjozNjAwLCJydW50aW1lIjoibGVnYWN5Iiwic2NoZWR1bGVfbmFtZSI6Im9wYy10YWdkYiIsInNjaGVtYSI6InNpZW50aWFfZGF0YSIsInRhYmxlX25hbWUiOiJsYWJvcmlvdXNfZGF0YSIsInRhc2tfdGltZW91dF9zZWNvbmRzIjozMDAsInRvcGljIjoicmF3X29wYy10YWdkYiIsInRyaWdnZXJfbGFib3Jpb3VzIjpmYWxzZSwidXBkYXRlZF9hdCI6IjIwMjYtMTAtMjcgMTM6MTM6MTIuMzkxMDAwKzAwMDAiLCJ3b3JrZmxvd190eXBlIjoic2NvdXRlciJ9fX0="
+ }
+ ]
+ },
+ "scheduleToCloseTimeout": "0s",
+ "scheduleToStartTimeout": "0s",
+ "startToCloseTimeout": "60s",
+ "heartbeatTimeout": "0s",
+ "workflowTaskCompletedEventId": "4",
+ "retryPolicy": {
+ "initialInterval": "1s",
+ "backoffCoefficient": 2.0,
+ "maximumInterval": "60s",
+ "maximumAttempts": 1
+ },
+ "useWorkflowBuildId": true,
+ "priority": {}
+ }
+ },
+ {
+ "eventId": "16",
+ "eventTime": "2026-10-29T13:13:12.659676093Z",
+ "eventType": "EVENT_TYPE_MARKER_RECORDED",
+ "taskId": "1048611",
+ "markerRecordedEventAttributes": {
+ "markerName": "core_local_activity",
+ "details": {
+ "data": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJzZXEiOjEwLCJhdHRlbXB0IjoxLCJhY3Rpdml0eV9pZCI6IjEwIiwiYWN0aXZpdHlfdHlwZSI6ImNyZWF0ZV9zY2hlZHVsZV9jb25maWciLCJjb21wbGV0ZV90aW1lIjp7InNlY29uZHMiOjE3OTMyNzk1OTIsIm5hbm9zIjo1NjEzMzAwOTh9LCJiYWNrb2ZmIjpudWxsLCJvcmlnaW5hbF9zY2hlZHVsZV90aW1lIjp7InNlY29uZHMiOjE3OTMyNzk1OTIsIm5hbm9zIjo2NDg1NDUwMTV9fQ=="
+ }
+ ]
+ },
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJ0b19jcmVhdGUiOnsibGFib3Jpb3VzIjp7InByZWQtdGFnZGIiOnsiY29uZmlnX2hhc2giOiI3NDVlMzhiYWE5MmRmNTU4MWY0YmY4YjNmMDhmYTdmNTM4NGNmZDkyMGE2YTg4MGMzZjE1MDM4Njg0NTQzYTY3IiwiZGF0ZXRpbWVfY29sdW1ucyI6W10sImV4ZWN1dGlvbl90aW1lb3V0X3NlY29uZHMiOjMwMCwiZnJlcXVlbmN5IjoiMW0iLCJpbnB1dF9maWx0ZXJzIjp7IkVNUFRZX0RBVEEiOnsiY29uZmlnIjp7fSwicG9saWN5IjoiU1RPUCJ9fSwibWF4X3JldHJ5X3BvbGljeSI6MSwibWxmbG93X3ByZWRpY3RfZmlsdGVycyI6eyJBUElfRVJST1IiOnsiY29uZmlnIjp7fSwicG9saWN5IjoiU1RPUCJ9fSwibWxmbG93X3RyYW5zZm9ybV9maWx0ZXJzIjp7IkFQSV9FUlJPUiI6eyJjb25maWciOnt9LCJwb2xpY3kiOiJTVE9QIn0sIkVNUFRZX0RBVEEiOnsiY29uZmlnIjp7fSwicG9saWN5IjoiU1RPUCJ9fSwibW9kZWxfY29uZmlnIjp7fSwibW9kZWxfaWQiOiJtb2RlbC1vcGMiLCJtb2RlbF9uYW1lIjoiTW9kZWwgbW9kZWwtb3BjIiwib2Zmc2V0IjoiMG0iLCJvbl9jb25mbGljdCI6ImVycm9yIiwib3BjX291dHB1dF9jb25maWciOnsicGxhbnQiOnsiY29uZmlkZW5jZV90YWdzIjp7Im5zPTM7aT0yMDAyIjp7ImRhdGFfdHlwZSI6ImZsb2F0In19LCJwcmVkaWN0aW9uX3RhZ3MiOnsibnM9MztpPTIwMDEiOnsiZGF0YV90eXBlIjoiZG91YmxlIn19fX0sInBhdGhfcHJpb3JpdHkiOlsiU1RPUCIsIkNPTlRJTlVFIiwiUkVQRUFUIl0sInBpX3dlYl9hcGlfb3V0cHV0X2NvbmZpZyI6e30sInByZWRpY3Rpb25zX3N0b3JhZ2VfcG9saWN5IjoibHRzOjEiLCJxdWVyeSI6IlNFTEVDVCAxIiwicmV0ZW50aW9uX3RpbWUiOjM2MDAsInJ1bnRpbWUiOiJsZWdhY3kiLCJzYXZlX3RyYW5zZm9ybSI6dHJ1ZSwic2NoZWR1bGVfbmFtZSI6InByZWQtdGFnZGIiLCJzY2hlbWEiOiJzaWVudGlhX2RhdGEiLCJ0YWJsZV9uYW1lIjoicHJlZGljdGlvbnMiLCJ0YXNrX3RpbWVvdXRfc2Vjb25kcyI6MzAwLCJ0cmFuc2Zvcm1fdGFibGVfbmFtZSI6InRyYW5zZm9ybWVkX2RhdGEiLCJ1cGRhdGVkX2F0IjoiMjAyNi0xMC0yNyAxMzoxMzoxMi4zOTEwMDArMDAwMCIsIndvcmtmbG93X3R5cGUiOiJwcmVkaWN0aW9uc19iYXRjaCJ9fSwic2NvdXRlciI6eyJvcGMtdGFnZGIiOnsiY29uZmlnX2hhc2giOiIxM2RlN2FjMDlkNDMxYTM1MTQ1YmE4M2ZlZWM0NzBkNzM5MDlmZjZjNTJmZTViMThkM2NjOGE3NzhjZjdjOGQ4IiwiZGVidWdfZGF0YV9wYWNrYWdlIjpmYWxzZSwiZXhlY3V0aW9uX3RpbWVvdXRfc2Vjb25kcyI6MzAwLCJmaWxsX21pc3NpbmdfdGFncyI6ZmFsc2UsImZpbHRlcnMiOnt9LCJmcmVxdWVuY3kiOiIxbSIsIm1heF9yZXRyeV9wb2xpY3kiOjEsIm1vZGVsX2NvbmZpZyI6e30sIm1vZGVsX2lkIjoibW9kZWwtb3BjIiwibW9kZWxfbmFtZSI6Ik1vZGVsIG1vZGVsLW9wYyIsIm1vZGVsX3RhZ3MiOnsiUHJlc3N1cmUiOnsiYWdncl9mdW5jIjoiYXZnIiwiZGF0YV9yYW5nZSI6Wy0xMDAsMTAwXX19LCJvZmZzZXQiOiIwbSIsIm9uX2NvbmZsaWN0IjoiZXJyb3IiLCJyZXRlbnRpb25fdGltZSI6MzYwMCwicnVudGltZSI6ImxlZ2FjeSIsInNjaGVkdWxlX25hbWUiOiJvcGMtdGFnZGIiLCJzY2hlbWEiOiJzaWVudGlhX2RhdGEiLCJ0YWJsZV9uYW1lIjoibGFib3Jpb3VzX2RhdGEiLCJ0YXNrX3RpbWVvdXRfc2Vjb25kcyI6MzAwLCJ0b3BpYyI6InJhd19vcGMtdGFnZGIiLCJ0cmlnZ2VyX2xhYm9yaW91cyI6ZmFsc2UsInVwZGF0ZWRfYXQiOiIyMDI2LTEwLTI3IDEzOjEzOjEyLjM5MTAwMCswMDAwIiwid29ya2Zsb3dfdHlwZSI6InNjb3V0ZXIifX19LCJ0b19kZWxldGUiOnsibGFib3Jpb3VzIjpbXSwic2NvdXRlciI6W119LCJ0b191cGRhdGUiOnsibGFib3Jpb3VzIjp7fSwic2NvdXRlciI6e319fQ=="
+ }
+ ]
+ }
+ },
+ "workflowTaskCompletedEventId": "4"
+ }
+ },
+ {
+ "eventId": "17",
+ "eventTime": "2026-10-29T13:13:12.659686364Z",
+ "eventType": "EVENT_TYPE_MARKER_RECORDED",
+ "taskId": "1048612",
+ "markerRecordedEventAttributes": {
+ "markerName": "core_local_activity",
+ "details": {
+ "data": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJzZXEiOjExLCJhdHRlbXB0IjoxLCJhY3Rpdml0eV9pZCI6IjExIiwiYWN0aXZpdHlfdHlwZSI6ImNyZWF0ZV9zbG90X2NvbmZpZyIsImNvbXBsZXRlX3RpbWUiOnsic2Vjb25kcyI6MTc5MzI3OTU5MiwibmFub3MiOjU2MDk5ODQ3Nn0sImJhY2tvZmYiOm51bGwsIm9yaWdpbmFsX3NjaGVkdWxlX3RpbWUiOnsic2Vjb25kcyI6MTc5MzI3OTU5MiwibmFub3MiOjY0ODU2MjMxNX19"
+ }
+ ]
+ },
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJ0b19kZWxldGUiOltdLCJ0b19pbnNlcnQiOnsiMSI6eyJwbGFudCI6eyJuYW1lIjoicGxhbnQiLCJzZXJ2ZXJfaWQiOiJwbGFudCIsInNlcnZlcl91cmkiOiJ1cm46cGxhbnQiLCJzdWJzY3JpcHRpb25fcGVyaW9kX21zIjo1MDAsInRhZ3MiOnsibnM9MztpPTEwMDEiOnsiYWN0aXZlX3JlYWQiOnsiZW5hYmxlZCI6ZmFsc2V9LCJmcmVxdWVuY3kiOjEwMDAsIm5hbWVzIjpbeyJjb2xsZWN0aW9uIjoicmF3X29wYy10YWdkYiIsIm1vZGVsX2lkIjoibW9kZWwtb3BjIiwibW9kZWxfbmFtZSI6Ik1vZGVsIG1vZGVsLW9wYyIsInRhZ19uYW1lIjoiUHJlc3N1cmUifV19fSwidXJsIjoib3BjLnRjcDovL3BsYW50OjQ4NDAiLCJ1c2Vfc2VjdXJpdHkiOmZhbHNlfX19fQ=="
+ }
+ ]
+ }
+ },
+ "workflowTaskCompletedEventId": "4"
+ }
+ },
+ {
+ "eventId": "18",
+ "eventTime": "2026-10-29T13:13:12.659701168Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
+ "taskId": "1048616",
+ "activityTaskStartedEventAttributes": {
+ "scheduledEventId": "14",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "7553de1d-b92b-45a2-8532-069e2d065d9c",
+ "attempt": 1,
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "19",
+ "eventTime": "2026-10-29T13:13:12.681770967Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
+ "taskId": "1048617",
+ "activityTaskCompletedEventAttributes": {
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "YmluYXJ5L251bGw="
+ }
+ }
+ ]
+ },
+ "scheduledEventId": "14",
+ "startedEventId": "18",
+ "identity": "69421@grezewave-HP-ZBook-17-G5"
+ }
+ },
+ {
+ "eventId": "20",
+ "eventTime": "2026-10-29T13:13:12.681978464Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
+ "taskId": "1048618",
+ "workflowTaskScheduledEventAttributes": {
+ "taskQueue": {
+ "name": "69421@grezewave-HP-ZBook-17-G5-6d452a12d16c4ff7b47360d8b9ac5b8e",
+ "kind": "TASK_QUEUE_KIND_STICKY",
+ "normalName": "orchestrator-test-queue"
+ },
+ "startToCloseTimeout": "10s",
+ "attempt": 1
+ }
+ },
+ {
+ "eventId": "21",
+ "eventTime": "2026-10-29T13:13:12.688466109Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
+ "taskId": "1048623",
+ "workflowTaskStartedEventAttributes": {
+ "scheduledEventId": "20",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "c79c65f2-fc73-4613-b95d-d6e66a5a592a",
+ "historySizeBytes": "11803",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "22",
+ "eventTime": "2026-10-29T13:13:12.702514917Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
+ "taskId": "1048627",
+ "workflowTaskCompletedEventAttributes": {
+ "scheduledEventId": "20",
+ "startedEventId": "21",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ },
+ "sdkMetadata": {},
+ "meteringMetadata": {}
+ }
+ },
+ {
+ "eventId": "23",
+ "eventTime": "2026-10-29T13:13:12.659759616Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
+ "taskId": "1048629",
+ "activityTaskStartedEventAttributes": {
+ "scheduledEventId": "15",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "f2253b04-baa7-4545-ba04-c3a17f49f363",
+ "attempt": 1,
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "24",
+ "eventTime": "2026-10-29T13:13:12.723748812Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
+ "taskId": "1048630",
+ "activityTaskCompletedEventAttributes": {
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "YmluYXJ5L251bGw="
+ }
+ }
+ ]
+ },
+ "scheduledEventId": "15",
+ "startedEventId": "23",
+ "identity": "69421@grezewave-HP-ZBook-17-G5"
+ }
+ },
+ {
+ "eventId": "25",
+ "eventTime": "2026-10-29T13:13:12.723789639Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
+ "taskId": "1048631",
+ "workflowTaskScheduledEventAttributes": {
+ "taskQueue": {
+ "name": "69421@grezewave-HP-ZBook-17-G5-6d452a12d16c4ff7b47360d8b9ac5b8e",
+ "kind": "TASK_QUEUE_KIND_STICKY",
+ "normalName": "orchestrator-test-queue"
+ },
+ "startToCloseTimeout": "10s",
+ "attempt": 1
+ }
+ },
+ {
+ "eventId": "26",
+ "eventTime": "2026-10-29T13:13:12.729387003Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
+ "taskId": "1048635",
+ "workflowTaskStartedEventAttributes": {
+ "scheduledEventId": "25",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "7e0b1d18-74d0-4a48-a46f-f01f95bdd563",
+ "historySizeBytes": "12399",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "27",
+ "eventTime": "2026-10-29T13:13:12.750783679Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
+ "taskId": "1048642",
+ "workflowTaskCompletedEventAttributes": {
+ "scheduledEventId": "25",
+ "startedEventId": "26",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ },
+ "sdkMetadata": {},
+ "meteringMetadata": {}
+ }
+ },
+ {
+ "eventId": "28",
+ "eventTime": "2026-10-29T13:13:12.750882598Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
+ "taskId": "1048643",
+ "activityTaskScheduledEventAttributes": {
+ "activityId": "14",
+ "activityType": {
+ "name": "delete_slots"
+ },
+ "taskQueue": {
+ "name": "orchestrator-test-queue",
+ "kind": "TASK_QUEUE_KIND_NORMAL"
+ },
+ "header": {},
+ "input": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJtZXRhZGF0YSI6eyJtb2RlbF9pZCI6Ii0iLCJtb2RlbF9uYW1lIjoiLSIsInNjaGVkdWxlX25hbWUiOiJvcmNoZXN0cmF0b3ItZTJlLXRhZy1kYXRhYmFzZSIsIndvcmtmbG93X25hbWUiOiJvcmNoZXN0cmF0b3IifSwidG9fZGVsZXRlIjpbXX0="
+ }
+ ]
+ },
+ "scheduleToCloseTimeout": "0s",
+ "scheduleToStartTimeout": "0s",
+ "startToCloseTimeout": "60s",
+ "heartbeatTimeout": "0s",
+ "workflowTaskCompletedEventId": "27",
+ "retryPolicy": {
+ "initialInterval": "1s",
+ "backoffCoefficient": 2.0,
+ "maximumInterval": "60s",
+ "maximumAttempts": 1
+ },
+ "useWorkflowBuildId": true,
+ "priority": {}
+ }
+ },
+ {
+ "eventId": "29",
+ "eventTime": "2026-10-29T13:13:12.750943708Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
+ "taskId": "1048644",
+ "activityTaskScheduledEventAttributes": {
+ "activityId": "15",
+ "activityType": {
+ "name": "update_slots"
+ },
+ "taskQueue": {
+ "name": "orchestrator-test-queue",
+ "kind": "TASK_QUEUE_KIND_NORMAL"
+ },
+ "header": {},
+ "input": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJtZXRhZGF0YSI6eyJtb2RlbF9pZCI6Ii0iLCJtb2RlbF9uYW1lIjoiLSIsInNjaGVkdWxlX25hbWUiOiJvcmNoZXN0cmF0b3ItZTJlLXRhZy1kYXRhYmFzZSIsIndvcmtmbG93X25hbWUiOiJvcmNoZXN0cmF0b3IifSwidG9faW5zZXJ0Ijp7IjEiOnsicGxhbnQiOnsibmFtZSI6InBsYW50Iiwic2VydmVyX2lkIjoicGxhbnQiLCJzZXJ2ZXJfdXJpIjoidXJuOnBsYW50Iiwic3Vic2NyaXB0aW9uX3BlcmlvZF9tcyI6NTAwLCJ0YWdzIjp7Im5zPTM7aT0xMDAxIjp7ImFjdGl2ZV9yZWFkIjp7ImVuYWJsZWQiOmZhbHNlfSwiZnJlcXVlbmN5IjoxMDAwLCJuYW1lcyI6W3siY29sbGVjdGlvbiI6InJhd19vcGMtdGFnZGIiLCJtb2RlbF9pZCI6Im1vZGVsLW9wYyIsIm1vZGVsX25hbWUiOiJNb2RlbCBtb2RlbC1vcGMiLCJ0YWdfbmFtZSI6IlByZXNzdXJlIn1dfX0sInVybCI6Im9wYy50Y3A6Ly9wbGFudDo0ODQwIiwidXNlX3NlY3VyaXR5IjpmYWxzZX19fX0="
+ }
+ ]
+ },
+ "scheduleToCloseTimeout": "0s",
+ "scheduleToStartTimeout": "0s",
+ "startToCloseTimeout": "60s",
+ "heartbeatTimeout": "0s",
+ "workflowTaskCompletedEventId": "27",
+ "retryPolicy": {
+ "initialInterval": "1s",
+ "backoffCoefficient": 2.0,
+ "maximumInterval": "60s",
+ "maximumAttempts": 1
+ },
+ "useWorkflowBuildId": true,
+ "priority": {}
+ }
+ },
+ {
+ "eventId": "30",
+ "eventTime": "2026-10-29T13:13:12.750977175Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
+ "taskId": "1048645",
+ "activityTaskScheduledEventAttributes": {
+ "activityId": "16",
+ "activityType": {
+ "name": "delete_schedules"
+ },
+ "taskQueue": {
+ "name": "orchestrator-test-queue",
+ "kind": "TASK_QUEUE_KIND_NORMAL"
+ },
+ "header": {},
+ "input": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJtZXRhZGF0YSI6eyJtb2RlbF9pZCI6Ii0iLCJtb2RlbF9uYW1lIjoiLSIsInNjaGVkdWxlX25hbWUiOiJvcmNoZXN0cmF0b3ItZTJlLXRhZy1kYXRhYmFzZSIsIndvcmtmbG93X25hbWUiOiJvcmNoZXN0cmF0b3IifSwic2NoZWR1bGVzIjp7ImxhYm9yaW91cyI6W10sInNjb3V0ZXIiOltdfX0="
+ }
+ ]
+ },
+ "scheduleToCloseTimeout": "0s",
+ "scheduleToStartTimeout": "0s",
+ "startToCloseTimeout": "60s",
+ "heartbeatTimeout": "0s",
+ "workflowTaskCompletedEventId": "27",
+ "retryPolicy": {
+ "initialInterval": "1s",
+ "backoffCoefficient": 2.0,
+ "maximumInterval": "60s",
+ "maximumAttempts": 1
+ },
+ "useWorkflowBuildId": true,
+ "priority": {}
+ }
+ },
+ {
+ "eventId": "31",
+ "eventTime": "2026-10-29T13:13:12.751005908Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
+ "taskId": "1048646",
+ "activityTaskScheduledEventAttributes": {
+ "activityId": "17",
+ "activityType": {
+ "name": "create_schedules"
+ },
+ "taskQueue": {
+ "name": "orchestrator-test-queue",
+ "kind": "TASK_QUEUE_KIND_NORMAL"
+ },
+ "header": {},
+ "input": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJtZXRhZGF0YSI6eyJtb2RlbF9pZCI6Ii0iLCJtb2RlbF9uYW1lIjoiLSIsInNjaGVkdWxlX25hbWUiOiJvcmNoZXN0cmF0b3ItZTJlLXRhZy1kYXRhYmFzZSIsIndvcmtmbG93X25hbWUiOiJvcmNoZXN0cmF0b3IifSwic2NoZWR1bGVzIjp7ImxhYm9yaW91cyI6eyJwcmVkLXRhZ2RiIjp7ImNvbmZpZ19oYXNoIjoiNzQ1ZTM4YmFhOTJkZjU1ODFmNGJmOGIzZjA4ZmE3ZjUzODRjZmQ5MjBhNmE4ODBjM2YxNTAzODY4NDU0M2E2NyIsImRhdGV0aW1lX2NvbHVtbnMiOltdLCJleGVjdXRpb25fdGltZW91dF9zZWNvbmRzIjozMDAsImZyZXF1ZW5jeSI6IjFtIiwiaW5wdXRfZmlsdGVycyI6eyJFTVBUWV9EQVRBIjp7ImNvbmZpZyI6e30sInBvbGljeSI6IlNUT1AifX0sIm1heF9yZXRyeV9wb2xpY3kiOjEsIm1sZmxvd19wcmVkaWN0X2ZpbHRlcnMiOnsiQVBJX0VSUk9SIjp7ImNvbmZpZyI6e30sInBvbGljeSI6IlNUT1AifX0sIm1sZmxvd190cmFuc2Zvcm1fZmlsdGVycyI6eyJBUElfRVJST1IiOnsiY29uZmlnIjp7fSwicG9saWN5IjoiU1RPUCJ9LCJFTVBUWV9EQVRBIjp7ImNvbmZpZyI6e30sInBvbGljeSI6IlNUT1AifX0sIm1vZGVsX2NvbmZpZyI6e30sIm1vZGVsX2lkIjoibW9kZWwtb3BjIiwibW9kZWxfbmFtZSI6Ik1vZGVsIG1vZGVsLW9wYyIsIm9mZnNldCI6IjBtIiwib25fY29uZmxpY3QiOiJlcnJvciIsIm9wY19vdXRwdXRfY29uZmlnIjp7InBsYW50Ijp7ImNvbmZpZGVuY2VfdGFncyI6eyJucz0zO2k9MjAwMiI6eyJkYXRhX3R5cGUiOiJmbG9hdCJ9fSwicHJlZGljdGlvbl90YWdzIjp7Im5zPTM7aT0yMDAxIjp7ImRhdGFfdHlwZSI6ImRvdWJsZSJ9fX19LCJwYXRoX3ByaW9yaXR5IjpbIlNUT1AiLCJDT05USU5VRSIsIlJFUEVBVCJdLCJwaV93ZWJfYXBpX291dHB1dF9jb25maWciOnt9LCJwcmVkaWN0aW9uc19zdG9yYWdlX3BvbGljeSI6Imx0czoxIiwicXVlcnkiOiJTRUxFQ1QgMSIsInJldGVudGlvbl90aW1lIjozNjAwLCJydW50aW1lIjoibGVnYWN5Iiwic2F2ZV90cmFuc2Zvcm0iOnRydWUsInNjaGVkdWxlX25hbWUiOiJwcmVkLXRhZ2RiIiwic2NoZW1hIjoic2llbnRpYV9kYXRhIiwidGFibGVfbmFtZSI6InByZWRpY3Rpb25zIiwidGFza190aW1lb3V0X3NlY29uZHMiOjMwMCwidHJhbnNmb3JtX3RhYmxlX25hbWUiOiJ0cmFuc2Zvcm1lZF9kYXRhIiwidXBkYXRlZF9hdCI6IjIwMjYtMTAtMjcgMTM6MTM6MTIuMzkxMDAwKzAwMDAiLCJ3b3JrZmxvd190eXBlIjoicHJlZGljdGlvbnNfYmF0Y2gifX0sInNjb3V0ZXIiOnsib3BjLXRhZ2RiIjp7ImNvbmZpZ19oYXNoIjoiMTNkZTdhYzA5ZDQzMWEzNTE0NWJhODNmZWVjNDcwZDczOTA5ZmY2YzUyZmU1YjE4ZDNjYzhhNzc4Y2Y3YzhkOCIsImRlYnVnX2RhdGFfcGFja2FnZSI6ZmFsc2UsImV4ZWN1dGlvbl90aW1lb3V0X3NlY29uZHMiOjMwMCwiZmlsbF9taXNzaW5nX3RhZ3MiOmZhbHNlLCJmaWx0ZXJzIjp7fSwiZnJlcXVlbmN5IjoiMW0iLCJtYXhfcmV0cnlfcG9saWN5IjoxLCJtb2RlbF9jb25maWciOnt9LCJtb2RlbF9pZCI6Im1vZGVsLW9wYyIsIm1vZGVsX25hbWUiOiJNb2RlbCBtb2RlbC1vcGMiLCJtb2RlbF90YWdzIjp7IlByZXNzdXJlIjp7ImFnZ3JfZnVuYyI6ImF2ZyIsImRhdGFfcmFuZ2UiOlstMTAwLDEwMF19fSwib2Zmc2V0IjoiMG0iLCJvbl9jb25mbGljdCI6ImVycm9yIiwicmV0ZW50aW9uX3RpbWUiOjM2MDAsInJ1bnRpbWUiOiJsZWdhY3kiLCJzY2hlZHVsZV9uYW1lIjoib3BjLXRhZ2RiIiwic2NoZW1hIjoic2llbnRpYV9kYXRhIiwidGFibGVfbmFtZSI6ImxhYm9yaW91c19kYXRhIiwidGFza190aW1lb3V0X3NlY29uZHMiOjMwMCwidG9waWMiOiJyYXdfb3BjLXRhZ2RiIiwidHJpZ2dlcl9sYWJvcmlvdXMiOmZhbHNlLCJ1cGRhdGVkX2F0IjoiMjAyNi0xMC0yNyAxMzoxMzoxMi4zOTEwMDArMDAwMCIsIndvcmtmbG93X3R5cGUiOiJzY291dGVyIn19fX0="
+ }
+ ]
+ },
+ "scheduleToCloseTimeout": "0s",
+ "scheduleToStartTimeout": "0s",
+ "startToCloseTimeout": "60s",
+ "heartbeatTimeout": "0s",
+ "workflowTaskCompletedEventId": "27",
+ "retryPolicy": {
+ "initialInterval": "1s",
+ "backoffCoefficient": 2.0,
+ "maximumInterval": "60s",
+ "maximumAttempts": 1
+ },
+ "useWorkflowBuildId": true,
+ "priority": {}
+ }
+ },
+ {
+ "eventId": "32",
+ "eventTime": "2026-10-29T13:13:12.751078821Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
+ "taskId": "1048647",
+ "activityTaskScheduledEventAttributes": {
+ "activityId": "18",
+ "activityType": {
+ "name": "update_schedules"
+ },
+ "taskQueue": {
+ "name": "orchestrator-test-queue",
+ "kind": "TASK_QUEUE_KIND_NORMAL"
+ },
+ "header": {},
+ "input": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJtZXRhZGF0YSI6eyJtb2RlbF9pZCI6Ii0iLCJtb2RlbF9uYW1lIjoiLSIsInNjaGVkdWxlX25hbWUiOiJvcmNoZXN0cmF0b3ItZTJlLXRhZy1kYXRhYmFzZSIsIndvcmtmbG93X25hbWUiOiJvcmNoZXN0cmF0b3IifSwic2NoZWR1bGVzIjp7ImxhYm9yaW91cyI6e30sInNjb3V0ZXIiOnt9fX0="
+ }
+ ]
+ },
+ "scheduleToCloseTimeout": "0s",
+ "scheduleToStartTimeout": "0s",
+ "startToCloseTimeout": "60s",
+ "heartbeatTimeout": "0s",
+ "workflowTaskCompletedEventId": "27",
+ "retryPolicy": {
+ "initialInterval": "1s",
+ "backoffCoefficient": 2.0,
+ "maximumInterval": "60s",
+ "maximumAttempts": 1
+ },
+ "useWorkflowBuildId": true,
+ "priority": {}
+ }
+ },
+ {
+ "eventId": "33",
+ "eventTime": "2026-10-29T13:13:12.751119658Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
+ "taskId": "1048656",
+ "activityTaskStartedEventAttributes": {
+ "scheduledEventId": "28",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "33ab8a43-ec09-4464-b50f-ce84378f87ff",
+ "attempt": 1,
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "34",
+ "eventTime": "2026-10-29T13:13:12.764121779Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
+ "taskId": "1048657",
+ "activityTaskCompletedEventAttributes": {
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "e30="
+ }
+ ]
+ },
+ "scheduledEventId": "28",
+ "startedEventId": "33",
+ "identity": "69421@grezewave-HP-ZBook-17-G5"
+ }
+ },
+ {
+ "eventId": "35",
+ "eventTime": "2026-10-29T13:13:12.764184929Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
+ "taskId": "1048658",
+ "workflowTaskScheduledEventAttributes": {
+ "taskQueue": {
+ "name": "69421@grezewave-HP-ZBook-17-G5-6d452a12d16c4ff7b47360d8b9ac5b8e",
+ "kind": "TASK_QUEUE_KIND_STICKY",
+ "normalName": "orchestrator-test-queue"
+ },
+ "startToCloseTimeout": "10s",
+ "attempt": 1
+ }
+ },
+ {
+ "eventId": "36",
+ "eventTime": "2026-10-29T13:13:12.751156959Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
+ "taskId": "1048663",
+ "activityTaskStartedEventAttributes": {
+ "scheduledEventId": "30",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "7495f701-9a71-4e44-b4ff-17b529b2f7cd",
+ "attempt": 1,
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "37",
+ "eventTime": "2026-10-29T13:13:12.768596952Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
+ "taskId": "1048664",
+ "activityTaskCompletedEventAttributes": {
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "W10="
+ }
+ ]
+ },
+ "scheduledEventId": "30",
+ "startedEventId": "36",
+ "identity": "69421@grezewave-HP-ZBook-17-G5"
+ }
+ },
+ {
+ "eventId": "38",
+ "eventTime": "2026-10-29T13:13:12.751144516Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
+ "taskId": "1048666",
+ "activityTaskStartedEventAttributes": {
+ "scheduledEventId": "29",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "983ccf51-0537-447c-b1e5-4428c8b5f78b",
+ "attempt": 1,
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "39",
+ "eventTime": "2026-10-29T13:13:12.772076522Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
+ "taskId": "1048667",
+ "activityTaskCompletedEventAttributes": {
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyIxIjp7Im1lc3NhZ2UiOiJTbG90IHVwZGF0ZWQgc3VjY2Vzc2Z1bGx5Iiwic3VjY2VzcyI6dHJ1ZX19"
+ }
+ ]
+ },
+ "scheduledEventId": "29",
+ "startedEventId": "38",
+ "identity": "69421@grezewave-HP-ZBook-17-G5"
+ }
+ },
+ {
+ "eventId": "40",
+ "eventTime": "2026-10-29T13:13:12.775934616Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
+ "taskId": "1048670",
+ "workflowTaskStartedEventAttributes": {
+ "scheduledEventId": "35",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "c41bb5cb-286f-43c3-8675-faf8803edae2",
+ "historySizeBytes": "17197",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "41",
+ "eventTime": "2026-10-29T13:13:12.797434708Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
+ "taskId": "1048683",
+ "workflowTaskCompletedEventAttributes": {
+ "scheduledEventId": "35",
+ "startedEventId": "40",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ },
+ "sdkMetadata": {},
+ "meteringMetadata": {}
+ }
+ },
+ {
+ "eventId": "42",
+ "eventTime": "2026-10-29T13:13:12.761503422Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
+ "taskId": "1048684",
+ "activityTaskStartedEventAttributes": {
+ "scheduledEventId": "32",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "0a66e272-f244-4ab3-ad7c-faacbe134a38",
+ "attempt": 1,
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "43",
+ "eventTime": "2026-10-29T13:13:12.780025765Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
+ "taskId": "1048685",
+ "activityTaskCompletedEventAttributes": {
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "W10="
+ }
+ ]
+ },
+ "scheduledEventId": "32",
+ "startedEventId": "42",
+ "identity": "69421@grezewave-HP-ZBook-17-G5"
+ }
+ },
+ {
+ "eventId": "44",
+ "eventTime": "2026-10-29T13:13:12.797565457Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
+ "taskId": "1048686",
+ "workflowTaskScheduledEventAttributes": {
+ "taskQueue": {
+ "name": "69421@grezewave-HP-ZBook-17-G5-6d452a12d16c4ff7b47360d8b9ac5b8e",
+ "kind": "TASK_QUEUE_KIND_STICKY",
+ "normalName": "orchestrator-test-queue"
+ },
+ "startToCloseTimeout": "10s",
+ "attempt": 1
+ }
+ },
+ {
+ "eventId": "45",
+ "eventTime": "2026-10-29T13:13:12.797584648Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
+ "taskId": "1048687",
+ "workflowTaskStartedEventAttributes": {
+ "scheduledEventId": "44",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "request-from-RespondWorkflowTaskCompleted",
+ "historySizeBytes": "17336",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "46",
+ "eventTime": "2026-10-29T13:13:12.810959189Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
+ "taskId": "1048693",
+ "workflowTaskCompletedEventAttributes": {
+ "scheduledEventId": "44",
+ "startedEventId": "45",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ },
+ "sdkMetadata": {},
+ "meteringMetadata": {}
+ }
+ },
+ {
+ "eventId": "47",
+ "eventTime": "2026-10-29T13:13:12.758738065Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
+ "taskId": "1048712",
+ "activityTaskStartedEventAttributes": {
+ "scheduledEventId": "31",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "6a390f2e-4583-4a16-aba6-f351ba3b6296",
+ "attempt": 1,
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "48",
+ "eventTime": "2026-10-29T13:13:12.829505616Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
+ "taskId": "1048713",
+ "activityTaskCompletedEventAttributes": {
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "W3siY29uZmlnX2hhc2giOiI3NDVlMzhiYWE5MmRmNTU4MWY0YmY4YjNmMDhmYTdmNTM4NGNmZDkyMGE2YTg4MGMzZjE1MDM4Njg0NTQzYTY3IiwibWVzc2FnZSI6IlNjaGVkdWxlIGNyZWF0ZWQgc3VjY2Vzc2Z1bGx5IiwibmFtZXNwYWNlIjoibGFib3Jpb3VzIiwic2NoZWR1bGVfbmFtZSI6InByZWQtdGFnZGIiLCJzdWNjZXNzIjp0cnVlfSx7ImNvbmZpZ19oYXNoIjoiMTNkZTdhYzA5ZDQzMWEzNTE0NWJhODNmZWVjNDcwZDczOTA5ZmY2YzUyZmU1YjE4ZDNjYzhhNzc4Y2Y3YzhkOCIsIm1lc3NhZ2UiOiJTY2hlZHVsZSBjcmVhdGVkIHN1Y2Nlc3NmdWxseSIsIm5hbWVzcGFjZSI6InNjb3V0ZXIiLCJzY2hlZHVsZV9uYW1lIjoib3BjLXRhZ2RiIiwic3VjY2VzcyI6dHJ1ZX1d"
+ }
+ ]
+ },
+ "scheduledEventId": "31",
+ "startedEventId": "47",
+ "identity": "69421@grezewave-HP-ZBook-17-G5"
+ }
+ },
+ {
+ "eventId": "49",
+ "eventTime": "2026-10-29T13:13:12.829543912Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
+ "taskId": "1048714",
+ "workflowTaskScheduledEventAttributes": {
+ "taskQueue": {
+ "name": "69421@grezewave-HP-ZBook-17-G5-6d452a12d16c4ff7b47360d8b9ac5b8e",
+ "kind": "TASK_QUEUE_KIND_STICKY",
+ "normalName": "orchestrator-test-queue"
+ },
+ "startToCloseTimeout": "10s",
+ "attempt": 1
+ }
+ },
+ {
+ "eventId": "50",
+ "eventTime": "2026-10-29T13:13:12.849228832Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
+ "taskId": "1048721",
+ "workflowTaskStartedEventAttributes": {
+ "scheduledEventId": "49",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "e9631f45-027f-4ce8-8473-3d83e87e4c7e",
+ "historySizeBytes": "18789",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "51",
+ "eventTime": "2026-10-29T13:13:12.871586052Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
+ "taskId": "1048737",
+ "workflowTaskCompletedEventAttributes": {
+ "scheduledEventId": "49",
+ "startedEventId": "50",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ },
+ "sdkMetadata": {},
+ "meteringMetadata": {}
+ }
+ },
+ {
+ "eventId": "52",
+ "eventTime": "2026-10-29T13:13:12.871676710Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
+ "taskId": "1048738",
+ "activityTaskScheduledEventAttributes": {
+ "activityId": "19",
+ "activityType": {
+ "name": "report_schedule_orchestration"
+ },
+ "taskQueue": {
+ "name": "orchestrator-test-queue",
+ "kind": "TASK_QUEUE_KIND_NORMAL"
+ },
+ "header": {},
+ "input": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJjcmVhdGVkX3NjaGVkdWxlcyI6W3siY29uZmlnX2hhc2giOiI3NDVlMzhiYWE5MmRmNTU4MWY0YmY4YjNmMDhmYTdmNTM4NGNmZDkyMGE2YTg4MGMzZjE1MDM4Njg0NTQzYTY3IiwibWVzc2FnZSI6IlNjaGVkdWxlIGNyZWF0ZWQgc3VjY2Vzc2Z1bGx5IiwibmFtZXNwYWNlIjoibGFib3Jpb3VzIiwic2NoZWR1bGVfbmFtZSI6InByZWQtdGFnZGIiLCJzdWNjZXNzIjp0cnVlfSx7ImNvbmZpZ19oYXNoIjoiMTNkZTdhYzA5ZDQzMWEzNTE0NWJhODNmZWVjNDcwZDczOTA5ZmY2YzUyZmU1YjE4ZDNjYzhhNzc4Y2Y3YzhkOCIsIm1lc3NhZ2UiOiJTY2hlZHVsZSBjcmVhdGVkIHN1Y2Nlc3NmdWxseSIsIm5hbWVzcGFjZSI6InNjb3V0ZXIiLCJzY2hlZHVsZV9uYW1lIjoib3BjLXRhZ2RiIiwic3VjY2VzcyI6dHJ1ZX1dLCJkZWxldGVkX3NjaGVkdWxlcyI6W10sIm1ldGFkYXRhIjp7Im1vZGVsX2lkIjoiLSIsIm1vZGVsX25hbWUiOiItIiwic2NoZWR1bGVfbmFtZSI6Im9yY2hlc3RyYXRvci1lMmUtdGFnLWRhdGFiYXNlIiwid29ya2Zsb3dfbmFtZSI6Im9yY2hlc3RyYXRvciJ9LCJ1cGRhdGVkX3NjaGVkdWxlcyI6W119"
+ }
+ ]
+ },
+ "scheduleToCloseTimeout": "0s",
+ "scheduleToStartTimeout": "0s",
+ "startToCloseTimeout": "60s",
+ "heartbeatTimeout": "0s",
+ "workflowTaskCompletedEventId": "51",
+ "retryPolicy": {
+ "initialInterval": "1s",
+ "backoffCoefficient": 2.0,
+ "maximumInterval": "60s",
+ "maximumAttempts": 1
+ },
+ "useWorkflowBuildId": true,
+ "priority": {}
+ }
+ },
+ {
+ "eventId": "53",
+ "eventTime": "2026-10-29T13:13:12.871762444Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
+ "taskId": "1048739",
+ "activityTaskScheduledEventAttributes": {
+ "activityId": "20",
+ "activityType": {
+ "name": "report_slot_orchestration"
+ },
+ "taskQueue": {
+ "name": "orchestrator-test-queue",
+ "kind": "TASK_QUEUE_KIND_NORMAL"
+ },
+ "header": {},
+ "input": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJkZWxldGVkX3Nsb3RzIjp7fSwiaW5zZXJ0ZWRfc2xvdHMiOnsiMSI6eyJtZXNzYWdlIjoiU2xvdCB1cGRhdGVkIHN1Y2Nlc3NmdWxseSIsInN1Y2Nlc3MiOnRydWV9fSwibWV0YWRhdGEiOnsibW9kZWxfaWQiOiItIiwibW9kZWxfbmFtZSI6Ii0iLCJzY2hlZHVsZV9uYW1lIjoib3JjaGVzdHJhdG9yLWUyZS10YWctZGF0YWJhc2UiLCJ3b3JrZmxvd19uYW1lIjoib3JjaGVzdHJhdG9yIn19"
+ }
+ ]
+ },
+ "scheduleToCloseTimeout": "0s",
+ "scheduleToStartTimeout": "0s",
+ "startToCloseTimeout": "60s",
+ "heartbeatTimeout": "0s",
+ "workflowTaskCompletedEventId": "51",
+ "retryPolicy": {
+ "initialInterval": "1s",
+ "backoffCoefficient": 2.0,
+ "maximumInterval": "60s",
+ "maximumAttempts": 1
+ },
+ "useWorkflowBuildId": true,
+ "priority": {}
+ }
+ },
+ {
+ "eventId": "54",
+ "eventTime": "2026-10-29T13:13:12.871812472Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
+ "taskId": "1048740",
+ "activityTaskScheduledEventAttributes": {
+ "activityId": "21",
+ "activityType": {
+ "name": "create_pipelines_timestamps"
+ },
+ "taskQueue": {
+ "name": "orchestrator-test-queue",
+ "kind": "TASK_QUEUE_KIND_NORMAL"
+ },
+ "header": {},
+ "input": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "anNvbi9wbGFpbg=="
+ },
+ "data": "eyJjcmVhdGVkX3BpcGVsaW5lcyI6W3siY29uZmlnX2hhc2giOiI3NDVlMzhiYWE5MmRmNTU4MWY0YmY4YjNmMDhmYTdmNTM4NGNmZDkyMGE2YTg4MGMzZjE1MDM4Njg0NTQzYTY3IiwibWVzc2FnZSI6IlNjaGVkdWxlIGNyZWF0ZWQgc3VjY2Vzc2Z1bGx5IiwibmFtZXNwYWNlIjoibGFib3Jpb3VzIiwic2NoZWR1bGVfbmFtZSI6InByZWQtdGFnZGIiLCJzdWNjZXNzIjp0cnVlfSx7ImNvbmZpZ19oYXNoIjoiMTNkZTdhYzA5ZDQzMWEzNTE0NWJhODNmZWVjNDcwZDczOTA5ZmY2YzUyZmU1YjE4ZDNjYzhhNzc4Y2Y3YzhkOCIsIm1lc3NhZ2UiOiJTY2hlZHVsZSBjcmVhdGVkIHN1Y2Nlc3NmdWxseSIsIm5hbWVzcGFjZSI6InNjb3V0ZXIiLCJzY2hlZHVsZV9uYW1lIjoib3BjLXRhZ2RiIiwic3VjY2VzcyI6dHJ1ZX1dLCJtZXRhZGF0YSI6eyJtb2RlbF9pZCI6Ii0iLCJtb2RlbF9uYW1lIjoiLSIsInNjaGVkdWxlX25hbWUiOiJvcmNoZXN0cmF0b3ItZTJlLXRhZy1kYXRhYmFzZSIsIndvcmtmbG93X25hbWUiOiJvcmNoZXN0cmF0b3IifX0="
+ }
+ ]
+ },
+ "scheduleToCloseTimeout": "0s",
+ "scheduleToStartTimeout": "0s",
+ "startToCloseTimeout": "60s",
+ "heartbeatTimeout": "0s",
+ "workflowTaskCompletedEventId": "51",
+ "retryPolicy": {
+ "initialInterval": "1s",
+ "backoffCoefficient": 2.0,
+ "maximumInterval": "60s",
+ "maximumAttempts": 1
+ },
+ "useWorkflowBuildId": true,
+ "priority": {}
+ }
+ },
+ {
+ "eventId": "55",
+ "eventTime": "2026-10-29T13:13:12.871846968Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
+ "taskId": "1048743",
+ "activityTaskStartedEventAttributes": {
+ "scheduledEventId": "52",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "a5d4cb77-1202-4005-9375-a6635e735ba0",
+ "attempt": 1,
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "56",
+ "eventTime": "2026-10-29T13:13:12.892333210Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
+ "taskId": "1048744",
+ "activityTaskCompletedEventAttributes": {
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "YmluYXJ5L251bGw="
+ }
+ }
+ ]
+ },
+ "scheduledEventId": "52",
+ "startedEventId": "55",
+ "identity": "69421@grezewave-HP-ZBook-17-G5"
+ }
+ },
+ {
+ "eventId": "57",
+ "eventTime": "2026-10-29T13:13:12.892372531Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
+ "taskId": "1048745",
+ "workflowTaskScheduledEventAttributes": {
+ "taskQueue": {
+ "name": "69421@grezewave-HP-ZBook-17-G5-6d452a12d16c4ff7b47360d8b9ac5b8e",
+ "kind": "TASK_QUEUE_KIND_STICKY",
+ "normalName": "orchestrator-test-queue"
+ },
+ "startToCloseTimeout": "10s",
+ "attempt": 1
+ }
+ },
+ {
+ "eventId": "58",
+ "eventTime": "2026-10-29T13:13:12.897584855Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
+ "taskId": "1048750",
+ "workflowTaskStartedEventAttributes": {
+ "scheduledEventId": "57",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "a358def5-8ae5-4a28-a7f1-2ab33529a3fe",
+ "historySizeBytes": "21206",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "59",
+ "eventTime": "2026-10-29T13:13:12.907954989Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
+ "taskId": "1048756",
+ "workflowTaskCompletedEventAttributes": {
+ "scheduledEventId": "57",
+ "startedEventId": "58",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ },
+ "sdkMetadata": {},
+ "meteringMetadata": {}
+ }
+ },
+ {
+ "eventId": "60",
+ "eventTime": "2026-10-29T13:13:12.871865177Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
+ "taskId": "1048757",
+ "activityTaskStartedEventAttributes": {
+ "scheduledEventId": "53",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "b63fdeda-da38-4d22-909c-ce4feb5e2a31",
+ "attempt": 1,
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "61",
+ "eventTime": "2026-10-29T13:13:12.900797555Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
+ "taskId": "1048758",
+ "activityTaskCompletedEventAttributes": {
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "YmluYXJ5L251bGw="
+ }
+ }
+ ]
+ },
+ "scheduledEventId": "53",
+ "startedEventId": "60",
+ "identity": "69421@grezewave-HP-ZBook-17-G5"
+ }
+ },
+ {
+ "eventId": "62",
+ "eventTime": "2026-10-29T13:13:12.908030764Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
+ "taskId": "1048759",
+ "workflowTaskScheduledEventAttributes": {
+ "taskQueue": {
+ "name": "69421@grezewave-HP-ZBook-17-G5-6d452a12d16c4ff7b47360d8b9ac5b8e",
+ "kind": "TASK_QUEUE_KIND_STICKY",
+ "normalName": "orchestrator-test-queue"
+ },
+ "startToCloseTimeout": "10s",
+ "attempt": 1
+ }
+ },
+ {
+ "eventId": "63",
+ "eventTime": "2026-10-29T13:13:12.908040957Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
+ "taskId": "1048760",
+ "workflowTaskStartedEventAttributes": {
+ "scheduledEventId": "62",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "request-from-RespondWorkflowTaskCompleted",
+ "historySizeBytes": "21345",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "64",
+ "eventTime": "2026-10-29T13:13:12.916225852Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
+ "taskId": "1048763",
+ "workflowTaskCompletedEventAttributes": {
+ "scheduledEventId": "62",
+ "startedEventId": "63",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ },
+ "sdkMetadata": {},
+ "meteringMetadata": {}
+ }
+ },
+ {
+ "eventId": "65",
+ "eventTime": "2026-10-29T13:13:12.871875343Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
+ "taskId": "1048765",
+ "activityTaskStartedEventAttributes": {
+ "scheduledEventId": "54",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "2b0899d9-a701-4062-95e2-9ebe8a6a1c52",
+ "attempt": 1,
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "66",
+ "eventTime": "2026-10-29T13:13:12.926242860Z",
+ "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
+ "taskId": "1048766",
+ "activityTaskCompletedEventAttributes": {
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "YmluYXJ5L251bGw="
+ }
+ }
+ ]
+ },
+ "scheduledEventId": "54",
+ "startedEventId": "65",
+ "identity": "69421@grezewave-HP-ZBook-17-G5"
+ }
+ },
+ {
+ "eventId": "67",
+ "eventTime": "2026-10-29T13:13:12.926277874Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
+ "taskId": "1048767",
+ "workflowTaskScheduledEventAttributes": {
+ "taskQueue": {
+ "name": "69421@grezewave-HP-ZBook-17-G5-6d452a12d16c4ff7b47360d8b9ac5b8e",
+ "kind": "TASK_QUEUE_KIND_STICKY",
+ "normalName": "orchestrator-test-queue"
+ },
+ "startToCloseTimeout": "10s",
+ "attempt": 1
+ }
+ },
+ {
+ "eventId": "68",
+ "eventTime": "2026-10-29T13:13:12.930711125Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
+ "taskId": "1048771",
+ "workflowTaskStartedEventAttributes": {
+ "scheduledEventId": "67",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "requestId": "1db5faf4-98df-406e-9984-a35af7e25e39",
+ "historySizeBytes": "22405",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ }
+ }
+ },
+ {
+ "eventId": "69",
+ "eventTime": "2026-10-29T13:13:12.941227851Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
+ "taskId": "1048775",
+ "workflowTaskCompletedEventAttributes": {
+ "scheduledEventId": "67",
+ "startedEventId": "68",
+ "identity": "69421@grezewave-HP-ZBook-17-G5",
+ "workerVersion": {
+ "buildId": "04de6f956d3ccd62d792de151ac59867"
+ },
+ "sdkMetadata": {},
+ "meteringMetadata": {}
+ }
+ },
+ {
+ "eventId": "70",
+ "eventTime": "2026-10-29T13:13:12.941400619Z",
+ "eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_COMPLETED",
+ "taskId": "1048776",
+ "workflowExecutionCompletedEventAttributes": {
+ "result": {
+ "payloads": [
+ {
+ "metadata": {
+ "encoding": "YmluYXJ5L251bGw="
+ }
+ }
+ ]
+ },
+ "workflowTaskCompletedEventId": "69"
+ }
+ }
+ ]
+}
\ No newline at end of file
diff --git a/tests/orchestrator/workflows/test_orchestrator.py b/tests/orchestrator/workflows/test_orchestrator.py
index 61243b0..b0664ab 100644
--- a/tests/orchestrator/workflows/test_orchestrator.py
+++ b/tests/orchestrator/workflows/test_orchestrator.py
@@ -3,7 +3,11 @@ from unittest.mock import ANY, AsyncMock, call, patch
from pytest import fixture, mark
from orchestrator.activities.activities import Activities
-from orchestrator.workflows.orchestrator import Orchestrator
+from orchestrator.workflows.orchestrator import (
+ DEFAULT_CONNECTOR_TAGS_QUERY,
+ DEFAULT_CONNECTORS_QUERY,
+ Orchestrator,
+)
@fixture
@@ -22,11 +26,11 @@ metadata = {
@mark.asyncio
+@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[7, '7'])
@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
-async def test_run(workflow_mock, orchestrator):
+async def test_run(workflow_mock, referenced_tag_ids_mock, orchestrator):
input_data = {
'pipelines_query': 'SELECT * FROM bucket',
- 'opc_servers_query': 'SELECT * FROM servers',
'schedule_name': 'test-schedule-name',
}
@@ -47,16 +51,13 @@ async def test_run(workflow_mock, orchestrator):
]
)
- workflow_mock.start_local_activity_method.assert_has_calls(
- [
- call(
- Activities.find_documents_in_mongodb,
- {**metadata, 'query': input_data['opc_servers_query']},
- retry_policy=ANY,
- start_to_close_timeout=ANY,
- )
- ]
- )
+ # connectors is the only source of the OPC servers: opc-servers is never loaded
+ assert not [
+ c
+ for c in workflow_mock.start_local_activity_method.call_args_list
+ if c.args[0] is Activities.find_documents_in_mongodb
+ and c.args[1]['query'] != {'collection': 'orchestrated_schedules'}
+ ]
workflow_mock.start_local_activity_method.assert_has_calls(
[
@@ -109,11 +110,54 @@ async def test_run(workflow_mock, orchestrator):
]
)
+ referenced_tag_ids_mock.assert_called_once_with(
+ workflow_mock.start_local_activity_method.return_value
+ )
+
+ workflow_mock.start_local_activity_method.assert_has_calls(
+ [
+ call(
+ Activities.aggregate_documents_in_mongodb,
+ {**metadata, 'query': DEFAULT_CONNECTORS_QUERY},
+ retry_policy=ANY,
+ start_to_close_timeout=ANY,
+ ),
+ call(
+ Activities.aggregate_documents_in_mongodb,
+ {
+ **metadata,
+ 'query': {
+ **DEFAULT_CONNECTOR_TAGS_QUERY,
+ 'aggregation': [
+ {'$match': {'tag_id': {'$in': [7, '7']}}},
+ *DEFAULT_CONNECTOR_TAGS_QUERY['aggregation'],
+ ],
+ },
+ },
+ retry_policy=ANY,
+ start_to_close_timeout=ANY,
+ ),
+ ]
+ )
+
+ # The activity appends a stage to the aggregation it gets: never the module default
+ tag_database_calls = _tag_database_calls(workflow_mock)
+ assert len(tag_database_calls) == 2
+ for c in tag_database_calls:
+ aggregation = c.args[1]['query']['aggregation']
+ assert aggregation is not DEFAULT_CONNECTORS_QUERY['aggregation']
+ assert aggregation is not DEFAULT_CONNECTOR_TAGS_QUERY['aggregation']
+
workflow_mock.start_local_activity_method.assert_has_calls(
[
call(
Activities.process_schedules,
- {**metadata, 'pipelines': workflow_mock.start_local_activity_method.return_value},
+ {
+ **metadata,
+ 'pipelines': workflow_mock.start_local_activity_method.return_value,
+ 'connectors': workflow_mock.start_local_activity_method.return_value,
+ 'connector_tags': workflow_mock.start_local_activity_method.return_value,
+ },
retry_policy=ANY,
start_to_close_timeout=ANY,
)
@@ -126,7 +170,8 @@ async def test_run(workflow_mock, orchestrator):
Activities.process_slots,
{
**metadata,
- 'opc_servers': workflow_mock.start_local_activity_method.return_value,
+ 'connectors': workflow_mock.start_local_activity_method.return_value,
+ 'connector_tags': workflow_mock.start_local_activity_method.return_value,
'active_ingestors': workflow_mock.start_local_activity_method.return_value,
'pipelines': workflow_mock.start_local_activity_method.return_value,
},
@@ -346,3 +391,136 @@ async def test_run(workflow_mock, orchestrator):
)
]
)
+
+
+def _tag_database_calls(workflow_mock):
+ return [
+ c
+ for c in workflow_mock.start_local_activity_method.call_args_list
+ if c.args[0] is Activities.aggregate_documents_in_mongodb
+ and isinstance(c.args[1]['query'], dict)
+ and c.args[1]['query'].get('collection') in ('connectors', 'connector_tags')
+ ]
+
+
+def _process_slots_input(workflow_mock):
+ return next(
+ c.args[1]
+ for c in workflow_mock.start_local_activity_method.call_args_list
+ if c.args[0] is Activities.process_slots
+ )
+
+
+@mark.asyncio
+@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[7])
+@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
+async def test_run_with_tag_database_queries_from_input(workflow_mock, _ids, orchestrator):
+ """The input overrides the default tag database aggregations (SIENTIAPDE-1974, D9)"""
+ connectors_query = {'collection': 'connectors', 'aggregation': [{'$match': {'type': 'opc_ua'}}]}
+ connector_tags_query = {
+ 'collection': 'connector_tags',
+ 'aggregation': [{'$match': {'category': 'Read'}}],
+ }
+ input_data = {
+ 'pipelines_query': 'SELECT * FROM bucket',
+ 'connectors_query': connectors_query,
+ 'connector_tags_query': connector_tags_query,
+ 'schedule_name': 'test-schedule-name',
+ }
+
+ await orchestrator.run(input_data)
+
+ queries = [c.args[1]['query'] for c in _tag_database_calls(workflow_mock)]
+ assert queries == [
+ connectors_query,
+ {
+ 'collection': 'connector_tags',
+ 'aggregation': [
+ {'$match': {'tag_id': {'$in': [7]}}},
+ {'$match': {'category': 'Read'}},
+ ],
+ },
+ ]
+
+
+@mark.asyncio
+@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[])
+@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
+async def test_run_without_tag_ids_skips_tag_database(workflow_mock, _ids, orchestrator):
+ """No read tag with tag_id: the tag database is not queried at all (P4)"""
+ input_data = {
+ 'pipelines_query': 'SELECT * FROM bucket',
+ 'schedule_name': 'test-schedule-name',
+ }
+
+ await orchestrator.run(input_data)
+
+ assert _tag_database_calls(workflow_mock) == []
+ slots_input = _process_slots_input(workflow_mock)
+ assert slots_input['connectors'] == []
+ assert slots_input['connector_tags'] == []
+
+
+def _process_schedules_input(workflow_mock):
+ return next(
+ c.args[1]
+ for c in workflow_mock.start_local_activity_method.call_args_list
+ if c.args[0] is Activities.process_schedules
+ )
+
+
+def test_default_connector_tags_query_loads_every_category_and_status():
+ """
+ The status / category filters are applied in the code. updated_at is not loaded: a
+ tag change reaches the schedules through their config_hash.
+ """
+ stages = DEFAULT_CONNECTOR_TAGS_QUERY['aggregation']
+ assert all('$match' not in stage for stage in stages)
+ projected = stages[-1]['$project']
+ for field in ('category', 'data_type', 'status'):
+ assert projected[field] == 1
+ assert 'updated_at' not in projected
+
+
+@mark.asyncio
+@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[77])
+@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
+async def test_run_hands_the_tag_database_to_process_schedules(workflow_mock, _ids, orchestrator):
+ """The write tags of process_schedules need the connectors as well as the tags"""
+ input_data = {
+ 'pipelines_query': 'SELECT * FROM bucket',
+ 'schedule_name': 'test-schedule-name',
+ }
+
+ await orchestrator.run(input_data)
+
+ loaded = workflow_mock.start_local_activity_method.return_value
+ schedules_input = _process_schedules_input(workflow_mock)
+ assert schedules_input['connectors'] is loaded
+ assert schedules_input['connector_tags'] is loaded
+ assert 'pi_connectors' not in schedules_input
+
+
+@mark.asyncio
+@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[])
+@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
+async def test_run_without_changes_skips_reports_and_timestamps(workflow_mock, _ids, orchestrator):
+ """Nothing created, updated or deleted: no report and no pipeline timestamp activity"""
+ workflow_mock.start_activity_method = AsyncMock(return_value=[])
+ input_data = {
+ 'pipelines_query': 'SELECT * FROM bucket',
+ 'schedule_name': 'test-schedule-name',
+ }
+
+ await orchestrator.run(input_data)
+
+ started = [c.args[0] for c in workflow_mock.start_activity_method.call_args_list]
+ assert started == [
+ Activities.normalize_schedules,
+ Activities.create_collection_with_ttl_index,
+ Activities.delete_slots,
+ Activities.update_slots,
+ Activities.delete_schedules,
+ Activities.create_schedules,
+ Activities.update_schedules,
+ ]
diff --git a/tests/orchestrator/workflows/test_orchestrator_replay.py b/tests/orchestrator/workflows/test_orchestrator_replay.py
new file mode 100644
index 0000000..6f6ac11
--- /dev/null
+++ b/tests/orchestrator/workflows/test_orchestrator_replay.py
@@ -0,0 +1,27 @@
+"""
+Replay a real Orchestrator history against the current workflow code.
+
+orchestrator_tag_database.json was recorded on the e2e stack (real Temporal, Mongo and
+Redis), with the connectors / connector_tags loads of read and write tags by tag_id.
+
+A NondeterminismError here means a change to the workflow breaks runs in flight.
+"""
+
+from pathlib import Path
+
+import pytest
+from temporalio.client import WorkflowHistory
+from temporalio.worker import Replayer
+
+from orchestrator.workflows.orchestrator import Orchestrator
+
+HISTORIES = Path(__file__).parent / 'histories'
+
+
+@pytest.mark.asyncio
+async def test_replay_recorded_history():
+ history = WorkflowHistory.from_json(
+ 'orchestrator-history', (HISTORIES / 'orchestrator_tag_database.json').read_text()
+ )
+
+ await Replayer(workflows=[Orchestrator]).replay_workflow(history)