chore: bump version to 0.9.0

SIENTIAPDE-1974: OPC tag database for read and write tags (slot v2, write_tags by tag_id)

.github/workflows/deploy.yml

@@ -1,10 +1,15 @@
 name: Deploy DataOps Worker
 
 on:
-  pull_request:
-    types: [closed]
-    branches: [main, 'release/**', 'feature/**']
   workflow_dispatch:
+    inputs:
+      chart_ref:
+        description: >
+          sientia-helm-chart branch to bump GITHUB_BRANCH on AND deploy — the literal
+          deploy target. Validated via git ls-remote; a branch that does not exist in
+          sientia-helm-chart is a hard error, not a silent fallback to main.
+        required: true
+        type: string
 
 permissions:
   contents: write
@@ -14,10 +19,10 @@ permissions:
 jobs:
   deploy:
     name: Deploy
-    if: github.event_name == 'workflow_dispatch' || github.event.pull_request.merged == true
     uses: Aignosi/github_workflow_templates/.github/workflows/reusable-deploy-dataops.yml@main
     with:
-      branch_name: ${{ github.event.pull_request.head.ref }}
+      branch_name: ${{ github.ref_name }}
+      chart_ref: ${{ inputs.chart_ref }}
       project_name: 'orchestrator'
       chart_name: 'orchestrator-worker'
       env_list_path: '[".env"]'

.github/workflows/quality-gate.yml

@@ -5,7 +5,6 @@ on:
     branches:
       - main
       - 'release/**'
-      - 'feature/**'
   pull_request:
     branches:
       - main

.github/workflows/require-chart-ref-label.yml

@@ -1,16 +0,0 @@
-name: Require chart-ref label
-
-on:
-  pull_request:
-    types: [opened, synchronize, reopened, labeled, unlabeled]
-    branches:
-      - main
-      - 'release/**'
-      - 'feature/**'
-
-permissions:
-  pull-requests: write
-
-jobs:
-  require-chart-ref-label:
-    uses: Aignosi/github_workflow_templates/.github/workflows/require-chart-ref-label.yml@main

.github/workflows/version-bump.yml

@@ -0,0 +1,17 @@
+name: Version Bump
+
+permissions:
+  contents: write
+
+on:
+  pull_request:
+    types: [closed]
+    branches: [main, 'release/**', 'feature/**']
+
+jobs:
+  bump:
+    if: github.event.pull_request.merged == true
+    uses: Aignosi/github_workflow_templates/.github/workflows/reusable-version-bump.yml@main
+    with:
+      update_version_in: '["pyproject"]'
+    secrets: inherit

.gitignore

@@ -49,4 +49,5 @@ git_log
 .env
 
 openspec/
-.cursor/
+.cursor/
+.claude/

CLAUDE.md

@@ -47,7 +47,8 @@ CI quality gate (`.github/workflows/quality-gate.yml`) calls the shared reusable
 - **`orchestrator/worker/worker.py`** — entrypoint (`python -m orchestrator.worker.worker`). Builds a single `Activities` instance, connects to Temporal, starts Prometheus (`HTTP_METRICS_PORT`) and Temporal SDK metrics (`HTTP_SDK_METRICS_PORT`) servers, then runs three `Worker`s concurrently via `asyncio.gather`, each on its own task queue: `orchestrator-queue`, `alerts-queue`, `reports-queue`. `prepare_worker.py` derives the queue name from the workflow class name (`camel_to_snake`) and reads per-queue concurrency/poller tuning from env vars prefixed with the workflow name (e.g. `ORCHESTRATOR_MAX_CONCURRENT_ACTIVITIES`).
 - **`orchestrator/workflows/`** — Temporal workflow definitions: `orchestrator.py` (pipeline/slot/schedule orchestration), `alerts.py` (real-time ERROR notifications), `reports.py` (scheduled all-level reports). `alerts.py` and `reports.py` both delegate to the same two subworkflows in `workflows/subworkflows/`: `load_notification_package.py` (load notifications + receiver configs from Mongo, incremental via Redis timestamp) and `process_notifications.py` (build HTML email, send, log to Postgres). The difference between alerts and reports is which filter activity they call (`filter_notification_alerts` vs `filter_notification_reports`) and their base data filter (`level: ERROR` by default vs all levels) — `load_notification_package.py`'s `_compute_query_time_filter` can widen either beyond that default to the union of active receiver groups' own `levels` (SIENTIAPDE-2011); the ERROR-only default only holds when no active group narrows further. Receiver groups also support a `model_ids` allow-list (SIENTIAPDE-2149), independent of the pre-existing `ignore_models_id` deny-list — both apply together (AND) when a group configures both, and `model_ids` follows the same union-at-query-time / precise-per-group-slice split as `levels`. `load_latest_data`'s Mongo query is capped at `NOTIFICATION_QUEUE_BATCH_SIZE` documents per run (default `0`, meaning unlimited — unchanged from pre-SIENTIAPDE-2013 behavior unless a deployment opts in), sorted by `timestamp` ascending so a capped run always returns the oldest pending batch first (SIENTIAPDE-2013) — a hard cap with no drain loop, since the incremental Redis cursor picks up any remainder on the next run.
 - **`orchestrator/activities/activities.py`** — `Activities` is a single class combining every activity mixin via multiple inheritance: `TemporalManager, SlotManager, Formatters, MongoDB, Email, Postgres` (Postgres comes from `sientia_do.temporal.activities.postgres`, the external `sientia-dataops-library` dependency). One instance is constructed in `worker.py` and its bound methods are handed to each `Worker` as the activity list — activities are plain methods on this shared object, not separate classes. Each mixin's `__init__` takes its own config dict plus shared `logger`, `notification_handler`, `metrics_controller`. `Activities.shutdown()` closes every mixin's connections.
-- **`orchestrator/utils/`** — `connectors_config.py` builds each config dict from env vars (Temporal/Redis/MongoDB/Postgres/Email); `email_builder.py` renders Jinja2 templates (`utils/templates/email_template.html`, `general_template.html`); `orchestrator_functions.py` transforms pipeline documents into pipeline-type-specific configs (`scouter`, `pi_web_api_scouter`, `predictions_batch`, `minimal_retrain`, `drift`, `simple_metrics`) and builds the OPC slot v2 the gateway reads (`gather_read_tags` / `build_tag_config`, contract C1: one entry per address with a `names[]` list, tags resolved through the tag database when they carry a `tag_id`; see the README section "OPC slots"). `resolve_tag_names` is the single source of tag names for both the scouter's `model_tags` and the slot's `names` (SIENTIAPDE-1974 SC1); `converters.py` handles Temporal schedule frequency parsing.
+- **`orchestrator/utils/`** — `connectors_config.py` builds each config dict from env vars (Temporal/Redis/MongoDB/Postgres/Email); `email_builder.py` renders Jinja2 templates (`utils/templates/email_template.html`, `general_template.html`); `pipeline_configs.py` transforms pipeline documents into pipeline-type-specific configs (`scouter`, `pi_web_api_scouter`, `predictions_batch`, `minimal_retrain`, `drift`, `simple_metrics`); `tag_database.py` indexes the tag database (`connectors` + `connector_tags`) and resolves tags by `tag_id` (scouter read tag names and the `predictions_batch` write tags → `opc_output_config` keyed by `connector_id`, via `resolve_write_tags`; tag changes reach the schedules through `config_hash`, see below); `opc_slots.py` builds the OPC slot v2 the gateway reads (`gather_read_tags` / `build_tag_config`, contract C1: one entry per address with a `names[]` list, and per server `security` {mode, policy} plus the `certificate` from `tag_database.resolve_certificate`; see the README section "OPC slots"). The certificate comes from `opc_certificates` through the `$lookup` in `DEFAULT_CONNECTORS_QUERY` (only `active` + `client`); no certificate means the gateway connects without security, and a `client_certificate_id` without one is a `WARNING` (SIENTIAPDE-1974, D5). `resolve_tag_names` (in `tag_database.py`) is the single source of tag names for both the scouter's `model_tags` and the slot's `names` (SIENTIAPDE-1974 SC1); `converters.py` handles Temporal schedule frequency parsing.
+- **Schedule redeploys** — `orchestrated_schedules` stores, per schedule, the last deploy time (`updated_at`) and the `config_hash` it was deployed with (`formatters.schedule_config_hash`: SHA-256 of the built schedule without `updated_at`). `compare_config_timestamps` redeploys when the pipeline's `updated_at` is newer (manual force) or the hash differs (any content change, including tag database edits/deletes made without bumping `updated_at`). `create_schedules`/`update_schedules` pop `config_hash` before sending the schedule to Temporal and return it in their report for `create_pipelines_timestamps`/`update_pipelines_timestamps`.
 
 ### Temporal-specific conventions (all workflow/activity files follow this)
 - Every workflow and activity-consuming module wraps non-deterministic/side-effecting imports in `with workflow.unsafe.imports_passed_through(): ...` right after `from temporalio import workflow`, because Temporal's sandbox restricts what workflow code can import directly. When adding a new workflow file, follow the existing pattern (see `orchestrator/workflows/orchestrator.py`, `orchestrator/activities/activities.py`) rather than importing at module top level.

README.md

@@ -97,13 +97,13 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
 - **Infrastructure Management**: Creates, updates, and deletes workflow schedules
 
 **Execution Flow**:
-1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`)
+1. **Parallel Data Loading**: Concurrently loads pipelines, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`, the only source of the OPC servers and tags of the slot and the schedules)
 2. **Parallel Processing**: Formats orchestrated schedules and processes new schedules and slots
 3. **Parallel Config Creation**: Creates schedule and slot action configurations, normalizes schedules, and creates collections with TTL indexes
 4. **Parallel Operations**: Executes slot deletions, slot updates, schedule deletions, schedule creations, and schedule updates concurrently
 5. **Parallel Reports & Timestamps**: Generates orchestration reports and updates MongoDB timestamps for created/updated/deleted pipelines
 
-**Input Parameters**:
+**Input Parameters** (minimal: each pipeline must already carry `model.name`; see "Workflow Configuration" for the `$lookup` on `models` that adds it):
 ```json
 {
   "schedule_name": "hourly_orchestration",
@@ -113,10 +113,6 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
       {"$match": {"active": true}},
       {"$sort": {"updated_at": -1}}
     ]
-  },
-  "opc_servers_query": {
-    "collection": "opc_servers",
-    "filters": {"active": true}
   }
 }
 ```
@@ -129,31 +125,50 @@ Optional: `connectors_query` and `connector_tags_query` override the tag databas
     "collection": "connectors",
     "aggregation": [
       {"$match": {"type": "opc_ua", "status": "Active"}},
-      {"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "server_uri": 1,
-                    "use_security": 1, "subscription_period_ms": 1}}
+      {"$lookup": {"from": "opc_certificates",
+                   "let": {"certificate_id": "$security.client_certificate_id"},
+                   "pipeline": [
+                     {"$match": {"$expr": {"$eq": ["$certificate_id", "$$certificate_id"]},
+                                 "status": "active", "kind": "client"}},
+                     {"$project": {"_id": 0, "certificate_id": 1, "application_uri": 1,
+                                   "cert.bucket": 1, "cert.object_name": 1,
+                                   "private_key.bucket": 1, "private_key.object_name": 1}}
+                   ],
+                   "as": "certificates"}},
+      {"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "subscription_period_ms": 1,
+                    "security.mode": 1, "security.policy": 1,
+                    "security.client_certificate_id": 1, "certificates": 1}}
     ]
   },
   "connector_tags_query": {
     "collection": "connector_tags",
     "aggregation": [
-      {"$match": {"category": "Read", "status": "Active"}},
-      {"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1}}
+      {"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1,
+                    "category": 1, "data_type": 1, "status": 1}}
     ]
   }
 }
 ```
 
-The tag database is read only when a scouter read tag carries a `tag_id`, and the workflow puts `{"$match": {"tag_id": {"$in": [...]}}}` in front of `connector_tags_query`, so it loads only the referenced tags. Environments without a tag database, or whose pipelines do not use `tag_id`, never query it. An override should keep a `$project` of these fields: any other attribute may hold a BSON type a Temporal payload cannot carry (such as `ObjectId`).
+The workflow puts `{"$match": {"tag_id": {"$in": [...]}}}` in front of `connector_tags_query`, so it loads only the tags the scouter read tags and `predictions_batch` write tags reference. When no pipeline references a `tag_id`, the tag database is not queried. An override should keep a `$project` of these fields: any other attribute may hold a BSON type a Temporal payload cannot carry (such as `ObjectId`).
+
+The referenced tags are loaded whatever their category and status: the code uses only the active ones (`Read` for the slot and the scouter names, `Write` / `Prediction Confidence` for the write tags), and a change to a referenced tag that reaches a schedule redeploys it through its `config_hash` (see "Redeploy on changes" below). A document without `status` counts as active.
 
 #### OPC slots (slot v2, contract C1)
 
-`process_slots` writes one slot per active ingestor to Redis (`slot:opc_tags:{n}`). The OPC Gateway reads it (SIENTIAPDE-1974). Each server carries its connection fields, and each tag address carries the list of names it is written under:
+`process_slots` writes one slot per active ingestor to Redis (`slot:opc_tags:{n}`). The OPC Gateway reads it (SIENTIAPDE-1974). Each server carries its connection fields, its security and the MinIO objects of its client certificate, and each tag address carries the list of names it is written under:
 
 ```json
 {
   "kepserver": {
     "server_id": "1", "name": "kepserver", "url": "opc.tcp://host:49320",
-    "server_uri": "urn:...", "use_security": true, "subscription_period_ms": 7500.0,
+    "subscription_period_ms": 7500.0,
+    "security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256"},
+    "certificate": {
+      "certificate_id": 1, "application_uri": "urn:sientia:opc-ingestor",
+      "cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-cert.pem"},
+      "private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-key.pem"}
+    },
     "tags": {
       "ns=2;s=Channel1.Device1.Tag1": {
         "frequency": 15000,
@@ -168,33 +183,75 @@ The tag database is read only when a scouter read tag carries a `tag_id`, and th
 }
 ```
 
-How each scouter `read_tags` entry is resolved:
+Each scouter `read_tags` entry references its tag by `tag_id` (`{"tag_id": 7, "frequency": 15000}`). `connectors` is the only source of the OPC servers:
 
-| Read tag | Address and server | Name |
-|---|---|---|
-| With `tag_id` | `connector_tags` (`address`, `connector_id`) and `connectors` (`url`, `server_uri`, `use_security`, `subscription_period_ms`) | The read tag's `tag_name`, else the `connector_tags` one |
-| Without `tag_id` | `server_id` + `tag_address`, server from `opc-servers` | The read tag's `tag_name` |
+| Field | Source |
+|---|---|
+| Address | `connector_tags.address` |
+| Server (`server_id`, `name`, `url`, `subscription_period_ms`, `security.mode`, `security.policy`) | The tag's `connector_tags.connector_id` in `connectors` (`server_id` = `connector_id`) |
+| `certificate` | The `opc_certificates` document of the connector's `security.client_certificate_id`, when it is `active` and of kind `client` (`connectors_query`'s `$lookup`) |
+| Name | The read tag's `tag_name`, else the `connector_tags` one |
 
 Rules:
 - **One entry per address.** An address read by N pipelines is read once and written to N collections: one `names` item per pipeline (`collection` = `raw_{schedule_name}`).
 - **Same name source as the scouter.** `resolve_tag_names` names both the slot's `names` and the scouter's `model_tags`, so the gateway always writes the name the scouter looks for.
-- **`frequency`:** the lowest among the pipelines of the address, as an integer. Every read tag needs one, with or without `tag_id`.
-- **`tag_id`:** a missing, `null` or empty `tag_id` means the read tag does not use the tag database.
-- **`subscription_period_ms`:** the connector's value. For `opc-servers`, it is half the server's lowest tag frequency in the slot, as before.
-- **`use_security`:** the connector's value. For `opc-servers`, it is `true` when the server has a `cert_path`. The certificate itself comes from MinIO on the gateway, so the slot carries no certificate path.
+- **`frequency`:** the lowest among the pipelines of the address, as an integer. Every read tag needs one.
+- **`tag_id`:** required, an integer like in `connector_tags`. A read tag whose `tag_id` is missing, `null` or not an integer (`"7"` included) is left out and notified as invalid.
+- **`subscription_period_ms`:** the connector's value. A connector without one gets half its lowest tag frequency in the slot.
+- **`security`:** the connector's `mode` and `policy`, the source of the OPC security (`use_security` and `server_uri` no longer exist). A connector without them is left out like an incomplete one.
+- **`certificate`:** `certificate_id`, `application_uri` (the SAN URI of the certificate, used as the client application URI) and the `bucket`/`object_name` of `cert` and `private_key`; the gateway downloads them from MinIO. The certificate dates, `sha256` and audit fields stay out of the slot. There is no default certificate: `null` means the gateway connects without security, and a server that requires a certificate refuses the connection. A connector whose `client_certificate_id` is set but has no active `client` certificate (missing id, `retired`, another kind or an incomplete document) still goes into the slot with `certificate: null`, and is reported (see below).
 - **`active_read`:** off by default. It is on when any pipeline sets `read_tags[].active_read.enabled`, with the lowest `cycle_multiple`.
-- **Same `server_id` in both sources:** a server known through `connectors` and `opc-servers` under the same id is one server, and the `connectors` fields win. Two different servers with the same name get distinct slot keys (`name`, then `name:server_id`).
+- **Server keys:** two different connectors with the same name get distinct slot keys (`name`, then `name:server_id`).
 - **Slot division:** the addresses are split across the slots, sorted by server and address, and the names of an address always stay in the same slot.
 - **Unresolved tags:** a read tag that cannot be resolved is left out of the slot, and the rest of the slot is still written. It is notified once per run:
   - `ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the `tag_id` is not an active Read tag, or its document lacks `connector_id`, `address` or `tag_name`;
-  - `ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the connector is missing or inactive, or lacks `name`, `url` or `server_uri`;
-  - `ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the `server_id` is not in `opc-servers`;
-  - `ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION`: the read tag has no integer `frequency` or, without `tag_id`, lacks `server_id`, `tag_address` or `tag_name`.
-- **Only changed slots are written.** The slot is built in a stable order, so unchanged pipelines produce an identical slot, and the gateway does not re-subscribe. A slot still in the v1 format (`topics`, `cert_path`) is replaced once.
+  - `ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the connector is missing or inactive, or lacks `name`, `url`, `security.mode` or `security.policy`;
+  - `ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION`: the read tag has no integer `tag_id`, no integer `frequency`, or an enabled `active_read` without an integer `cycle_multiple`; or its pipeline lacks `schedule_name`, `model_id` or `model.name` (the value is then the `schedule_name`).
+  - `ORCHESTRATOR_CERTIFICATE_NOT_FOUND_DURING_SLOT_CONFIGURATION` (`WARNING`): the connector's `client_certificate_id` has no active `client` certificate, so its tags go into the slot without a certificate. Most likely a registration error.
+- **Only changed slots are written.** The slot is built in a stable order, so unchanged pipelines produce an identical slot, and the gateway does not re-subscribe.
 
-**Gateway plan B:** the gateway's legacy read engine (`READ_ENGINE=legacy`, kept until the end of SIENTIAPDE-1974) reads the slot v1 only. Once this version writes the slot v2, switching the gateway to the legacy engine also requires rolling the Orchestrator back to a version that writes v1.
+#### Write tags (predictions_batch, through the tag database)
 
-**Known limitation:** scouter schedules are only redeployed when the pipeline's `updated_at` changes. If a tag is renamed in `connector_tags`, the slot picks up the new name on the next run, but the scouter keeps the old `model_tags` until the pipeline is updated. When renaming, also update the pipeline, or keep the name in `read_tags[].tag_name`.
+A `predictions_batch` (or `xgboost_predictions_batch`) references its write tags by `tag_id` only, exactly like the read tags. Everything else comes from the tag database, so the pipeline does not say which server, address or type a tag has:
+
+```json
+{"workflow_type": "predictions_batch", "write_tags": [{"tag_id": 101}, {"tag_id": 102}]}
+```
+
+`process_schedules` turns them into the `opc_output_config` the Laborious already reads:
+
+```json
+{"plant": {
+  "prediction_tags": {"ns=3;i=2001": {"data_type": "double"}},
+  "confidence_tags": {"ns=3;i=2002": {"data_type": "float"}}}}
+```
+
+| Field | Source |
+|---|---|
+| Server key | `connector_tags.connector_id`. The Laborious publishes it as `server_id` on NATS, and the OPC Gateway resolves it in `connectors` |
+| `prediction_tags` / `confidence_tags` | `connector_tags.category`: `Write` / `Prediction Confidence` |
+| Address | `connector_tags.address` |
+| `data_type` | `connector_tags.data_type` (`float`, `double`, `real`, `int`, `integer`, `bool`, `boolean`, `str`, `string`), else `float` |
+
+A write tag that cannot be resolved is left out, the rest of the pipeline is still scheduled, and it is notified once per run (values are `schedule_name/tag_id`):
+- `ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the `tag_id` is not an active `Write` / `Prediction Confidence` tag, or its document lacks `connector_id`, `address`, `tag_name` or `category`;
+- `ORCHESTRATOR_WRITE_TAG_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the tag's connector is missing, inactive or incomplete in `connectors`;
+- `ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION`: the entry has no integer `tag_id`, or the tag's `data_type` is not one the gateway accepts.
+
+#### Redeploy on changes (`updated_at` and `config_hash`)
+
+`orchestrated_schedules` keeps, per schedule, when it was last deployed (`updated_at`) and the `config_hash` it was deployed with: a SHA-256 of the schedule as the Orchestrator builds it (the workflow input, frequency and offset, with tags and connectors already resolved), without `updated_at`. A schedule is redeployed when either:
+
+- the pipeline's `updated_at` is newer than its last deploy. Bumping `updated_at` by hand forces a redeploy even when nothing else changed; or
+- its `config_hash` differs from the stored one. Any change that reaches the schedule redeploys it, whether or not someone bumped an `updated_at`: a pipeline edited by hand, a tag renamed, moved, retyped, deactivated or deleted in `connector_tags`, a connector changed, or a new Orchestrator version that builds the schedule differently.
+
+A change that does not reach the schedule (e.g. a tag's `description`) does not redeploy. The slot is compared by content and follows on its own. A pipeline without `updated_at` is still redeployed on every run. Schedules deployed before `config_hash` existed have none stored, so the first run of this version redeploys all of them once and records it.
+
+#### Deploying
+
+1. Run the migrations that create the tag database and add `data_type` to the `connector_tags` validator (`sientia-dataops-database-migrations`).
+2. Every scouter `read_tags` and every `predictions_batch` / `xgboost_predictions_batch` `write_tags` entry is a `tag_id` of `connector_tags`, as an integer. Anything else is left out and notified.
+3. Deploy this version. Its first run redeploys every schedule once, to record their `config_hash` (see above).
 
 #### Architecture
 
@@ -202,7 +259,7 @@ Rules:
 flowchart LR
     subgraph "Parallel Data Loading"
         A[1. aggregate_documents_in_mongodb]
-        B[2. find_documents_in_mongodb<br/>OPC Servers]
+        B[2. aggregate_documents_in_mongodb<br/>connectors + connector_tags]
         C[3. find_documents_in_mongodb<br/>Orchestrated Schedules]
         D[4. load_opc_slots]
         E[5. load_active_ingestors]
@@ -631,6 +688,112 @@ python -m orchestrator.worker.worker
 | `LOG_LEVEL` | Application logging level | `INFO` | No |
 | `KAFKA_BOOTSTRAP_SERVERS` | Kafka bootstrap servers | - | No |
 
+### MongoDB Collections
+
+| Collection | Read / written by | Purpose |
+|---|---|---|
+| `pipelines` | Orchestrator (`pipelines_query`) | One document per Temporal schedule (`schedule_name`, `workflow_type`, `model_id`, `active`, `frequency`, ...) |
+| `models` | Orchestrator (`$lookup` in `pipelines_query`) | Gives each pipeline its `model` (`name`, `active`, `model_config`) |
+| `connectors` | Orchestrator (`connectors_query`) | Tag database: the OPC UA servers (`connector_id`, `type`, `name`, `url`, `security`, `subscription_period_ms`, `status`) |
+| `connector_tags` | Orchestrator (`connector_tags_query`) | Tag database: the tags (`tag_id`, `connector_id`, `address`, `tag_name`, `category`, `data_type`, `status`) |
+| `opc_certificates` | Orchestrator (`$lookup` in `connectors_query`) | The client certificate of each connector (`security.client_certificate_id` → `certificate_id`), only when `active` and of kind `client`: `application_uri` and the MinIO objects of `cert` and `private_key` go into the slot |
+| `orchestrated_schedules` | Orchestrator (read and written) | Per deployed schedule: `schedule_name`, `namespace`, `updated_at` (last deploy) and `config_hash` |
+| `raw_{schedule_name}` | Orchestrator (created) | One per scouter `topic`, where the OPC Gateway writes its tags. Created with a TTL index on `inserted_at` (`MONGODB_TTL_INDEX_HOURS`) |
+| `notification_queue` | Alerts / Reports (read) | Notifications (`notification_id`, `trigger`, `level`, `model_id`, `timestamp`, ...), loaded incrementally by `timestamp` |
+| `receiver_groups` | Alerts / Reports (read, `active: true`) | Who receives what (see below) |
+
+#### Tags in a pipeline
+
+Pipelines reference tags only by `tag_id` (an integer, as in `connector_tags`). Servers, addresses and data types come from the tag database, never from the pipeline.
+
+Scouter (`scouter` namespace), `workflow_type: "scouter"`:
+
+```json
+{
+  "schedule_name": "scouter-linreg",
+  "workflow_type": "scouter",
+  "model_id": "1",
+  "active": true,
+  "frequency": "1m",
+  "read_tags": [
+    {
+      "tag_id": 7,
+      "frequency": 15000,
+      "tag_name": "Pressure",
+      "aggr_func": "avg",
+      "data_range": [0, 100],
+      "active_read": {"enabled": true, "cycle_multiple": 2}
+    }
+  ]
+}
+```
+
+| Field | Required | Meaning |
+|---|---|---|
+| `tag_id` | Yes | An active `Read` tag of `connector_tags` |
+| `frequency` | Yes | Read period in ms, an integer. The slot uses the lowest among the pipelines of the address |
+| `tag_name` | No | Overrides the `connector_tags` name: the name the gateway writes and the scouter looks for |
+| `aggr_func` | No | Default `lts` |
+| `data_range` | No | Default `[-100, 100]` |
+| `active_read` | No | Off by default. When `enabled`, `cycle_multiple` must be an integer (default `1`) |
+
+Laborious (`laborious` namespace): only `predictions_batch` and `xgboost_predictions_batch` reference tags; `minimal_retrain`, `drift` and `simple_metrics` have none.
+
+```json
+{
+  "schedule_name": "pred-linreg",
+  "workflow_type": "predictions_batch",
+  "model_id": "1",
+  "active": true,
+  "frequency": "1m",
+  "query": "SELECT ...",
+  "write_tags": [{"tag_id": 101}, {"tag_id": 102}]
+}
+```
+
+Each `write_tags` entry is only a `tag_id`, of an active `Write` (prediction) or `Prediction Confidence` tag. The Orchestrator builds `opc_output_config` from them (see "Write tags" above) and overwrites any `opc_output_config` stored in the pipeline.
+
+The `connector_tags` documents those pipelines reference:
+
+```json
+{"tag_id": 7,   "connector_id": "plant", "address": "ns=3;i=1001", "tag_name": "Pressure", "category": "Read", "status": "Active"}
+{"tag_id": 101, "connector_id": "plant", "address": "ns=3;i=2001", "tag_name": "PRED-101", "category": "Write", "data_type": "double", "status": "Active"}
+{"tag_id": 102, "connector_id": "plant", "address": "ns=3;i=2002", "tag_name": "CONF-101", "category": "Prediction Confidence", "status": "Active"}
+```
+
+`pi_web_api_scouter` (`scouter` namespace) does not use the tag database: its `read_tags` is an object keyed by tag name, and it is not part of the OPC slot:
+
+```json
+{
+  "workflow_type": "pi_web_api_scouter",
+  "read_tags": {"Pressure": {"webid": "F1DP...", "aggr_func": "lts", "data_range": [0, 100]}},
+  "pi_web_api_config": {"endpoint": "https://pi/piwebapi", "period": "*-1d", "max_count": 1, "api_timeout": 30}
+}
+```
+
+#### Receiver groups
+
+```json
+{
+  "group_name": "admin_team",
+  "active": true,
+  "members": ["admin@example.com"],
+  "contents": ["core_alerts", "persistent_alerts", "reports"],
+  "levels": ["ERROR", "WARNING"],
+  "model_ids": ["1"],
+  "ignore_models_id": ["2"],
+  "ignore": ["SOME_NOTIFICATION_ID"]
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `contents` | What the group gets: `core_alerts` / `persistent_alerts` (Alerts workflow), `reports` (Reports workflow) |
+| `levels` | Optional allow-list of notification levels. Absent = every level the workflow loads |
+| `model_ids` | Optional allow-list of `model_id`s |
+| `ignore_models_id` | Optional deny-list of `model_id`s (applies together with `model_ids`) |
+| `ignore` | Optional deny-list of `notification_id`s |
+
 ### Workflow Configuration
 
 Temporal input configuration sample:
@@ -658,7 +821,7 @@ Temporal input configuration sample:
             },
             {
                 "$addFields": {
-                    "models": {
+                    "model": {
                         "$arrayElemAt": [
                             "$model_docs",
                             0
@@ -668,7 +831,7 @@ Temporal input configuration sample:
             },
             {
                 "$match": {
-                    "models.active": True
+                    "model.active": True
                 }
             },
             {
@@ -678,31 +841,69 @@ Temporal input configuration sample:
             }
         ]
     },
-    "opc_servers_query": {
-        "collection": "opc-servers",
-        "filters": {
-
-        }
-    },
     "connectors_query": {
         "collection": "connectors",
         "aggregation": [
             {"$match": {"type": "opc_ua", "status": "Active"}},
-            {"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "server_uri": 1,
-                          "use_security": 1, "subscription_period_ms": 1}}
+            {"$lookup": {"from": "opc_certificates",
+                         "let": {"certificate_id": "$security.client_certificate_id"},
+                         "pipeline": [
+                           {"$match": {"$expr": {"$eq": ["$certificate_id", "$$certificate_id"]},
+                                       "status": "active", "kind": "client"}},
+                           {"$project": {"_id": 0, "certificate_id": 1, "application_uri": 1,
+                                         "cert.bucket": 1, "cert.object_name": 1,
+                                         "private_key.bucket": 1, "private_key.object_name": 1}}
+                         ],
+                         "as": "certificates"}},
+            {"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "subscription_period_ms": 1,
+                          "security.mode": 1, "security.policy": 1,
+                          "security.client_certificate_id": 1, "certificates": 1}}
         ]
     },
     "connector_tags_query": {
         "collection": "connector_tags",
         "aggregation": [
-            {"$match": {"category": "Read", "status": "Active"}},
-            {"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1}}
+            {"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1,
+                          "category": 1, "data_type": 1, "status": 1}}
         ]
     }
 }
 ```
 
-`connectors_query` and `connector_tags_query` are optional: the values above are the defaults in the code (see "OPC slots" above).
+`connectors_query` and `connector_tags_query` are optional: the values above are the defaults in the code (see "OPC slots" and "Write tags" above).
+
+The `$lookup` must end in a `model` field (singular): every pipeline type reads `model.name` (and `model.model_config`, when present), so a pipeline without `model` fails to build its schedule.
+
+#### Simple metrics: parallel windows & adaptive frequency
+
+One `pipelines` doc per window, all with the same `model_id`. Name them `<model>-simple_metrics-<interval_minutes>m`; each doc becomes its own Temporal schedule. Thresholds are shared through `model.model_config`. The adaptive state is per doc.
+
+```json
+{
+    "schedule_name": "model-x-simple_metrics-60m",
+    "workflow_type": "simple_metrics",
+    "model_id": "model-x",
+    "active": true,
+    "interval_minutes": 60,
+    "frequency": "1h",
+    "frequency_alert": "5m"
+}
+```
+
+| Field | Meaning | Default |
+|---|---|---|
+| `frequency_alert` | Frequency while escalated, same format as `frequency`. Absent = adaptive mode off. Below `1m` (or unparseable), the orchestrator sends a WARNING each time it creates or updates the schedule. A value below `1m` is kept; an unparseable value is ignored (the schedule keeps `frequency`) | — |
+| `adaptive_state` | `{escalated, healthy_streak}`, written **only** by laborious (with `updated_at` as a BSON datetime when `escalated` flips) | `{false, 0}` |
+| `model.model_config.simple_metrics_thresholds` | `{<metric>_max, <metric>_min}` shared by all windows | — |
+| `model.model_config.simple_metrics_min_samples` | Windows with fewer rows neither alert nor escalate, and count toward recovery | — |
+
+Escalation, recovery and the per-model alert are implemented in laborious (SIENTIAPDE-2189, SIENTIAPDE-2191); the orchestrator only picks `frequency_alert` while `adaptive_state.escalated` is true.
+
+Rules:
+- **Escalation:** laborious escalates on the first breach.
+- **Recovery:** after `min(12, ceil(interval_minutes / frequency_alert))` consecutive runs without one. Empty windows count as runs without a breach.
+- **Alerts** are per model (trigger `<model_id>-simple_metrics`), so several windows seeing the same breach produce one alert.
+- **Delay:** the schedule switches on the orchestrator tick after laborious flips the state, at most one tick later.
 
 #### Alerts Workflow
 
@@ -884,7 +1085,9 @@ orchestrator/
 ├── utils/                   # Utility functions
 │   ├── connectors_config.py # Database configuration
 │   ├── email_builder.py    # Email template generation
-│   ├── orchestrator_functions.py # Pipeline utilities
+│   ├── pipeline_configs.py # Schedule input builders per pipeline type
+│   ├── tag_database.py     # Tag database indexes, OPC read names and write tags
+│   ├── opc_slots.py        # OPC slot v2 (contract C1)
 │   ├── converters.py       # Data type conversion utilities
 │   └── templates/          # Email HTML templates
 │       ├── email_template.html    # Report email template

e2e/conftest.py

@@ -92,9 +92,7 @@ async def register_namespace_if_missing(env: WorkflowEnvironment, namespace: str
     raise TimeoutError(f'Namespace {namespace} not ready within 5s')
 
 
-async def ensure_namespace_search_attributes(
-    env: WorkflowEnvironment, namespace: str
-) -> None:
+async def ensure_namespace_search_attributes(env: WorkflowEnvironment, namespace: str) -> None:
     """
     Register the orchestrator search attributes on a namespace, if missing.
 

e2e/helpers.py

@@ -25,8 +25,8 @@ MONGO_COLLECTIONS = (
     'receiver_groups',
     'orchestrated_schedules',
     'pipelines',
-    'opc_servers',
-    'opc-servers',
+    'connectors',
+    'connector_tags',
 )
 
 DATETIME_FORMAT_MS_WITH_TZ = '%Y-%m-%d %H:%M:%S.%f%z'
@@ -143,19 +143,25 @@ def seed_pipelines(
         client.close()
 
 
-def seed_opc_servers(
+def seed_tag_database(
     mongo_uri: str,
     database: str,
-    servers: list[dict[str, Any]],
-    *,
-    collection: str = 'opc_servers',
+    connectors: list[dict[str, Any]],
+    connector_tags: list[dict[str, Any]],
+    opc_certificates: list[dict[str, Any]] | None = None,
 ) -> None:
-    """Insert OPC server documents into the test Mongo database."""
+    """
+    Insert tag database documents (connectors, connector_tags and the opc_certificates
+    the connectors point to) into the test Mongo database.
+    """
     client = MongoClient(mongo_uri)
     try:
-        coll = client[database][collection]
-        if servers:
-            coll.insert_many(servers)
+        if connectors:
+            client[database]['connectors'].insert_many(connectors)
+        if connector_tags:
+            client[database]['connector_tags'].insert_many(connector_tags)
+        if opc_certificates:
+            client[database]['opc_certificates'].insert_many(opc_certificates)
     finally:
         client.close()
 

e2e/scenario_inputs/orchestrator_conflict.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -22,13 +18,16 @@
       "frequency": "1m",
       "offset": "0m",
       "query": "SELECT 1",
-      "write_tags": [
-        {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
-      ]
+      "write_tags": [{"tag_id": 101}]
     }
   ],
-  "opc_servers": [
-    {"id": "srv-1", "active": true}
+  "connectors": [
+    {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+     "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+  ],
+  "connector_tags": [
+    {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+     "category": "Write", "status": "Active", "updated_at": "@now-1d"}
   ],
   "active_ingestors": ["heartbeat:ingestor:1"]
 }

e2e/scenario_inputs/orchestrator_create_only.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -22,13 +18,16 @@
       "frequency": "1m",
       "offset": "0m",
       "query": "SELECT 1",
-      "write_tags": [
-        {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
-      ]
+      "write_tags": [{"tag_id": 101}]
     }
   ],
-  "opc_servers": [
-    {"id": "srv-1", "active": true}
+  "connectors": [
+    {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+     "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+  ],
+  "connector_tags": [
+    {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+     "category": "Write", "status": "Active", "updated_at": "@now-1d"}
   ],
   "active_ingestors": ["heartbeat:ingestor:1"]
 }

e2e/scenario_inputs/orchestrator_delete_only.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -24,8 +20,5 @@
       "interval_minutes": 60
     }
   ],
-  "opc_servers": [
-    {"id": "srv-1", "active": true}
-  ],
   "active_ingestors": ["heartbeat:ingestor:1"]
 }

e2e/scenario_inputs/orchestrator_drift_window.json

@@ -0,0 +1,33 @@
+{
+  "workflow_input": {
+    "schedule_name": "orchestrator-e2e-drift-window",
+    "pipelines_query": {
+      "collection": "pipelines",
+      "aggregation": [{"$match": {"active": true}}]
+    }
+  },
+  "pipelines": [
+    {
+      "schedule_name": "model-dr-drift-default",
+      "workflow_type": "drift",
+      "model_id": "model-dr",
+      "model": {"name": "Model model-dr"},
+      "active": true,
+      "updated_at": "@now",
+      "frequency": "1m",
+      "chunk_period": "hour"
+    },
+    {
+      "schedule_name": "model-dr-drift-explicit",
+      "workflow_type": "drift",
+      "model_id": "model-dr",
+      "model": {"name": "Model model-dr"},
+      "active": true,
+      "updated_at": "@now",
+      "frequency": "1m",
+      "interval_minutes": 60,
+      "chunk_period": "s"
+    }
+  ],
+  "active_ingestors": ["heartbeat:ingestor:1"]
+}

e2e/scenario_inputs/orchestrator_empty.json

@@ -4,13 +4,8 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [],
-  "opc_servers": [],
   "active_ingestors": []
 }

e2e/scenario_inputs/orchestrator_happy_path.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -22,9 +18,7 @@
       "frequency": "1m",
       "offset": "0m",
       "query": "SELECT 1",
-      "write_tags": [
-        {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
-      ]
+      "write_tags": [{"tag_id": 101}]
     },
     {
       "schedule_name": "drift-gpu",
@@ -39,8 +33,13 @@
       "interval_minutes": 60
     }
   ],
-  "opc_servers": [
-    {"id": "srv-1", "active": true, "name": "opc-1"}
+  "connectors": [
+    {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+     "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+  ],
+  "connector_tags": [
+    {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+     "category": "Write", "status": "Active", "updated_at": "@now-1d"}
   ],
   "active_ingestors": ["heartbeat:ingestor:1"]
 }

e2e/scenario_inputs/orchestrator_noop.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -22,13 +18,16 @@
       "frequency": "1m",
       "offset": "0m",
       "query": "SELECT 1",
-      "write_tags": [
-        {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
-      ]
+      "write_tags": [{"tag_id": 101}]
     }
   ],
-  "opc_servers": [
-    {"id": "srv-1", "active": true}
+  "connectors": [
+    {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+     "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+  ],
+  "connector_tags": [
+    {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+     "category": "Write", "status": "Active", "updated_at": "@now-1d"}
   ],
   "active_ingestors": ["heartbeat:ingestor:1"],
   "orchestrated_schedules": [

e2e/scenario_inputs/orchestrator_simple_metrics_windows.json

@@ -0,0 +1,40 @@
+{
+  "workflow_input": {
+    "schedule_name": "orchestrator-e2e-sm-windows",
+    "pipelines_query": {
+      "collection": "pipelines",
+      "aggregation": [{"$match": {"active": true}}]
+    }
+  },
+  "pipelines": [
+    {
+      "schedule_name": "model-sm-simple_metrics-60m",
+      "workflow_type": "simple_metrics",
+      "model_id": "model-sm",
+      "model": {
+        "name": "Model model-sm",
+        "model_config": {"target": "y", "simple_metrics_thresholds": {"rmse_max": 1.0}}
+      },
+      "active": true,
+      "updated_at": "@now",
+      "frequency": "1h",
+      "frequency_alert": "5m",
+      "interval_minutes": 60
+    },
+    {
+      "schedule_name": "model-sm-simple_metrics-1440m",
+      "workflow_type": "simple_metrics",
+      "model_id": "model-sm",
+      "model": {
+        "name": "Model model-sm",
+        "model_config": {"target": "y", "simple_metrics_thresholds": {"rmse_max": 1.0}}
+      },
+      "active": true,
+      "updated_at": "@now",
+      "frequency": "1d",
+      "frequency_alert": "1h",
+      "interval_minutes": 1440
+    }
+  ],
+  "active_ingestors": ["heartbeat:ingestor:1"]
+}

e2e/scenario_inputs/orchestrator_tag_database.json

@@ -0,0 +1,71 @@
+{
+  "workflow_input": {
+    "schedule_name": "orchestrator-e2e-tag-database",
+    "pipelines_query": {
+      "collection": "pipelines",
+      "aggregation": [{"$match": {"active": true}}]
+    }
+  },
+  "connectors": [
+    {
+      "connector_id": "plant",
+      "type": "opc_ua",
+      "name": "plant",
+      "url": "opc.tcp://plant:4840",
+      "subscription_period_ms": 500,
+      "security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256", "client_certificate_id": 1},
+      "status": "Active",
+      "updated_at": "@now-1d"
+    }
+  ],
+  "opc_certificates": [
+    {"certificate_id": 1, "kind": "client", "name": "sientia-platform", "status": "active",
+     "application_uri": "urn:sientia:opc-ingestor",
+     "cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/cert.pem", "sha256": null},
+     "private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/key.pem", "sha256": null},
+     "thumbprint_sha1": null, "not_before": "@now-1d", "not_after": "@now+365d",
+     "created_at": "@now-1d", "updated_at": "@now-1d", "created_by": "e2e", "updated_by": "e2e"},
+    {"certificate_id": 2, "kind": "client", "name": "retired", "status": "retired",
+     "application_uri": "urn:sientia:old",
+     "cert": {"bucket": "opc-certificates", "object_name": "client/retired/cert.pem"},
+     "private_key": {"bucket": "opc-certificates", "object_name": "client/retired/key.pem"},
+     "created_at": "@now-1d", "updated_at": "@now-1d", "created_by": "e2e", "updated_by": "e2e"}
+  ],
+  "connector_tags": [
+    {"tag_id": 7, "connector_id": "plant", "address": "ns=3;i=1001", "tag_name": "Pressure", "category": "Read", "status": "Active", "updated_at": "@now-1d"},
+    {"tag_id": 101, "connector_id": "plant", "address": "ns=3;i=2001", "tag_name": "PRED-101", "category": "Write", "data_type": "double", "status": "Active", "updated_at": "@now-1d"},
+    {"tag_id": 102, "connector_id": "plant", "address": "ns=3;i=2002", "tag_name": "CONF-101", "category": "Prediction Confidence", "status": "Active", "updated_at": "@now-1d"},
+    {"tag_id": 103, "connector_id": "plant", "address": "ns=3;i=2003", "tag_name": "PRED-OLD", "category": "Write", "status": "Inactive", "updated_at": "@now-1d"}
+  ],
+  "pipelines": [
+    {
+      "schedule_name": "opc-tagdb",
+      "workflow_type": "scouter",
+      "model_id": "model-opc",
+      "model": {"name": "Model model-opc"},
+      "active": true,
+      "updated_at": "@now-2d",
+      "frequency": "1m",
+      "read_tags": [{"tag_id": 7, "frequency": 1000, "aggr_func": "avg"}]
+    },
+    {
+      "schedule_name": "pred-tagdb",
+      "workflow_type": "predictions_batch",
+      "model_id": "model-opc",
+      "model": {"name": "Model model-opc"},
+      "active": true,
+      "updated_at": "@now-2d",
+      "frequency": "1m",
+      "query": "SELECT 1",
+      "write_tags": [
+        {"tag_id": 101},
+        {"tag_id": 102},
+        {"tag_id": 7},
+        {"tag_id": 103},
+        {"tag_id": 404},
+        {"tag_name": "PRED-NO-ID"}
+      ]
+    }
+  ],
+  "active_ingestors": ["heartbeat:ingestor:1"]
+}

e2e/scenario_inputs/orchestrator_ttl_index.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -22,24 +18,17 @@
       "frequency": "1m",
       "offset": "0m",
       "read_tags": [
-        {
-          "server_id": "srv-1",
-          "tag_name": "Read1",
-          "tag_address": "ns=2;s=Read1",
-          "aggr_func": "lts",
-          "frequency": 1000
-        }
+        {"tag_id": 1, "aggr_func": "lts", "frequency": 1000}
       ]
     }
   ],
-  "opc_servers": [
-    {
-      "id": "srv-1",
-      "active": true,
-      "server_name": "opc-1",
-      "url": "opc.tcp://localhost:4840",
-      "uri": "urn:opcfoundation:UA:DemoServer"
-    }
+  "connectors": [
+    {"connector_id": "opc-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://localhost:4840",
+     "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+  ],
+  "connector_tags": [
+    {"tag_id": 1, "connector_id": "opc-1", "address": "ns=2;s=Read1", "tag_name": "Read1",
+     "category": "Read", "status": "Active"}
   ],
   "active_ingestors": ["heartbeat:ingestor:1"]
 }

e2e/scenario_inputs/orchestrator_update_only.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -22,13 +18,16 @@
       "frequency": "5m",
       "offset": "0m",
       "query": "SELECT 1",
-      "write_tags": [
-        {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
-      ]
+      "write_tags": [{"tag_id": 101}]
     }
   ],
-  "opc_servers": [
-    {"id": "srv-1", "active": true}
+  "connectors": [
+    {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+     "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+  ],
+  "connector_tags": [
+    {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+     "category": "Write", "status": "Active", "updated_at": "@now-1d"}
   ],
   "active_ingestors": ["heartbeat:ingestor:1"],
   "orchestrated_schedules": [

e2e/scenarios.md

@@ -26,7 +26,9 @@ Source: `e2e/test_orchestrator_main_workflow.py`
 
 ### 1.1.1 Happy path
 
-Pipelines in Mongo → Temporal schedule created in the `laborious` namespace, `orchestrated_schedules` entries written.
+Pipelines in Mongo → both Temporal schedules created in the `laborious` namespace, one
+`orchestrated_schedules` entry per pipeline, and the `predictions_batch` write tag (`tag_id`)
+resolved into `opc_output_config` through the tag database.
 
 ### 1.3.1 Create-only
 
@@ -45,6 +47,53 @@ unrelated Temporal schedule present.
 
 First run creates TTL index on notification collection used by scouter pipelines.
 
+### 1.7.1 Simple metrics parallel windows
+
+Two `simple_metrics` pipelines for the same `model_id` (60 min and 1440 min windows) → one
+Temporal schedule per window, each with its own `frequency` as interval.
+
+### 1.7.2 Adaptive frequency escalation/recovery round trip
+
+Laborious-style write (`adaptive_state.escalated=true` + BSON-datetime `updated_at`) on one
+window → next run switches only that schedule to `frequency_alert`; the recovery write switches it
+back to `frequency`.
+
+### 1.8.1 Drift window and chunk_period
+
+Two `drift` pipelines on `frequency='1m'`: one without `interval_minutes` and with an invalid
+`chunk_period='hour'` → workflow input `interval=1` (frequency, no overlap),
+`chunk_period='min'` and exactly one `DRIFT_CHUNK_PERIOD_INVALID` notification; one with
+`interval_minutes=60` and `chunk_period='s'` → both forwarded unchanged, no warning
+(SIENTIAPDE-2212).
+
+### 1.9 OPC tag database: read and write tags by tag_id
+
+Scenario input: `orchestrator_tag_database.json` (one `opc_ua` connector, its `connector_tags` of
+every category, one inactive, an OPC scouter reading by `tag_id` and a `predictions_batch` writing by
+`tag_id`, with a Read tag, an inactive tag, an unknown `tag_id` and an entry without `tag_id` among its
+write tags).
+
+#### 1.9.1 Write tags through the tag database
+
+`write_tags: [{tag_id}]` → `opc_output_config` keyed by `connector_id`, address and `data_type`
+from `connector_tags` (`float` when absent), `Write` → `prediction_tags`, `Prediction Confidence` →
+`confidence_tags`. The Read, inactive and unknown tags are left out and notified once
+(`ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`); the entry without `tag_id` is notified
+(`ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION`).
+
+#### 1.9.2 A tag change redeploys its pipelines
+
+A second run with nothing changed redeploys nothing (`orchestrated_schedules` is not stamped
+again). Then renaming the read tag, moving the write tag and deleting the confidence tag in
+`connector_tags` by hand (no `updated_at` bumped anywhere, pipelines untouched) → the next run
+redeploys both through their `config_hash`: the scouter `model_tags` and the slot `names` take
+the new name, the `opc_output_config` the new address without the confidence tag, and
+`orchestrated_schedules` records the new `config_hash`.
+
+#### 1.9.3 OPC slot from the tag database
+
+The slot v2 written to Redis holds only the active Read tag, with the connector's server fields.
+
 ---
 
 ## 2. Alerts workflow

e2e/test_orchestrator_main_workflow.py

@@ -1,5 +1,8 @@
 """E2E tests for the Orchestrator main workflow."""
 
+import json
+from datetime import UTC, datetime, timedelta
+
 import pytest
 from pymongo import MongoClient
 from redis import Redis
@@ -12,9 +15,9 @@ from e2e.helpers import (
     load_scenario_input,
     make_workflow_id,
     seed_active_ingestors,
-    seed_opc_servers,
     seed_orchestrated_schedules,
     seed_pipelines,
+    seed_tag_database,
     start_and_await_workflow,
 )
 from orchestrator.workflows.orchestrator import Orchestrator
@@ -26,13 +29,20 @@ async def _schedule_ids(host: str, namespace: str) -> list[str]:
     return [schedule.id async for schedule in await client.list_schedules()]
 
 
+async def _schedule_every(host: str, schedule_id: str) -> timedelta:
+    """Return the interval of a laborious-namespace schedule."""
+    client = await Client.connect(host, namespace='laborious')
+    description = await client.get_schedule_handle(schedule_id).describe()
+    return description.schedule.spec.intervals[0].every
+
+
 def _apply_orchestrator_seeds(
     scenario: dict,
     mongo_uri: str,
     redis_client: Redis,
 ) -> None:
     """
-    Seed Mongo and Redis with the pipelines/opc_servers/ingestors declared in a scenario.
+    Seed Mongo and Redis with the pipelines/tag database/ingestors of a scenario.
 
     Args:
         scenario: Scenario payload returned by ``load_scenario_input``.
@@ -40,7 +50,13 @@ def _apply_orchestrator_seeds(
         redis_client: Redis client connected to the test instance.
     """
     seed_pipelines(mongo_uri, E2E_DATABASE, scenario.get('pipelines', []))
-    seed_opc_servers(mongo_uri, E2E_DATABASE, scenario.get('opc_servers', []))
+    seed_tag_database(
+        mongo_uri,
+        E2E_DATABASE,
+        scenario.get('connectors', []),
+        scenario.get('connector_tags', []),
+        scenario.get('opc_certificates', []),
+    )
     seed_active_ingestors(redis_client, scenario.get('active_ingestors', []))
     if scenario.get('orchestrated_schedules'):
         seed_orchestrated_schedules(
@@ -75,14 +91,19 @@ async def test_scenario_1_1_1_happy_path(
     try:
         tracked = list(mongo[E2E_DATABASE]['orchestrated_schedules'].find())
         names = {doc['schedule_name'] for doc in tracked}
-        assert names, f'Expected orchestrated_schedules rows, got {tracked}'
+        assert names == {'pred-legacy', 'drift-gpu'}, f'Unexpected orchestrated_schedules {tracked}'
     finally:
         mongo.close()
 
     laborious_schedules = await _schedule_ids(temporal_host, 'laborious')
-    assert 'pred-legacy' in laborious_schedules or 'drift-gpu' in laborious_schedules, (
+    assert {'pred-legacy', 'drift-gpu'} <= set(laborious_schedules), (
         f'Expected Temporal schedules in laborious, got {laborious_schedules}'
     )
+    # The write tag comes from the tag database, keyed by the connector the gateway writes to
+    pred = await _schedule_input(temporal_host, 'pred-legacy')
+    assert pred['opc_output_config'] == {
+        'srv-1': {'prediction_tags': {'ns=2;s=Tag1': {'data_type': 'float'}}}
+    }
 
 
 @pytest.mark.e2e
@@ -224,3 +245,326 @@ async def test_scenario_1_6_1_ttl_index_bootstrap(
         assert any('expireAfterSeconds' in info for info in indexes.values())
     finally:
         mongo.close()
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_7_1_simple_metrics_parallel_windows(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    temporal_host: str,
+    redis_client: Redis,
+):
+    """1.7.1: One pipelines doc per window gives one schedule per window, each with its own interval."""
+    scenario = load_scenario_input('orchestrator_simple_metrics_windows')
+    _apply_orchestrator_seeds(scenario, mongo_uri, redis_client)
+
+    await start_and_await_workflow(
+        temporal_env.client,
+        Orchestrator.run,
+        scenario['workflow_input'],
+        make_workflow_id('orchestrator-sm-windows'),
+    )
+
+    short, long = 'model-sm-simple_metrics-60m', 'model-sm-simple_metrics-1440m'
+    assert await _schedule_every(temporal_host, short) == timedelta(hours=1)
+    assert await _schedule_every(temporal_host, long) == timedelta(days=1)
+
+
+def _laborious_adaptive_write(mongo_uri: str, schedule_name: str, escalated: bool) -> None:
+    """Replicates laborious `update_simple_metrics_adaptive_state` on an escalated flip."""
+    mongo = MongoClient(mongo_uri)
+    try:
+        mongo[E2E_DATABASE]['pipelines'].update_one(
+            {'schedule_name': schedule_name},
+            {
+                '$set': {
+                    'adaptive_state': {'escalated': escalated, 'healthy_streak': 0},
+                    # BSON datetime, as laborious writes it; a string would break the tick.
+                    'updated_at': datetime.now(UTC),
+                }
+            },
+        )
+    finally:
+        mongo.close()
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_7_2_simple_metrics_adaptive_frequency_round_trip(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    temporal_host: str,
+    redis_client: Redis,
+):
+    """1.7.2: Escalation written by laborious switches only that window to frequency_alert; recovery switches it back."""
+    scenario = load_scenario_input('orchestrator_simple_metrics_windows')
+    _apply_orchestrator_seeds(scenario, mongo_uri, redis_client)
+    short, long = 'model-sm-simple_metrics-60m', 'model-sm-simple_metrics-1440m'
+
+    async def run(tag: str) -> None:
+        await start_and_await_workflow(
+            temporal_env.client,
+            Orchestrator.run,
+            scenario['workflow_input'],
+            make_workflow_id(f'orchestrator-sm-adaptive-{tag}'),
+        )
+
+    await run('create')
+    assert await _schedule_every(temporal_host, short) == timedelta(hours=1)
+
+    _laborious_adaptive_write(mongo_uri, short, escalated=True)
+    await run('escalate')
+    assert await _schedule_every(temporal_host, short) == timedelta(minutes=5)
+    assert await _schedule_every(temporal_host, long) == timedelta(days=1)
+
+    _laborious_adaptive_write(mongo_uri, short, escalated=False)
+    await run('recover')
+    assert await _schedule_every(temporal_host, short) == timedelta(hours=1)
+
+
+async def _schedule_input(host: str, schedule_id: str, namespace: str = 'laborious') -> dict:
+    """Return the decoded workflow input of a schedule (laborious namespace by default)."""
+    client = await Client.connect(host, namespace=namespace)
+    description = await client.get_schedule_handle(schedule_id).describe()
+    action = description.schedule.action
+    assert isinstance(action, ScheduleActionStartWorkflow)
+    [input_data] = await client.data_converter.decode(action.args)
+    return input_data
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_8_1_drift_window_and_chunk_period(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    temporal_host: str,
+    redis_client: Redis,
+    notification_handler,
+):
+    """1.8.1: Drift window defaults to frequency; invalid chunk_period warns and falls back."""
+    scenario = load_scenario_input('orchestrator_drift_window')
+    _apply_orchestrator_seeds(scenario, mongo_uri, redis_client)
+
+    await start_and_await_workflow(
+        temporal_env.client,
+        Orchestrator.run,
+        scenario['workflow_input'],
+        make_workflow_id('orchestrator-drift-window'),
+    )
+
+    default = await _schedule_input(temporal_host, 'model-dr-drift-default')
+    assert default['interval'] == 1
+    assert default['chunk_period'] == 'min'
+    assert default['chunk_period_configured'] == 'hour'
+
+    explicit = await _schedule_input(temporal_host, 'model-dr-drift-explicit')
+    assert explicit['interval'] == 60
+    assert explicit['chunk_period'] == 's'
+    assert 'chunk_period_configured' not in explicit
+
+    warnings = list(
+        notification_handler.mongo_collection.find(
+            {'notification_id': 'DRIFT_CHUNK_PERIOD_INVALID'}
+        )
+    )
+    assert len(warnings) == 1
+    assert warnings[0]['model_id'] == 'model-dr'
+
+
+# 1.9 OPC tag database: read and write tags by tag_id (SIENTIAPDE-1974)
+
+
+async def _run_tag_database_scenario(temporal_env: WorkflowEnvironment, tag: str) -> None:
+    await start_and_await_workflow(
+        temporal_env.client,
+        Orchestrator.run,
+        load_scenario_input('orchestrator_tag_database')['workflow_input'],
+        make_workflow_id(f'orchestrator-tagdb-{tag}'),
+    )
+
+
+def _notifications(notification_handler, notification_id: str) -> list[dict]:
+    return list(notification_handler.mongo_collection.find({'notification_id': notification_id}))
+
+
+def _opc_slots(redis_client: Redis) -> dict[str, dict]:
+    return {
+        key if isinstance(key, str) else key.decode(): json.loads(redis_client.get(key))
+        for key in redis_client.keys('slot:opc_tags:*')
+    }
+
+
+def _orchestrated_schedules(mongo_uri: str) -> dict[str, dict]:
+    """orchestrated_schedules by schedule_name: updated_at (last deploy) and config_hash."""
+    mongo = MongoClient(mongo_uri)
+    try:
+        return {
+            doc['schedule_name']: {
+                'updated_at': doc['updated_at'],
+                'config_hash': doc.get('config_hash'),
+            }
+            for doc in mongo[E2E_DATABASE]['orchestrated_schedules'].find()
+        }
+    finally:
+        mongo.close()
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_9_1_write_tags_through_tag_database(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    temporal_host: str,
+    redis_client: Redis,
+    notification_handler,
+):
+    """1.9.1: write_tags by tag_id become the opc_output_config; the rest is notified."""
+    _apply_orchestrator_seeds(
+        load_scenario_input('orchestrator_tag_database'), mongo_uri, redis_client
+    )
+
+    await _run_tag_database_scenario(temporal_env, 'write')
+
+    pred = await _schedule_input(temporal_host, 'pred-tagdb')
+    assert pred['opc_output_config'] == {
+        'plant': {
+            'prediction_tags': {'ns=3;i=2001': {'data_type': 'double'}},
+            'confidence_tags': {'ns=3;i=2002': {'data_type': 'float'}},
+        }
+    }
+
+    not_found = _notifications(
+        notification_handler, 'ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION'
+    )
+    assert len(not_found) == 1
+    # A Read tag, an inactive tag and an unknown tag_id
+    for value in ('pred-tagdb/7', 'pred-tagdb/103', 'pred-tagdb/404'):
+        assert value in not_found[0]['message']
+    invalid = _notifications(
+        notification_handler, 'ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION'
+    )
+    assert len(invalid) == 1
+    assert 'PRED-NO-ID' in invalid[0]['message']
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_9_2_tag_change_redeploys_pipelines(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    temporal_host: str,
+    redis_client: Redis,
+):
+    """
+    1.9.2: changing or deleting a tag in connector_tags, without touching any updated_at,
+    redeploys the pipelines that reference it (config_hash); an unchanged run does not.
+    """
+    _apply_orchestrator_seeds(
+        load_scenario_input('orchestrator_tag_database'), mongo_uri, redis_client
+    )
+    await _run_tag_database_scenario(temporal_env, 'before-change')
+
+    scouter = await _schedule_input(temporal_host, 'opc-tagdb', namespace='scouter')
+    assert set(scouter['model_tags']) == {'Pressure'}
+    deployed = _orchestrated_schedules(mongo_uri)
+    assert all(doc.get('config_hash') for doc in deployed.values()), deployed
+
+    # Nothing changed: no redeploy, so orchestrated_schedules is not stamped again
+    await _run_tag_database_scenario(temporal_env, 'unchanged')
+    assert _orchestrated_schedules(mongo_uri) == deployed
+
+    # As an edit by hand in Compass: rename the read tag, move a write tag and delete the
+    # confidence tag, with no updated_at bumped anywhere
+    mongo = MongoClient(mongo_uri)
+    try:
+        tags = mongo[E2E_DATABASE]['connector_tags']
+        tags.update_one({'tag_id': 7}, {'$set': {'tag_name': 'Pressao'}})
+        tags.update_one({'tag_id': 101}, {'$set': {'address': 'ns=3;i=3001'}})
+        tags.delete_one({'tag_id': 102})
+    finally:
+        mongo.close()
+
+    await _run_tag_database_scenario(temporal_env, 'after-change')
+
+    scouter = await _schedule_input(temporal_host, 'opc-tagdb', namespace='scouter')
+    assert set(scouter['model_tags']) == {'Pressao'}
+    pred = await _schedule_input(temporal_host, 'pred-tagdb')
+    assert pred['opc_output_config'] == {
+        'plant': {'prediction_tags': {'ns=3;i=3001': {'data_type': 'double'}}}
+    }
+    redeployed = _orchestrated_schedules(mongo_uri)
+    for name in ('opc-tagdb', 'pred-tagdb'):
+        assert redeployed[name]['config_hash'] != deployed[name]['config_hash']
+        assert redeployed[name]['updated_at'] > deployed[name]['updated_at']
+    # The slot names the read tag like the scouter does (SC1)
+    [slot] = _opc_slots(redis_client).values()
+    [name] = slot['plant']['tags']['ns=3;i=1001']['names']
+    assert name['tag_name'] == 'Pressao'
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_9_3_opc_slot_from_tag_database(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    redis_client: Redis,
+):
+    """1.9.3: the slot v2 holds only the active Read tags of the tag database."""
+    _apply_orchestrator_seeds(
+        load_scenario_input('orchestrator_tag_database'), mongo_uri, redis_client
+    )
+
+    await _run_tag_database_scenario(temporal_env, 'slot')
+
+    assert _opc_slots(redis_client) == {
+        'slot:opc_tags:1': {
+            'plant': {
+                'server_id': 'plant',
+                'name': 'plant',
+                'url': 'opc.tcp://plant:4840',
+                'subscription_period_ms': 500,
+                # The active client certificate of client_certificate_id, without its
+                # dates, sha256 or audit fields (D5)
+                'security': {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'},
+                'certificate': {
+                    'certificate_id': 1,
+                    'application_uri': 'urn:sientia:opc-ingestor',
+                    'cert': {
+                        'bucket': 'opc-certificates',
+                        'object_name': 'client/sientia-platform/cert.pem',
+                    },
+                    'private_key': {
+                        'bucket': 'opc-certificates',
+                        'object_name': 'client/sientia-platform/key.pem',
+                    },
+                },
+                'tags': {
+                    'ns=3;i=1001': {
+                        'frequency': 1000,
+                        'active_read': {'enabled': False},
+                        'names': [
+                            {
+                                'collection': 'raw_opc-tagdb',
+                                'tag_name': 'Pressure',
+                                'model_id': 'model-opc',
+                                'model_name': 'Model model-opc',
+                            }
+                        ],
+                    }
+                },
+            }
+        }
+    }

orchestrator/activities/email.py

@@ -177,14 +177,13 @@ class Email(SientiaMonitoring):
             self.logger.error(f'SMTP server disconnected: {e}')
             self.logger.info(f'Reconnecting to {self.smtp_server}:{self.smtp_port}')
 
-            if self.server:
-                try:
-                    self.server.quit()
-                except SMTPServerDisconnected as e:
-                    self.logger.info(f'Server already disconnected: {e}')
-                except Exception as e:
-                    self.logger.error(f'Failed to quit server: {e}')
-                    raise e
+            try:
+                self.server.quit()
+            except SMTPServerDisconnected as e:
+                self.logger.info(f'Server already disconnected: {e}')
+            except Exception as e:
+                self.logger.error(f'Failed to quit server: {e}')
+                raise e
 
             self.server = smtplib.SMTP(self.smtp_server, self.smtp_port, timeout=20)
             if self.sender_password:

orchestrator/activities/formatters.py

@@ -1,6 +1,7 @@
 from temporalio import activity, workflow
 
 with workflow.unsafe.imports_passed_through():
+    import hashlib
     import json
     from collections.abc import Callable
     from logging import Logger
@@ -17,22 +18,95 @@ with workflow.unsafe.imports_passed_through():
         now,
     )
 
-    from orchestrator.utils.orchestrator_functions import (
-        build_tag_config,
+    from orchestrator.utils.opc_slots import build_tag_config, gather_read_tags
+    from orchestrator.utils.pipeline_configs import (
+        chunk_period_warning,
         drift,
-        gather_read_tags,
-        index_connector_tags,
-        index_connectors,
+        frequency_alert_warning,
         minimal_retrain,
         pi_web_api_scouter,
         predictions_batch,
-        resolve_tag_names,
         scouter,
         simple_metrics,
     )
+    from orchestrator.utils.tag_database import (
+        WRITE_CATEGORIES,
+        WRITE_WORKFLOW_TYPES,
+        index_connector_tags,
+        index_connectors,
+        resolve_tag_names,
+        resolve_write_tags,
+    )
 
 topic_separator = '\n ========== \n'
 
+# Schedule fields left out of config_hash: updated_at is compared on its own
+CONFIG_HASH_EXCLUDED_FIELDS = ('updated_at', 'config_hash')
+
+# Read tags left out of the slots, by gather_read_tags issue kind:
+# (kind, notification_id, message, level)
+SLOT_ISSUES = (
+    (
+        'tags',
+        'ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+        'Tags {} not found among the active, complete Read tags of connector_tags',
+        NotificationLevel.ERROR,
+    ),
+    (
+        'connectors',
+        'ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+        'Connectors {} not found, not active or incomplete in connectors',
+        NotificationLevel.ERROR,
+    ),
+    (
+        'invalid',
+        'ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION',
+        'Read tags {} have no integer tag_id, no integer frequency or an active_read without an '
+        'integer cycle_multiple; or their pipeline lacks schedule_name, model_id or model.name',
+        NotificationLevel.ERROR,
+    ),
+    (
+        'certificates',
+        'ORCHESTRATOR_CERTIFICATE_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+        'Connectors {} have a security.client_certificate_id without an active client '
+        'certificate in opc_certificates: they go into the slot without a certificate, so '
+        'without security',
+        NotificationLevel.WARNING,
+    ),
+)
+
+# Write tags left out of the predictions_batch opc_output_config, by resolve_write_tags
+# issue kind; values are schedule_name/<tag_id or connector_id>
+WRITE_TAG_ISSUES = (
+    (
+        'tags',
+        'ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION',
+        'Write tags {} not found among the active, complete Write and Prediction Confidence '
+        'tags of connector_tags',
+        NotificationLevel.ERROR,
+    ),
+    (
+        'connectors',
+        'ORCHESTRATOR_WRITE_TAG_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION',
+        'Connectors {} of write tags not found, not active or incomplete in connectors',
+        NotificationLevel.ERROR,
+    ),
+    (
+        'invalid',
+        'ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION',
+        'Write tags {} have no integer tag_id or a data_type the OPC Gateway does not accept',
+        NotificationLevel.ERROR,
+    ),
+)
+
+
+def _merge_issues(
+    issues: dict[str, set[str]], pipeline_issues: dict[str, set[str]], schedule_name: str
+) -> None:
+    """Add one pipeline's issues to the run's, each value prefixed with schedule_name/."""
+    for kind, values in pipeline_issues.items():
+        issues.setdefault(kind, set()).update(f'{schedule_name}/{value}' for value in values)
+
 
 class ScheduleType(TypedDict):
     """
@@ -104,6 +178,20 @@ Each entry specifies:
 """
 
 
+def schedule_config_hash(schedule: dict[str, Any]) -> str:
+    """
+    Fingerprint of a schedule as process_schedules builds it, without updated_at.
+
+    Stored in orchestrated_schedules when the schedule is deployed: any change to what
+    reaches Temporal (a pipeline edited without bumping updated_at, a tag renamed, moved,
+    retyped, deactivated or deleted, a connector changed, or a new Orchestrator version
+    building it differently) changes the hash and redeploys the schedule.
+    """
+    content = {k: v for k, v in schedule.items() if k not in CONFIG_HASH_EXCLUDED_FIELDS}
+    serialized = json.dumps(content, sort_keys=True, separators=(',', ':'), default=str)
+    return hashlib.sha256(serialized.encode()).hexdigest()
+
+
 class Formatters(SientiaMonitoring):
     """
     Schedule and slot configuration formatting and notification filtering activity.
@@ -176,8 +264,16 @@ class Formatters(SientiaMonitoring):
             - input_data (dict[str, Any]): The input data containing
               the schedules to process.
                 - pipelines (list[dict[str, Any]]): The schedules to process.
-                - connector_tags (list[dict[str, Any]], optional): Active Read tags of
-                  the tag database, to name the scouter read tags that carry only a tag_id.
+                - connectors (list[dict[str, Any]], optional): OPC UA connectors of the
+                  tag database, for the write tags.
+                - connector_tags (list[dict[str, Any]], optional): The connector_tags the
+                  pipelines reference, of every category and status.
+
+        Scouter read tags that carry only a tag_id are named from the tag database (the
+        same names as the slot). A predictions_batch gets its opc_output_config from its
+        write_tags ([{tag_id}]) through the tag database; what cannot be resolved is left
+        out and notified once per run. A change to a tag reaches the schedule through
+        create_schedule_config, which compares the resolved schedule (config_hash).
 
         Returns:
             - dict[str, Any]: The schedule configuration dictionary keyed by namespace
@@ -188,7 +284,11 @@ class Formatters(SientiaMonitoring):
         self.info('Processing schedules...', metadata=metadata)
 
         pipelines = input_data['pipelines']
-        tags_by_id = index_connector_tags(input_data.get('connector_tags', []))
+        connector_tags = input_data.get('connector_tags', [])
+        read_tags_by_id = index_connector_tags(connector_tags)
+        write_tags_by_id = index_connector_tags(connector_tags, WRITE_CATEGORIES)
+        connectors = index_connectors(input_data.get('connectors', []))
+        write_issues: dict[str, set[str]] = {}
 
         schedule_config: dict[str, dict[str, Any]] = {
             self.scouter_namespace: {},
@@ -202,24 +302,34 @@ class Formatters(SientiaMonitoring):
                 self.error(f'Workflow type {workflow_type} not supported', metadata=metadata)
                 continue
 
+            schedule_type = schedule_types[workflow_type]
+            namespace = schedule_type['namespace']
+            function = schedule_type['function']
+            schedule_name = pipeline['schedule_name']
+
             if workflow_type == 'scouter' and 'read_tags' in pipeline:
                 # Same names as the slot's names[] (process_slots): SIENTIAPDE-1974 SC1
                 pipeline = {
                     **pipeline,
-                    'read_tags': resolve_tag_names(pipeline['read_tags'], tags_by_id),
+                    'read_tags': resolve_tag_names(pipeline['read_tags'], read_tags_by_id),
                 }
 
-            schedule_type = schedule_types[workflow_type]
-            namespace = schedule_type['namespace']
-            function = schedule_type['function']
+            if workflow_type in WRITE_WORKFLOW_TYPES:
+                opc_output_config, issues = resolve_write_tags(
+                    pipeline.get('write_tags') or [], write_tags_by_id, connectors
+                )
+                _merge_issues(write_issues, issues, schedule_name)
+                pipeline = {**pipeline, 'opc_output_config': opc_output_config}
 
-            schedule_config[namespace][pipeline['schedule_name']] = {
+            schedule_config[namespace][schedule_name] = {
                 **function(pipeline),
-                'updated_at': pipeline.get(
-                    'updated_at', now().strftime(DATETIME_FORMAT_MS_WITH_TZ)
-                ),
+                # A pipeline without updated_at is redeployed on every run, as before
+                'updated_at': pipeline.get('updated_at')
+                or now().strftime(DATETIME_FORMAT_MS_WITH_TZ),
             }
 
+        self._notify_issues(write_issues, WRITE_TAG_ISSUES, metadata)
+
         self.info('Processed schedules', metadata=metadata)
         self.debug(json.dumps(schedule_config, indent=4, sort_keys=True), metadata=metadata)
 
@@ -234,20 +344,20 @@ class Formatters(SientiaMonitoring):
 
         The division is by address: all the names of an address stay in the same slot
         (SIENTIAPDE-1974, O5). Read tags that cannot be resolved are left out and
-        notified once per run: a tag_id missing from connector_tags, a connector missing
-        or inactive, a server_id missing from opc_servers, or a read tag without a usable
-        frequency (or, without tag_id, without server_id, tag_address or tag_name).
+        notified once per run: a read tag without tag_id or a usable frequency, a tag_id
+        missing from connector_tags, or a connector missing or inactive. A connector whose
+        client_certificate_id has no active client certificate goes into the slot without
+        a certificate, with a warning (D5).
 
         Args:
             - input_data (dict[str, Any]): The input data containing
               the schedules to process.
                 - pipelines (list[dict[str, Any]]): The schedules to process.
-                - opc_servers (list[dict[str, Any]]): The OPC servers (opc-servers), for
-                  read tags without tag_id.
                 - connectors (list[dict[str, Any]], optional): Active OPC UA connectors of
-                  the tag database, for read tags with tag_id.
-                - connector_tags (list[dict[str, Any]], optional): Active Read tags of the
-                  tag database, for read tags with tag_id.
+                  the tag database, the only source of the OPC servers, each with the
+                  certificates of its client_certificate_id (connectors_query).
+                - connector_tags (list[dict[str, Any]], optional): The connector_tags the
+                  pipelines reference; only the active Read tags are used.
                 - active_ingestors (list[str]): The active ingestors to divide into slots.
 
         Returns:
@@ -260,12 +370,11 @@ class Formatters(SientiaMonitoring):
 
         pipelines = input_data['pipelines']
         active_ingestors = input_data['active_ingestors']
-        opc_servers = {str(server['id']): server for server in input_data['opc_servers']}
         connectors = index_connectors(input_data.get('connectors', []))
         tags_by_id = index_connector_tags(input_data.get('connector_tags', []))
 
-        entries, issues = gather_read_tags(pipelines, opc_servers, connectors, tags_by_id)
-        self._notify_slot_issues(issues, metadata)
+        entries, issues = gather_read_tags(pipelines, connectors, tags_by_id)
+        self._notify_issues(issues, SLOT_ISSUES, metadata)
 
         number_of_slots = len(active_ingestors) if active_ingestors else 1
         entries_per_slot = ceil(len(entries) / number_of_slots)
@@ -280,46 +389,30 @@ class Formatters(SientiaMonitoring):
 
         return slot_config
 
-    def _notify_slot_issues(self, issues: dict[str, set[str]], metadata: dict[str, Any]) -> None:
+    def _notify_issues(
+        self,
+        issues: dict[str, set[str]],
+        kinds: tuple[tuple[str, str, str, str], ...],
+        metadata: dict[str, Any],
+    ) -> None:
         """
-        Notify the read tags left out of the slots, one notification per kind of issue.
+        Notify what was left out of the slots or schedules, one notification per kind.
 
         Args:
-            issues (dict[str, set[str]]): What gather_read_tags left out, by kind:
-                'servers' (opc-servers server_id), 'tags' (tag_id), 'connectors'
-                (connector_id) and 'invalid' (schedule_name/read tag)
+            issues (dict[str, set[str]]): What was left out, by kind
+            kinds (tuple[tuple[str, str, str, str], ...]): (kind, notification_id, message
+                with one {} for the joined values, level), e.g. SLOT_ISSUES or
+                WRITE_TAG_ISSUES
             metadata (dict[str, Any]): Metadata for the notifications
         """
-        kinds = (
-            (
-                'servers',
-                'ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION',
-                'Servers {} not found in opc_servers',
-            ),
-            (
-                'tags',
-                'ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION',
-                'Tags {} not found among the active, complete Read tags of connector_tags',
-            ),
-            (
-                'connectors',
-                'ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION',
-                'Connectors {} not found, not active or incomplete in connectors',
-            ),
-            (
-                'invalid',
-                'ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION',
-                'Read tags {} have no integer frequency or lack server_id, tag_address or tag_name',
-            ),
-        )
-        for kind, notification_id, message in kinds:
+        for kind, notification_id, message, level in kinds:
             if issues.get(kind):
                 self.send_notification(
                     metadata=metadata,
                     notification_id=notification_id,
                     message=message.format(', '.join(sorted(issues[kind]))),
                     block='orchestrator',
-                    level=NotificationLevel.ERROR,
+                    level=level,
                 )
 
     @activity.defn(name='format_schedule_config')
@@ -329,6 +422,8 @@ class Formatters(SientiaMonitoring):
 
         Transforms a list of schedule configurations into a nested dictionary structure
         organized by namespace and schedule name for efficient lookup and comparison.
+        Each schedule keeps when it was last deployed (updated_at) and the config_hash
+        it was deployed with (None for schedules deployed before config_hash existed).
 
         Args:
             input_data (dict[str, Any]): The input data containing:
@@ -336,7 +431,7 @@ class Formatters(SientiaMonitoring):
                 - metadata (dict): Metadata for logging purposes
 
         Returns:
-            dict[str, Any]: The formatted schedule config organized by namespace and schedule name
+            dict[str, Any]: {namespace: {schedule_name: {'updated_at', 'config_hash'}}}
         """
         metadata = input_data['metadata']
 
@@ -344,16 +439,18 @@ class Formatters(SientiaMonitoring):
 
         schedule_config = input_data['schedule_config']
 
-        config: dict[str, dict[str, str]] = {}
+        config: dict[str, dict[str, dict[str, Any]]] = {}
         for schedule in schedule_config:
             namespace = schedule['namespace']
             schedule_name = schedule['schedule_name']
-            updated_at = schedule['updated_at']
 
             if namespace not in config:
                 config[namespace] = {}
 
-            config[namespace][schedule_name] = updated_at
+            config[namespace][schedule_name] = {
+                'updated_at': schedule['updated_at'],
+                'config_hash': schedule.get('config_hash'),
+            }
 
         self.info('Formatted schedule config', metadata=metadata)
         self.debug(json.dumps(config, indent=4, sort_keys=True), metadata=metadata)
@@ -372,32 +469,42 @@ class Formatters(SientiaMonitoring):
         """
         Compare new and current schedules to determine which should be updated or created.
 
-        Compares schedule timestamps to identify schedules that need updating (newer timestamp)
-        or creating (schedule doesn't exist). Results are accumulated in the provided dictionaries.
+        A deployed schedule is updated when its updated_at is newer than its last deploy
+        (bumping updated_at forces a redeploy) or when its config_hash differs from the one
+        it was deployed with (any change to the resolved schedule, including tag database
+        changes and deleted tags). A schedule not deployed yet is created. Each schedule
+        sent to update or create carries its config_hash, stored once it is deployed.
+        Results are accumulated in the provided dictionaries.
 
         Args:
             schedules (dict[str, Any]): New schedules to compare
-            current_schedules (dict[str, Any]): Existing schedules to compare against
+            current_schedules (dict[str, Any]): Existing schedules to compare against, as
+                format_schedule_config gives them
             to_update (dict[str, Any]): Output accumulator for schedules that need updating
             to_create (dict[str, Any]): Output accumulator for schedules that need creating
             namespace (str): Namespace for the schedules being compared
             metadata (dict[str, Any]): Metadata for logging purposes
         """
         for schedule_name, schedule in schedules.items():
+            config_hash = schedule_config_hash(schedule)
+            deployed = {**schedule, 'config_hash': config_hash}
+
             if schedule_name in current_schedules:
                 update_timestamp = schedule.get('updated_at', now())
 
-                old_timestamp = current_schedules[schedule_name]
+                current = current_schedules[schedule_name]
+                old_timestamp = current['updated_at']
 
                 self.debug(
-                    f'Comparing schedule {schedule_name}:{update_timestamp} vs {old_timestamp}',
+                    f'Comparing schedule {schedule_name}:{update_timestamp} vs {old_timestamp}'
+                    f' (config_hash {config_hash} vs {current.get("config_hash")})',
                     metadata=metadata,
                 )
 
-                if update_timestamp > old_timestamp:
-                    to_update[namespace][schedule_name] = schedule
+                if update_timestamp > old_timestamp or config_hash != current.get('config_hash'):
+                    to_update[namespace][schedule_name] = deployed
             else:
-                to_create[namespace][schedule_name] = schedule
+                to_create[namespace][schedule_name] = deployed
 
     @activity.defn(name='create_schedule_config')
     def create_schedule_config(self, input_data: dict[str, Any]) -> dict[str, Any]:
@@ -406,7 +513,12 @@ class Formatters(SientiaMonitoring):
 
         Compares the current schedule configuration in Temporal with the new schedule
         configuration to determine which schedules need to be created, updated, or deleted.
-        Uses timestamp comparison to identify schedules that have changed.
+        Uses the updated_at timestamps and the config_hash (compare_config_timestamps) to
+        identify schedules that have changed.
+        Also sends a SIMPLE_METRICS_FREQUENCY_ALERT_BELOW_FLOOR WARNING for laborious schedules
+        whose frequency_alert is below 1m (the value is kept) or unparseable (it is ignored),
+        and a DRIFT_CHUNK_PERIOD_INVALID WARNING for drift schedules whose configured
+        chunk_period is not supported by laborious (replaced by 'min').
 
         Args:
             input_data (dict[str, Any]): The input data containing:
@@ -447,6 +559,39 @@ class Formatters(SientiaMonitoring):
                 schedules, current_schedules, to_update, to_create, namespace, metadata
             )
 
+        # Fires whenever the schedule is created or updated (each `updated_at` or
+        # config_hash change, including escalation flips).
+        changed = {**to_create[self.laborious_namespace], **to_update[self.laborious_namespace]}
+        for schedule_name, schedule in changed.items():
+            warning = frequency_alert_warning(schedule.get('frequency_alert'))
+            if warning:
+                self.send_notification(
+                    metadata={
+                        **metadata,
+                        'schedule_name': schedule_name,
+                        'model_id': schedule.get('model_id'),
+                        'model_name': schedule.get('model_name'),
+                    },
+                    notification_id='SIMPLE_METRICS_FREQUENCY_ALERT_BELOW_FLOOR',
+                    message=warning,
+                    block='create_schedule_config',
+                    level=NotificationLevel.WARNING,
+                )
+            warning = chunk_period_warning(schedule.get('chunk_period_configured'))
+            if warning:
+                self.send_notification(
+                    metadata={
+                        **metadata,
+                        'schedule_name': schedule_name,
+                        'model_id': schedule.get('model_id'),
+                        'model_name': schedule.get('model_name'),
+                    },
+                    notification_id='DRIFT_CHUNK_PERIOD_INVALID',
+                    message=warning,
+                    block='create_schedule_config',
+                    level=NotificationLevel.WARNING,
+                )
+
         for namespace, schedules in current_schedule_config.items():
             for schedule_name in schedules:
                 if schedule_name not in schedule_config[namespace]:

orchestrator/activities/mongo_db.py

@@ -193,7 +193,8 @@ class MongoDB(SientiaMonitoring):
 
             for document in aggregated_documents:
                 for timestamp_field in timestamp_fields:
-                    if timestamp_field in document:
+                    # Only BSON dates: a field already stored as text is left as it is
+                    if isinstance(document.get(timestamp_field), datetime):
                         document[timestamp_field] = (
                             document[timestamp_field]
                             .replace(tzinfo=UTC)
@@ -223,12 +224,14 @@ class MongoDB(SientiaMonitoring):
         """
         Update `updated_at` timestamps for successfully updated pipelines.
 
-        Updates the `updated_at` field in the `orchestrated_schedules` collection
-        for all pipelines that were successfully updated in Temporal.
+        Updates the `updated_at` and `config_hash` fields in the `orchestrated_schedules`
+        collection for all pipelines that were successfully updated in Temporal, so the
+        next run compares against what was deployed.
 
         Args:
             input_data (dict[str, Any]): Input data containing:
-                - updated_pipelines (list[dict]): Pipelines with success flags to consider
+                - updated_pipelines (list[dict]): Pipelines with success flags (and
+                  config_hash) to consider
                 - metadata (dict): Metadata for logging purposes
         """
         updated_pipelines = input_data.get('updated_pipelines', [])
@@ -237,29 +240,53 @@ class MongoDB(SientiaMonitoring):
 
         self.info('Updating pipelines timestamps...', metadata=metadata)
 
-        argument = [
-            {'schedule_name': pipeline['schedule_name'], 'namespace': pipeline['namespace']}
-            for pipeline in updated_pipelines
-            if pipeline['success']
-        ]
-        data_filter = {'$or': argument} if argument else {}
+        succeeded = [pipeline for pipeline in updated_pipelines if pipeline['success']]
+        if not succeeded:
+            # Every update failed: stamp nothing so they are retried next tick (same guard as
+            # create_pipelines_timestamps).
+            self.info('No successful pipeline updates to timestamp', metadata=metadata)
+            return
 
-        try:
-            self.mongo_db_repository.update_many(
-                'orchestrated_schedules', data_filter, {'$set': {'updated_at': date_now}}, metadata
-            )
-        except Exception as e:
-            trace = traceback.format_exc()
+        # One update per schedule (each has its own config_hash). A failure does not stop
+        # the others: they are all tried, and the failures are notified together.
+        failures: dict[str, str] = {}
+        traces: list[str] = []
+        for pipeline in succeeded:
+            schedule = f'{pipeline["namespace"]}/{pipeline["schedule_name"]}'
+            try:
+                self.mongo_db_repository.update_one(
+                    'orchestrated_schedules',
+                    {
+                        'schedule_name': pipeline['schedule_name'],
+                        'namespace': pipeline['namespace'],
+                    },
+                    {
+                        '$set': {
+                            'updated_at': date_now,
+                            'config_hash': pipeline.get('config_hash'),
+                        }
+                    },
+                    metadata,
+                )
+            except Exception as e:
+                failures[schedule] = str(e)
+                traces.append(f'{schedule}\n{traceback.format_exc()}')
+
+        if failures:
+            trace = '\n'.join(traces)
+            details = '; '.join(f'{schedule}: {error}' for schedule, error in failures.items())
             self.send_notification(
                 metadata=metadata,
                 notification_id='MONGODB_UPDATE_PIPELINES_ERROR',
-                message=f'Failed to update pipelines timestamps: {e}',
+                message=f'Failed to update pipelines timestamps: {details}',
                 block='update_pipelines_timestamps',
                 level=NotificationLevel.ERROR,
                 attachment_content=trace,
             )
             self.error(trace, metadata=metadata)
-            raise e
+            raise RuntimeError(
+                f'Failed to update {len(failures)} of {len(succeeded)} pipelines timestamps'
+            )
 
         self.info(
             f'Updated {len(updated_pipelines)} pipelines timestamps',
@@ -272,11 +299,13 @@ class MongoDB(SientiaMonitoring):
         Insert `updated_at` timestamps for newly created pipelines.
 
         Inserts new documents into the `orchestrated_schedules` collection
-        for all pipelines that were successfully created in Temporal.
+        for all pipelines that were successfully created in Temporal, with the
+        `config_hash` they were created with.
 
         Args:
             input_data (dict[str, Any]): Input data containing:
-                - created_pipelines (list[dict]): Pipelines with success flags to consider
+                - created_pipelines (list[dict]): Pipelines with success flags (and
+                  config_hash) to consider
                 - metadata (dict): Metadata for logging purposes
         """
         created_pipelines = input_data.get('created_pipelines', [])
@@ -291,6 +320,7 @@ class MongoDB(SientiaMonitoring):
                 'schedule_name': pipeline['schedule_name'],
                 'namespace': pipeline['namespace'],
                 'updated_at': date_now,
+                'config_hash': pipeline.get('config_hash'),
             }
             for pipeline in created_pipelines
             if pipeline['success']
@@ -343,7 +373,13 @@ class MongoDB(SientiaMonitoring):
             for pipeline in deleted_pipelines
             if pipeline['success']
         ]
-        data_filter = {'$or': argument} if argument else {}
+        if not argument:
+            # Every delete failed: remove nothing (same guard as create_pipelines_timestamps).
+            # An empty filter would wipe orchestrated_schedules and make the next tick delete
+            # and recreate every orchestrated schedule.
+            self.info('No successful pipeline deletions to untrack', metadata=metadata)
+            return
+        data_filter = {'$or': argument}
 
         try:
             self.mongo_db_repository.delete_many('orchestrated_schedules', data_filter, metadata)

orchestrator/activities/temporal_manager.py

@@ -177,7 +177,9 @@ class TemporalManager(SientiaMonitoring):
                 - schedules (dict[str, Any]): The schedules to create.
 
         Returns:
-            - list[dict[str, Any]]: Report entries for each attempted schedule creation.
+            - list[dict[str, Any]]: Report entries for each attempted schedule creation,
+              with the schedule's config_hash (create_schedule_config) for
+              create_pipelines_timestamps.
         """
 
         schedules_to_create = input_data['schedules']
@@ -198,6 +200,8 @@ class TemporalManager(SientiaMonitoring):
                 )
 
             for schedule_name, schedule in schedules.items():
+                # Stored in orchestrated_schedules, not sent to the workflow
+                config_hash = schedule.pop('config_hash', None)
                 search_attributes = TypedSearchAttributes(
                     [
                         SearchAttributePair(key=self.model_id_id_key, value=schedule['model_id']),
@@ -220,6 +224,9 @@ class TemporalManager(SientiaMonitoring):
                     task_queue_name = self._build_task_queue_name(workflow_type, schedule)
                     schedule['task_queue'] = task_queue_name
 
+                    # No `policy`: schedules rely on Temporal's default SKIP overlap. The
+                    # laborious `DriftAnalysis` cache assumes at most one running drift
+                    # workflow per model; revisit its locking if a policy is ever added here.
                     await client.create_schedule(
                         schedule_name,
                         Schedule(
@@ -255,6 +262,7 @@ class TemporalManager(SientiaMonitoring):
                             'schedule_name': schedule_name,
                             'success': True,
                             'message': 'Schedule created successfully',
+                            'config_hash': config_hash,
                         }
                     )
 
@@ -269,6 +277,7 @@ class TemporalManager(SientiaMonitoring):
                             'schedule_name': schedule_name,
                             'success': False,
                             'message': str(e),
+                            'config_hash': config_hash,
                         }
                     )
 
@@ -357,7 +366,9 @@ class TemporalManager(SientiaMonitoring):
                 - schedules (dict[str, Any]): The schedules to update.
 
         Returns:
-            - list[dict[str, Any]]: Report entries for each attempted schedule update.
+            - list[dict[str, Any]]: Report entries for each attempted schedule update,
+              with the schedule's config_hash (create_schedule_config) for
+              update_pipelines_timestamps.
         """
 
         schedules_to_update = input_data['schedules']
@@ -377,6 +388,8 @@ class TemporalManager(SientiaMonitoring):
                 )
 
             for schedule_name, schedule in schedules.items():
+                # Stored in orchestrated_schedules, not sent to the workflow
+                config_hash = schedule.pop('config_hash', None)
                 try:
                     await self._update_single_schedule(client, schedule_name, schedule, metadata)
 
@@ -386,6 +399,7 @@ class TemporalManager(SientiaMonitoring):
                             'schedule_name': schedule_name,
                             'success': True,
                             'message': 'Schedule updated successfully',
+                            'config_hash': config_hash,
                         }
                     )
 
@@ -400,6 +414,7 @@ class TemporalManager(SientiaMonitoring):
                             'schedule_name': schedule_name,
                             'success': False,
                             'message': str(e),
+                            'config_hash': config_hash,
                         }
                     )
 

orchestrator/utils/opc_slots.py

@@ -0,0 +1,301 @@
+"""OPC slot v2 (contract C1): read tags grouped by server and address for the gateway."""
+
+from typing import Any, cast
+
+from orchestrator.utils.tag_database import (
+    client_certificate_id,
+    is_complete,
+    resolve_certificate,
+    resolve_tag_names,
+    tag_id_of,
+)
+
+
+def _connector_server(connector: dict[str, Any], issues: dict[str, set[str]]) -> dict[str, Any]:
+    """
+    Slot server fields of a tag database connector (connectors collection).
+
+    A connector whose client_certificate_id has no active client certificate still goes
+    into the slot, without a certificate, and is recorded in issues['certificates'] (D5).
+    """
+    certificate = resolve_certificate(connector)
+    if certificate is None and client_certificate_id(connector) is not None:
+        issues['certificates'].add(str(connector['connector_id']))
+    return {
+        'server_id': str(connector['connector_id']),
+        'name': connector['name'],
+        'url': connector['url'],
+        'subscription_period_ms': connector.get('subscription_period_ms'),
+        'security': {
+            'mode': connector['security']['mode'],
+            'policy': connector['security']['policy'],
+        },
+        'certificate': certificate,
+    }
+
+
+def _frequency(tag: dict[str, Any]) -> int | None:
+    """The read tag's frequency as int, or None when it is missing or not an integer."""
+    try:
+        return int(tag['frequency'])
+    except (KeyError, TypeError, ValueError):
+        return None
+
+
+def _locate_read_tag(
+    tag_id: int,
+    connectors: dict[str, dict[str, Any]],
+    tags_by_id: dict[int, dict[str, Any]],
+    issues: dict[str, set[str]],
+) -> tuple[dict[str, Any], str] | None:
+    """
+    Server and address of a read tag, or None when they cannot be resolved.
+
+    The address and connector come from connector_tags and the server from connectors
+    (RF3). What cannot be resolved is recorded in issues ('tags' or 'connectors'), and a
+    server without its registered certificate in issues['certificates'].
+    """
+    connector_tag = tags_by_id.get(tag_id)
+    if connector_tag is None:
+        issues['tags'].add(str(tag_id))
+        return None
+    connector_id = str(connector_tag['connector_id'])
+    connector = connectors.get(connector_id)
+    if connector is None:
+        issues['connectors'].add(connector_id)
+        return None
+    return _connector_server(connector, issues), connector_tag['address']
+
+
+def _cycle_multiple(active_read: Any) -> int | None:
+    """The cycle_multiple of an enabled active_read as int (default 1), or None when invalid."""
+    try:
+        return int(active_read.get('cycle_multiple', 1))
+    except (AttributeError, TypeError, ValueError):
+        return None
+
+
+def _invalid_read_tag(tag: dict[str, Any]) -> bool:
+    """
+    A read tag the slot cannot use: no integer frequency, or an enabled active_read
+    without an integer cycle_multiple.
+    """
+    if _frequency(tag) is None:
+        return True
+    active_read = tag.get('active_read')
+    return (
+        isinstance(active_read, dict)
+        and bool(active_read.get('enabled'))
+        and _cycle_multiple(active_read) is None
+    )
+
+
+def _read_tag_ref(tag: dict[str, Any]) -> Any:
+    """How an invalid read tag is reported: its tag_id, or its tag_name without one."""
+    return tag.get('tag_id') if tag.get('tag_id') not in (None, '') else tag.get('tag_name')
+
+
+def _is_slot_pipeline(pipeline: dict[str, Any]) -> bool:
+    """A scouter pipeline whose read tags can be named in the slot (schedule, model id and name)."""
+    model = pipeline.get('model')
+    return (
+        is_complete(pipeline, ('schedule_name', 'model_id'))
+        and isinstance(model, dict)
+        and is_complete(model, ('name',))
+    )
+
+
+def _merge_active_read(current: dict[str, Any], requested: dict[str, Any] | None) -> dict[str, Any]:
+    """Active read of an address: on if any pipeline asks, with the lowest cycle_multiple (D3)."""
+    if not isinstance(requested, dict) or not requested.get('enabled'):
+        return current
+    cycle_multiple = cast(int, _cycle_multiple(requested))
+    if current.get('enabled'):
+        cycle_multiple = min(cycle_multiple, current['cycle_multiple'])
+    return {'enabled': True, 'cycle_multiple': cycle_multiple}
+
+
+def _assign_server_keys(servers: dict[str, dict[str, Any]]) -> None:
+    """
+    Give each server a unique slot key: its name, or name:server_id (extended until
+    free) when another server already took it. Assigned in server_id order, so the keys
+    are the same on every run and in every slot.
+    """
+    taken: set[str] = set()
+    for server_id in sorted(servers):
+        server = servers[server_id]
+        key = server['name']
+        while key in taken:
+            key = f'{key}:{server_id}'
+        server['key'] = key
+        taken.add(key)
+
+
+def _add_read_tag(
+    entries: dict[tuple[str, str], dict[str, Any]],
+    servers: dict[str, dict[str, Any]],
+    pipeline: dict[str, Any],
+    tag: dict[str, Any],
+    server: dict[str, Any],
+    address: str,
+) -> None:
+    """Merge one resolved read tag into its (server, address) entry and its server."""
+    server_id = server['server_id']
+    servers[server_id] = server
+
+    # int, so the slot does not depend on how each pipeline stored the number
+    frequency = cast(int, _frequency(tag))
+    entry = entries.setdefault(
+        (server_id, address),
+        {
+            'address': address,
+            'frequency': frequency,
+            'active_read': {'enabled': False},
+            'names': [],
+        },
+    )
+    entry['frequency'] = min(entry['frequency'], frequency)
+    entry['active_read'] = _merge_active_read(entry['active_read'], tag.get('active_read'))
+
+    name = {
+        'collection': f'raw_{pipeline["schedule_name"]}',
+        'tag_name': tag['tag_name'],
+        'model_id': pipeline['model_id'],
+        'model_name': pipeline['model']['name'],
+    }
+    if name not in entry['names']:
+        entry['names'].append(name)
+
+
+def _add_pipeline_read_tags(
+    entries: dict[tuple[str, str], dict[str, Any]],
+    servers: dict[str, dict[str, Any]],
+    issues: dict[str, set[str]],
+    pipeline: dict[str, Any],
+    connectors: dict[str, dict[str, Any]],
+    tags_by_id: dict[int, dict[str, Any]],
+) -> None:
+    """Add the read tags of one slot pipeline to entries, reporting the ones left out in issues."""
+    for tag in resolve_tag_names(pipeline.get('read_tags') or [], tags_by_id):
+        tag_id = tag_id_of(tag)
+        if tag_id is None or _invalid_read_tag(tag):
+            issues['invalid'].add(f'{pipeline["schedule_name"]}/{_read_tag_ref(tag)}')
+            continue
+
+        located = _locate_read_tag(tag_id, connectors, tags_by_id, issues)
+        if located is None:
+            continue
+        _add_read_tag(entries, servers, pipeline, tag, *located)
+
+
+def gather_read_tags(
+    pipelines: list[dict[str, Any]],
+    connectors: dict[str, dict[str, Any]],
+    tags_by_id: dict[int, dict[str, Any]],
+) -> tuple[list[dict[str, Any]], dict[str, set[str]]]:
+    """
+    Group the read tags of all scouter pipelines by (server, address) for the slot v2.
+
+    An address read by N pipelines becomes ONE entry with N items in names[]: the
+    gateway reads it once and writes it to N collections (SIENTIAPDE-1974, RF2). Each
+    read tag is resolved through the tag database by its tag_id (RF3): connectors is the
+    only source of the OPC servers.
+
+    Args:
+        pipelines (list[dict[str, Any]]): Pipeline configurations. Only workflow_type
+            'scouter' is read; each needs schedule_name, model_id, model.name and
+            read_tags (with tag_id and frequency, and optionally active_read)
+        connectors (dict[str, dict[str, Any]]): Active OPC UA connectors indexed by
+            index_connectors
+        tags_by_id (dict[int, dict[str, Any]]): Active Read connector_tags indexed by
+            index_connector_tags
+
+    Returns:
+        tuple[list[dict[str, Any]], dict[str, set[str]]]:
+            - One entry per (server, address), sorted by server_id then address, with:
+                - server (dict): server_id, name, url, subscription_period_ms (None =
+                  derive from the frequencies), security ({mode, policy}), certificate
+                  (resolve_certificate; None = connect without security) and key (the
+                  server's unique slot key)
+                - address (str): The OPC node address
+                - frequency (int): The lowest frequency among the pipelines (RF6)
+                - active_read (dict): {'enabled': False} unless a pipeline enables it (D3)
+                - names (list[dict]): collection ('raw_{schedule_name}'), tag_name,
+                  model_id and model_name, one per pipeline name, sorted
+            - What was left out: 'tags' (tag_id), 'connectors' (connector_id) and
+              'invalid' (schedule_name/read tag without an int tag_id, a usable frequency or
+              cycle_multiple, or the schedule_name of a pipeline without schedule_name,
+              model_id or model.name); and 'certificates': connector_ids that went into the
+              slot without the certificate their client_certificate_id points to
+    """
+    issues: dict[str, set[str]] = {
+        'tags': set(),
+        'connectors': set(),
+        'invalid': set(),
+        'certificates': set(),
+    }
+    servers: dict[str, dict[str, Any]] = {}
+    entries: dict[tuple[str, str], dict[str, Any]] = {}
+
+    for pipeline in pipelines:
+        if pipeline.get('workflow_type') != 'scouter':
+            continue
+        if not _is_slot_pipeline(pipeline):
+            issues['invalid'].add(str(pipeline.get('schedule_name')))
+            continue
+        _add_pipeline_read_tags(entries, servers, issues, pipeline, connectors, tags_by_id)
+
+    _assign_server_keys(servers)
+
+    gathered = []
+    for (server_id, _address), entry in sorted(entries.items()):
+        # A stable order keeps the slot identical between runs, so the gateway does
+        # not re-subscribe an unchanged slot (O6)
+        entry['names'].sort(key=lambda n: (n['collection'], n['tag_name'], str(n['model_id'])))
+        gathered.append({'server': servers[server_id], **entry})
+
+    return gathered, issues
+
+
+def build_tag_config(entries: list[dict[str, Any]]) -> dict[str, Any]:
+    """
+    Build one slot in the slot v2 format (contract C1) from address entries. Each server
+    carries its security and the MinIO objects of its client certificate, which the
+    gateway downloads (SIENTIAPDE-1974); certificate None means no security (D5).
+
+    Args:
+        entries (list[dict[str, Any]]): Address entries from gather_read_tags for this slot
+
+    Returns:
+        dict[str, Any]: The slot keyed by server key, each server with server_id, name,
+            url, subscription_period_ms, security, certificate and tags. Each tag (keyed
+            by address) has frequency, active_read and names. A connector without a
+            subscription_period_ms gets half its lowest tag frequency in this slot.
+    """
+    slot_config: dict[str, Any] = {}
+
+    for entry in entries:
+        server = entry['server']
+        if server['key'] not in slot_config:
+            slot_config[server['key']] = {
+                'server_id': server['server_id'],
+                'name': server['name'],
+                'url': server['url'],
+                'subscription_period_ms': server['subscription_period_ms'],
+                'security': server['security'],
+                'certificate': server['certificate'],
+                'tags': {},
+            }
+
+        slot_config[server['key']]['tags'][entry['address']] = {
+            'frequency': entry['frequency'],
+            'active_read': entry['active_read'],
+            'names': entry['names'],
+        }
+
+    for server_config in slot_config.values():
+        if server_config['subscription_period_ms'] is None:
+            frequencies = [tag['frequency'] for tag in server_config['tags'].values()]
+            server_config['subscription_period_ms'] = min(frequencies) / 2
+
+    return slot_config

orchestrator/utils/orchestrator_functions.py

@@ -1,807 +0,0 @@
-from typing import Any, cast
-
-from orchestrator.utils.converters import parse_frequency
-
-
-def common_config(config: dict[str, Any]):
-    """
-    Extract common configuration parameters from a pipeline configuration.
-
-    Args:
-        config (dict[str, Any]): Pipeline configuration containing:
-            - workflow_type (str): Type of workflow (e.g., 'scouter', 'predictions_batch', 'drift')
-            - schedule_name (str): Unique name identifier for this schedule
-            - model_id (str): MongoDB ID of the associated model
-            - model (dict): Model configuration containing:
-                - name (str): Human-readable name of the model
-                - model_config (dict, optional): Additional model-specific configuration
-            - frequency (str, optional): Execution frequency (default: '1m')
-                Format: '{number}{unit}' where unit is 's', 'm', 'h', or 'd'
-            - offset (str, optional): Schedule offset/delay (default: '0m')
-            - max_retry_policy (int, optional): Maximum retry attempts on failure (default: 1)
-            - execution_timeout_seconds (int, optional): Workflow execution timeout (default: 300)
-            - task_timeout_seconds (int, optional): Individual task timeout (default: 300)
-
-    Returns:
-        dict[str, Any]: Common configuration dictionary with standardized parameters
-            for Temporal workflow execution
-    """
-    model = config['model']
-    return {
-        'workflow_type': config['workflow_type'],
-        'schedule_name': config['schedule_name'],
-        'frequency': config.get('frequency', '1m'),
-        'offset': config.get('offset', '0m'),
-        'max_retry_policy': config.get('max_retry_policy', 1),
-        'model_id': config['model_id'],
-        'model_name': model['name'],
-        'model_config': model.get('model_config', {}),
-        'execution_timeout_seconds': config.get('execution_timeout_seconds', 300),
-        'task_timeout_seconds': config.get('task_timeout_seconds', 300),
-        'on_conflict': config.get('on_conflict', 'error'),
-        'runtime': config.get('runtime', 'legacy'),
-    }
-
-
-def drift(config: dict[str, Any]):
-    """
-    Build drift configuration from pipeline config.
-
-    Creates a drift detection workflow configuration with database table mappings
-    and drift metric specifications for monitoring data distribution changes.
-
-    Args:
-        config (dict[str, Any]): Pipeline configuration containing:
-            - interval_minutes (int, optional): Time window for data comparison in minutes (default: 60)
-            - drift_metrics (list[str], optional): Statistical metrics to compute (default:
-                ['kolmogorov_smirnov', 'jensen_shannon', 'wasserstein'])
-            - Additional fields from common_config
-
-    Returns:
-        dict[str, Any]: Drift detection configuration with source/target tables,
-            time interval, and metrics specifications. Results stored in 'drift_metrics' table
-    """
-    return {
-        **common_config(config),
-        'schema': 'sientia_data',
-        'source_table_name': 'laborious_data',
-        'target_table_name': 'drift_metrics',
-        'interval': config.get('interval_minutes', 60),
-        'drift_metrics': config.get(
-            'drift_metrics', ['kolmogorov_smirnov', 'jensen_shannon', 'wasserstein']
-        ),
-    }
-
-
-def simple_metrics(config: dict[str, Any]):
-    """
-    Build simple metrics configuration from pipeline config.
-
-    Creates a simple metrics computation workflow configuration for calculating
-    model performance metrics like RMSE, MSE, MAE, and R².
-
-    Args:
-        config (dict[str, Any]): Pipeline configuration containing:
-            - interval_minutes (int, optional): Computation interval in minutes (default: 60)
-            - metrics (list[str], optional): List of metrics to compute
-                (default: ['rmse', 'mse', 'mae', 'r2'])
-            - Additional fields from common_config
-
-    Returns:
-        dict[str, Any]: Simple metrics configuration with source tables (predictions and
-            actual data), target table, time interval, and metrics list. Results stored in
-            'simple_metrics' table
-    """
-    return {
-        **common_config(config),
-        'schema': 'sientia_data',
-        'predictions_table_name': 'predictions',
-        'data_table_name': 'laborious_data',
-        'target_table_name': 'simple_metrics',
-        'interval_minutes': config.get('interval_minutes', 60),
-        'metrics': config.get('metrics', ['rmse', 'mse', 'mae', 'r2']),
-    }
-
-
-def minimal_retrain(config: dict[str, Any]):
-    """
-    Build minimal retrain configuration from pipeline config.
-
-    Args:
-        config (dict[str, Any]): Pipeline configuration containing:
-            - query (str): SQL query to retrieve training data. Should return features
-                and target variable in expected format
-            - datetime_columns (list[str], optional): Column names to parse as datetime
-                for proper temporal handling (default: [])
-            - Additional fields from common_config
-
-    Returns:
-        dict[str, Any]: Minimal retrain configuration with SQL query, database settings,
-            and datetime column specifications. Retraining logs are stored in 'log_retrain' table
-    """
-    return {
-        **common_config(config),
-        'query': config['query'],
-        'schema': 'sientia_data',
-        'table_name': 'log_retrain',
-        'datetime_columns': config.get('datetime_columns', []),
-    }
-
-
-def base_scouter(config: dict[str, Any]):
-    """
-    Build base scouter configuration shared by all scouter workflow types.
-
-    Creates the foundational configuration for OPC data collection workflows,
-    including filter policies, database settings, and data retention parameters.
-    This configuration is extended by specific scouter implementations (OPC UA, PI Web API).
-
-    Args:
-        config (dict[str, Any]): Pipeline configuration containing:
-            - filters (list[dict], optional): List of filter configurations with:
-                - filter_name (str): Name of the filter
-                - policy (str): Filter policy to apply
-            - tag_retention_minutes (int, optional): Tag retention time in minutes (default: 60)
-            - debug_data_package (bool, optional): Enable debug data package logging (default: False)
-            - fill_missing_tags (bool, optional): Fill missing tags with interpolation (default: False)
-            - Additional fields from common_config
-
-    Returns:
-        dict[str, Any]: Base scouter configuration with filters, database settings,
-            and retention policies
-    """
-
-    filters = {}
-    for f in config.get('filters', []):
-        filters[f['filter_name']] = {'policy': f['policy']}
-
-    return {
-        **common_config(config),
-        'trigger_laborious': False,
-        'filters': filters,
-        'schema': 'sientia_data',
-        'table_name': 'laborious_data',
-        'retention_time': config.get('tag_retention_minutes', 60) * 60,
-        'debug_data_package': config.get('debug_data_package', False),
-        'fill_missing_tags': config.get('fill_missing_tags', False),
-    }
-
-
-def scouter(config: dict[str, Any]):
-    """
-    Build scouter configuration from pipeline config.
-
-    Args:
-        config (dict[str, Any]): Pipeline configuration containing:
-            - schedule_name (str): Name of the schedule (used for topic generation)
-            - read_tags (list[dict]): List of tag configurations, already passed through
-              resolve_tag_names (process_schedules does it), with:
-                - tag_name (str): Name of the tag to read. A tag without one (a tag_id the
-                  tag database cannot resolve) is left out; process_slots notifies it
-                - aggr_func (str, optional): Aggregation function for data collection (default: 'lts')
-                    Common values: 'lts' (last), 'avg' (average), 'min', 'max', 'sum'
-                - data_range (list[int], optional): Valid data range [min, max] (default: [-100, 100])
-            - Additional fields from base_scouter
-
-    Returns:
-        dict[str, Any]: OPC UA scouter configuration with Kafka topic, tag mappings,
-            filters, and retention settings. Topic name follows pattern: 'raw_{schedule_name}'
-    """
-
-    tags = {}
-    for tag in config['read_tags']:
-        if not tag.get('tag_name'):
-            continue
-        tags[tag['tag_name']] = {
-            'aggr_func': tag.get('aggr_func', 'lts'),
-            'data_range': tag.get('data_range', [-100, 100]),
-        }
-
-    return {
-        **base_scouter(config),
-        'topic': f'raw_{config["schedule_name"]}',
-        'model_tags': tags,
-    }
-
-
-def pi_web_api_scouter(config: dict[str, Any]):
-    """
-    Build PI Web API scouter configuration from pipeline config.
-
-    Creates a data collection workflow configuration for OSIsoft PI servers using
-    the PI Web API REST interface. Configures tag mappings with WebIDs, aggregation
-    functions, and API query parameters including timeout management.
-
-    Args:
-        config (dict[str, Any]): Pipeline configuration containing:
-            - read_tags (list[dict]): List of tag configurations with:
-                - tag_name (str): Name of the tag
-                - webid (str): PI Web API WebID for the tag
-                - aggr_func (str, optional): Aggregation function (default: 'lts')
-                - data_range (list[int], optional): Valid data range (default: [-100, 100])
-            - pi_web_api_config (dict): PI Web API connection settings with:
-                - endpoint (str): PI Web API endpoint URL
-                - period (str, optional): Time period for data retrieval (default: '*-1d')
-                - max_count (int, optional): Maximum number of values to retrieve (default: 1)
-                - api_timeout (int, optional): API request timeout in seconds
-            - Additional fields from base_scouter
-
-    Returns:
-        dict[str, Any]: PI Web API scouter configuration with tag mappings and query settings.
-            API timeout is automatically adjusted to not exceed workflow frequency.
-    """
-    tags = {}
-    for tag, tag_config in config['read_tags'].items():
-        tags[tag] = {
-            'webid': tag_config['webid'],
-            'aggr_func': tag_config.get('aggr_func', 'lts'),
-            'data_range': tag_config.get('data_range', [-100, 100]),
-        }
-
-    base_config = base_scouter(config)
-
-    pi_web_api_config = config['pi_web_api_config']
-
-    config_timeout = pi_web_api_config.get('api_timeout', None)
-    frequency = parse_frequency(base_config['frequency'])
-
-    if config_timeout is None or config_timeout > frequency:
-        config_timeout = frequency
-
-    return {
-        **base_config,
-        'model_tags': tags,
-        'pi_web_api_query': {
-            'endpoint': pi_web_api_config['endpoint'],
-            'period': pi_web_api_config.get('period', '*-1d'),
-            'max_count': pi_web_api_config.get('max_count', 1),
-            'api_timeout': config_timeout,
-        },
-    }
-
-
-def overlap_filter_config(base_filter_config: dict[str, Any], config: list[dict[str, Any]]):
-    """
-    Merge filter configurations with base filter configuration.
-
-    Extends the base filter configuration dictionary by adding or overwriting
-    filters from the provided configuration list. Used in predictions_batch
-    workflows to combine default filters with user-defined custom filters.
-
-    Args:
-        base_filter_config (dict[str, Any]): Base filter configuration dictionary to extend.
-            Each filter entry contains 'policy' and optionally 'config' keys.
-        config (list[dict[str, Any]]): List of filter configurations to merge, each containing:
-            - filter_name (str): Name of the filter to add or update
-            - policy (str): Filter policy (e.g., 'STOP', 'CONTINUE', 'REPEAT')
-            - config (dict, optional): Additional filter-specific configuration
-
-    Returns:
-        dict[str, Any]: Extended filter configuration dictionary with merged filters.
-            Filters from config list overwrite or add to base_filter_config entries.
-    """
-    for fil in config:
-        base_filter_config[fil['filter_name']] = {
-            'policy': fil['policy'],
-            'config': fil.get('config', {}),
-        }
-
-    return base_filter_config
-
-
-def process_path_priority(path_priority: list[str]):
-    """
-    Process and normalize path priority list for filter policy execution order.
-
-    Validates and normalizes the priority list used to determine the order in which
-    filter policies are evaluated in prediction workflows. Invalid priorities are
-    removed, missing required priorities are appended, and the list is truncated to
-    exactly 3 elements.
-
-    Valid priorities define workflow behavior when filters are triggered:
-        - STOP: Halt workflow execution immediately
-        - CONTINUE: Proceed to next step despite filter trigger
-        - REPEAT: Retry the current step
-
-    Args:
-        path_priority (list[str]): User-provided list of path priorities. May contain
-            invalid values or be incomplete.
-
-    Returns:
-        list[str]: Normalized path priority list with exactly 3 elements in user-specified
-            or default order. Default order when priorities are missing: ["STOP", "CONTINUE", "REPEAT"]
-    """
-    for priority in path_priority[:]:
-        if priority not in ['STOP', 'CONTINUE', 'REPEAT']:
-            path_priority.remove(priority)
-
-    for priority in ['STOP', 'CONTINUE', 'REPEAT']:
-        if priority not in path_priority:
-            path_priority.append(priority)
-
-    return path_priority[0:3]
-
-
-def predictions_batch(config: dict[str, Any]):
-    """
-    Build predictions batch configuration from pipeline config.
-
-    Args:
-        config (dict[str, Any]): Pipeline configuration containing:
-            - query (str): SQL query to retrieve input data for predictions
-            - write_tags (list[dict]): List of OPC tag configurations for write-back with:
-                - server_id (str): ID of the target OPC server
-                - type (str): Tag type - 'prediction' (model output) or 'confidence' (prediction confidence)
-                - addr (str): OPC tag address/path
-                - data_type (str, optional): OPC data type (default: 'float')
-            - datetime_columns (list[str], optional): Column names to parse as datetime (default: [])
-            - path_priority (list[str], optional): Filter policy execution order (default: ["STOP", "CONTINUE", "REPEAT"])
-            - input_filters (list[dict], optional): Input data validation filters
-            - mlflow_transform_filters (list[dict], optional): Transform stage filters
-            - mlflow_predict_filters (list[dict], optional): Prediction stage filters
-            - model_retention_minutes (int, optional): Data retention time in minutes (default: 60)
-            - save_transform (bool, optional): Save transformed data to database (default: True)
-            - predictions_storage_policy (str, optional): Prediction storage policy (default: 'lts:1')
-            - pi_web_api_output_config (dict, optional): PI Web API output configuration for write-back (default: {})
-            - Additional fields from common_config
-
-    Returns:
-        dict[str, Any]: Complete predictions batch configuration with OPC output mappings,
-            PI Web API output configuration, multi-stage filters, SQL query, and retention policies
-    """
-    tags: dict[str, Any] = {}
-    for tag in config.get('write_tags', []):
-        if tag['server_id'] not in tags:
-            tags[tag['server_id']] = {}
-
-        tag_type = tag['type']
-
-        if tag_type == 'prediction' or tag_type == 'confidence':
-            tag_type_str = f'{tag_type}_tags'
-
-            if tag_type_str not in tags[tag['server_id']]:
-                tags[tag['server_id']][tag_type_str] = {}
-
-            tags[tag['server_id']][tag_type_str][tag['addr']] = {
-                'data_type': tag.get('data_type', 'float'),
-            }
-
-    path_priority = process_path_priority(
-        config.get('path_priority', ['STOP', 'CONTINUE', 'REPEAT'])
-    )
-
-    return {
-        **common_config(config),
-        'query': config['query'],
-        'datetime_columns': config.get('datetime_columns', []),
-        'schema': 'sientia_data',
-        'table_name': 'predictions',
-        'save_transform': config.get('save_transform', True),
-        'transform_table_name': 'transformed_data',
-        'retention_time': config.get('model_retention_minutes', 60) * 60,
-        'opc_output_config': tags,
-        'pi_web_api_output_config': config.get('pi_web_api_output_config', {}),
-        'input_filters': overlap_filter_config(
-            {'EMPTY_DATA': {'policy': 'STOP', 'config': {}}}, config.get('input_filters', [])
-        ),
-        'mlflow_transform_filters': overlap_filter_config(
-            {
-                'EMPTY_DATA': {'policy': 'STOP', 'config': {}},
-                'API_ERROR': {'policy': 'STOP', 'config': {}},
-            },
-            config.get('mlflow_transform_filters', []),
-        ),
-        'mlflow_predict_filters': overlap_filter_config(
-            {'API_ERROR': {'policy': 'STOP', 'config': {}}},
-            config.get('mlflow_predict_filters', []),
-        ),
-        'path_priority': path_priority,
-        'predictions_storage_policy': config.get('predictions_storage_policy', 'lts:1'),
-    }
-
-
-# Fields a tag database document needs to take part in a slot. Documents missing any
-# of them are left out of the index, so their read tags are notified as not found
-# instead of failing the whole run.
-CONNECTOR_TAG_FIELDS = ('tag_id', 'connector_id', 'address', 'tag_name')
-CONNECTOR_FIELDS = ('connector_id', 'name', 'url', 'server_uri')
-
-
-def _complete(document: dict[str, Any], fields: tuple[str, ...]) -> bool:
-    return all(document.get(field) not in (None, '') for field in fields)
-
-
-def _tag_id(tag: dict[str, Any]) -> str | None:
-    """The read tag's tag_id as text, or None when it has none (missing, null or empty)."""
-    tag_id = tag.get('tag_id')
-    return None if tag_id in (None, '') else str(tag_id)
-
-
-def referenced_tag_ids(pipelines: list[dict[str, Any]]) -> list[Any]:
-    """
-    The tag_ids the scouter pipelines reference, for the connector_tags query.
-
-    Each id goes in both forms, int and text, because the tag database stores int and
-    pipelines may hold either. Sorted, so the query is the same on every run.
-
-    Args:
-        pipelines (list[dict[str, Any]]): Pipeline configurations
-
-    Returns:
-        list[Any]: The referenced tag_ids (empty when no read tag carries a tag_id)
-    """
-    ids: set[Any] = set()
-    for pipeline in pipelines:
-        if pipeline.get('workflow_type') != 'scouter':
-            continue
-        for tag in pipeline.get('read_tags') or []:
-            tag_id = _tag_id(tag)
-            if tag_id is None:
-                continue
-            ids.add(tag_id)
-            if tag_id.lstrip('-').isdigit():
-                ids.add(int(tag_id))
-    return sorted(ids, key=lambda value: (isinstance(value, str), str(value)))
-
-
-def filter_by_tag_ids(query: dict[str, Any], tag_ids: list[Any]) -> dict[str, Any]:
-    """
-    Restrict a connector_tags aggregation to the referenced tag_ids.
-
-    Args:
-        query (dict[str, Any]): The connector_tags query ({'collection', 'aggregation'})
-        tag_ids (list[Any]): The ids from referenced_tag_ids
-
-    Returns:
-        dict[str, Any]: A new query whose aggregation starts with a tag_id $match. The
-            aggregation list is new too: aggregate_documents_in_mongodb appends a stage
-            to the list it gets.
-    """
-    return {
-        **query,
-        'aggregation': [{'$match': {'tag_id': {'$in': tag_ids}}}, *query['aggregation']],
-    }
-
-
-def index_connector_tags(connector_tags: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
-    """
-    Index the tag database's connector_tags by tag_id.
-
-    The key is the tag_id as text, because pipelines may store it as int or str.
-    Documents without tag_id, connector_id, address or tag_name are left out.
-
-    Args:
-        connector_tags (list[dict[str, Any]]): connector_tags documents (SIENTIAPDE-1569),
-            each with tag_id, connector_id, address and tag_name
-
-    Returns:
-        dict[str, dict[str, Any]]: connector_tags documents keyed by str(tag_id)
-    """
-    return {
-        str(tag['tag_id']): tag for tag in connector_tags if _complete(tag, CONNECTOR_TAG_FIELDS)
-    }
-
-
-def index_connectors(connectors: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
-    """
-    Index the tag database's connectors by connector_id (as text).
-
-    Documents without connector_id, name, url or server_uri are left out.
-
-    Args:
-        connectors (list[dict[str, Any]]): connectors documents (SIENTIAPDE-1569)
-
-    Returns:
-        dict[str, dict[str, Any]]: connectors documents keyed by str(connector_id)
-    """
-    return {
-        str(connector['connector_id']): connector
-        for connector in connectors
-        if _complete(connector, CONNECTOR_FIELDS)
-    }
-
-
-def resolve_tag_names(
-    read_tags: list[dict[str, Any]], tags_by_id: dict[str, dict[str, Any]]
-) -> list[dict[str, Any]]:
-    """
-    Fill in the tag_name of each read tag of a scouter pipeline.
-
-    The name is the pipeline's own tag_name when it has one, else the connector_tags
-    name of its tag_id. The scouter's model_tags (scouter()) and the slot's names[]
-    (gather_read_tags()) both take the name from here, so the name the gateway writes
-    is always the one the scouter looks for (SIENTIAPDE-1974, SC1).
-
-    Args:
-        read_tags (list[dict[str, Any]]): The pipeline's read_tags
-        tags_by_id (dict[str, dict[str, Any]]): connector_tags indexed by index_connector_tags
-
-    Returns:
-        list[dict[str, Any]]: The read tags, each with tag_name set when it could be
-            resolved. A tag_id missing from tags_by_id leaves the tag without a name.
-    """
-    resolved = []
-    for tag in read_tags:
-        name = tag.get('tag_name')
-        tag_id = _tag_id(tag)
-        if not name and tag_id is not None:
-            name = tags_by_id.get(tag_id, {}).get('tag_name')
-        resolved.append({**tag, 'tag_name': name} if name else tag)
-    return resolved
-
-
-def _connector_server(connector: dict[str, Any]) -> dict[str, Any]:
-    """Slot server fields of a tag database connector (connectors collection)."""
-    return {
-        'server_id': str(connector['connector_id']),
-        'name': connector['name'],
-        'url': connector['url'],
-        'server_uri': connector['server_uri'],
-        'use_security': bool(connector.get('use_security', False)),
-        'subscription_period_ms': connector.get('subscription_period_ms'),
-        'from_tag_database': True,
-    }
-
-
-def _opc_server(server_id: str, opc_server: dict[str, Any]) -> dict[str, Any]:
-    """
-    Slot server fields of a legacy opc-servers document.
-
-    The certificate now always comes from MinIO on the gateway side, so the file paths
-    are not carried: a cert_path only tells that the server uses security (RF4). The
-    subscription period is left to build_tag_config (half the lowest tag frequency).
-    """
-    return {
-        'server_id': server_id,
-        'name': opc_server['server_name'],
-        'url': opc_server['url'],
-        'server_uri': opc_server['uri'],
-        'use_security': bool(opc_server.get('cert_path')),
-        'subscription_period_ms': None,
-        'from_tag_database': False,
-    }
-
-
-def _frequency(tag: dict[str, Any]) -> int | None:
-    """The read tag's frequency as int, or None when it is missing or not an integer."""
-    try:
-        return int(tag['frequency'])
-    except (KeyError, TypeError, ValueError):
-        return None
-
-
-def _locate_read_tag(
-    tag: dict[str, Any],
-    opc_servers: dict[str, dict[str, Any]],
-    connectors: dict[str, dict[str, Any]],
-    tags_by_id: dict[str, dict[str, Any]],
-    issues: dict[str, set[str]],
-) -> tuple[dict[str, Any], str] | None:
-    """
-    Server and address of a read tag, or None when they cannot be resolved.
-
-    A tag with tag_id takes its address and connector from connector_tags and its server
-    from connectors (RF3). A tag without one keeps the current fields: server_id +
-    tag_address, with the server from opc-servers (RF4). What cannot be resolved is
-    recorded in issues ('tags', 'connectors' or 'servers').
-    """
-    tag_id = _tag_id(tag)
-    if tag_id is not None:
-        connector_tag = tags_by_id.get(tag_id)
-        if connector_tag is None:
-            issues['tags'].add(tag_id)
-            return None
-        connector_id = str(connector_tag['connector_id'])
-        connector = connectors.get(connector_id)
-        if connector is None:
-            issues['connectors'].add(connector_id)
-            return None
-        return _connector_server(connector), connector_tag['address']
-
-    server_id = str(tag['server_id'])
-    opc_server = opc_servers.get(server_id)
-    if opc_server is None:
-        issues['servers'].add(server_id)
-        return None
-    return _opc_server(server_id, opc_server), tag['tag_address']
-
-
-def _invalid_read_tag(tag: dict[str, Any]) -> bool:
-    """
-    A read tag the slot cannot use: no integer frequency, or (without tag_id) no
-    server_id, tag_address or tag_name.
-    """
-    if _frequency(tag) is None:
-        return True
-    if _tag_id(tag) is not None:
-        return False
-    return not _complete(tag, ('server_id', 'tag_address', 'tag_name'))
-
-
-def _merge_active_read(current: dict[str, Any], requested: dict[str, Any] | None) -> dict[str, Any]:
-    """Active read of an address: on if any pipeline asks, with the lowest cycle_multiple (D3)."""
-    if not requested or not requested.get('enabled'):
-        return current
-    cycle_multiple = int(requested.get('cycle_multiple', 1))
-    if current.get('enabled'):
-        cycle_multiple = min(cycle_multiple, current['cycle_multiple'])
-    return {'enabled': True, 'cycle_multiple': cycle_multiple}
-
-
-def _assign_server_keys(servers: dict[str, dict[str, Any]]) -> None:
-    """
-    Give each server a unique slot key: its name, or name:server_id (extended until
-    free) when another server already took it. Assigned in server_id order, so the keys
-    are the same on every run and in every slot.
-    """
-    taken: set[str] = set()
-    for server_id in sorted(servers):
-        server = servers[server_id]
-        key = server['name']
-        while key in taken:
-            key = f'{key}:{server_id}'
-        server['key'] = key
-        taken.add(key)
-
-
-def _add_read_tag(
-    entries: dict[tuple[str, str], dict[str, Any]],
-    servers: dict[str, dict[str, Any]],
-    pipeline: dict[str, Any],
-    tag: dict[str, Any],
-    server: dict[str, Any],
-    address: str,
-) -> None:
-    """Merge one resolved read tag into its (server, address) entry and its server."""
-    server_id = server['server_id']
-    known = servers.get(server_id)
-    if known is None or (server['from_tag_database'] and not known['from_tag_database']):
-        servers[server_id] = server
-
-    # int, so the slot does not depend on how each pipeline stored the number
-    frequency = cast(int, _frequency(tag))
-    entry = entries.setdefault(
-        (server_id, address),
-        {
-            'address': address,
-            'frequency': frequency,
-            'active_read': {'enabled': False},
-            'names': [],
-        },
-    )
-    entry['frequency'] = min(entry['frequency'], frequency)
-    entry['active_read'] = _merge_active_read(entry['active_read'], tag.get('active_read'))
-
-    name = {
-        'collection': f'raw_{pipeline["schedule_name"]}',
-        'tag_name': tag['tag_name'],
-        'model_id': pipeline['model_id'],
-        'model_name': pipeline['model']['name'],
-    }
-    if name not in entry['names']:
-        entry['names'].append(name)
-
-
-def gather_read_tags(
-    pipelines: list[dict[str, Any]],
-    opc_servers: dict[str, dict[str, Any]],
-    connectors: dict[str, dict[str, Any]],
-    tags_by_id: dict[str, dict[str, Any]],
-) -> tuple[list[dict[str, Any]], dict[str, set[str]]]:
-    """
-    Group the read tags of all scouter pipelines by (server, address) for the slot v2.
-
-    An address read by N pipelines becomes ONE entry with N items in names[]: the
-    gateway reads it once and writes it to N collections (SIENTIAPDE-1974, RF2). Each
-    read tag is resolved through the tag database when it has a tag_id (RF3), or through
-    the current server_id + tag_address + opc-servers fields when it has not (RF4).
-
-    A server known through both connectors and opc-servers under the same server_id is
-    the same server (the gateway keys its connections by server_id); the connectors
-    fields win.
-
-    Args:
-        pipelines (list[dict[str, Any]]): Pipeline configurations. Only workflow_type
-            'scouter' is read; each needs schedule_name, model_id, model.name and
-            read_tags (with frequency, and optionally tag_id and active_read)
-        opc_servers (dict[str, dict[str, Any]]): opc-servers documents keyed by str(id)
-        connectors (dict[str, dict[str, Any]]): Active OPC UA connectors indexed by
-            index_connectors
-        tags_by_id (dict[str, dict[str, Any]]): Active Read connector_tags indexed by
-            index_connector_tags
-
-    Returns:
-        tuple[list[dict[str, Any]], dict[str, set[str]]]:
-            - One entry per (server, address), sorted by server_id then address, with:
-                - server (dict): server_id, name, url, server_uri, use_security,
-                  subscription_period_ms (None = derive from the frequencies) and key
-                  (the server's unique slot key)
-                - address (str): The OPC node address
-                - frequency (int): The lowest frequency among the pipelines (RF6)
-                - active_read (dict): {'enabled': False} unless a pipeline enables it (D3)
-                - names (list[dict]): collection ('raw_{schedule_name}'), tag_name,
-                  model_id and model_name, one per pipeline name, sorted
-            - What was left out: 'tags' (tag_id), 'connectors' (connector_id), 'servers'
-              (opc-servers server_id) and 'invalid' (schedule_name/read tag without a
-              usable frequency or legacy fields)
-    """
-    issues: dict[str, set[str]] = {
-        'tags': set(),
-        'connectors': set(),
-        'servers': set(),
-        'invalid': set(),
-    }
-    servers: dict[str, dict[str, Any]] = {}
-    entries: dict[tuple[str, str], dict[str, Any]] = {}
-
-    for pipeline in pipelines:
-        if pipeline['workflow_type'] != 'scouter':
-            continue
-
-        for tag in resolve_tag_names(pipeline.get('read_tags') or [], tags_by_id):
-            if _invalid_read_tag(tag):
-                ref = _tag_id(tag) or tag.get('tag_address') or tag.get('tag_name')
-                issues['invalid'].add(f'{pipeline["schedule_name"]}/{ref}')
-                continue
-
-            located = _locate_read_tag(tag, opc_servers, connectors, tags_by_id, issues)
-            if located is None:
-                continue
-            _add_read_tag(entries, servers, pipeline, tag, *located)
-
-    _assign_server_keys(servers)
-
-    gathered = []
-    for (server_id, _address), entry in sorted(entries.items()):
-        # A stable order keeps the slot identical between runs, so the gateway does
-        # not re-subscribe an unchanged slot (O6)
-        entry['names'].sort(key=lambda n: (n['collection'], n['tag_name'], str(n['model_id'])))
-        gathered.append({'server': servers[server_id], **entry})
-
-    return gathered, issues
-
-
-def build_tag_config(entries: list[dict[str, Any]]) -> dict[str, Any]:
-    """
-    Build one slot in the slot v2 format (contract C1) from address entries.
-
-    The slot no longer carries cert_path, private_key_path or server_cert_path: the
-    gateway takes the certificate from MinIO (SIENTIAPDE-1974, O4).
-
-    Args:
-        entries (list[dict[str, Any]]): Address entries from gather_read_tags for this slot
-
-    Returns:
-        dict[str, Any]: The slot keyed by server key, each server with server_id, name,
-            url, server_uri, use_security, subscription_period_ms and tags. Each tag (keyed
-            by address) has frequency, active_read and names. A server without a
-            subscription period of its own (opc-servers) gets half its lowest tag
-            frequency in this slot, as before.
-    """
-    slot_config: dict[str, Any] = {}
-
-    for entry in entries:
-        server = entry['server']
-        if server['key'] not in slot_config:
-            slot_config[server['key']] = {
-                'server_id': server['server_id'],
-                'name': server['name'],
-                'url': server['url'],
-                'server_uri': server['server_uri'],
-                'use_security': server['use_security'],
-                'subscription_period_ms': server['subscription_period_ms'],
-                'tags': {},
-            }
-
-        slot_config[server['key']]['tags'][entry['address']] = {
-            'frequency': entry['frequency'],
-            'active_read': entry['active_read'],
-            'names': entry['names'],
-        }
-
-    for server_config in slot_config.values():
-        if server_config['subscription_period_ms'] is None:
-            frequencies = [tag['frequency'] for tag in server_config['tags'].values()]
-            server_config['subscription_period_ms'] = min(frequencies) / 2
-
-    return slot_config

orchestrator/utils/pipeline_configs.py

@@ -0,0 +1,474 @@
+"""Pipeline configuration builders: one Temporal schedule input per pipeline type."""
+
+from math import ceil
+from typing import Any
+
+from orchestrator.utils.converters import parse_frequency
+
+
+def common_config(config: dict[str, Any]):
+    """
+    Extract common configuration parameters from a pipeline configuration.
+
+    Args:
+        config (dict[str, Any]): Pipeline configuration containing:
+            - workflow_type (str): Type of workflow (e.g., 'scouter', 'predictions_batch', 'drift')
+            - schedule_name (str): Unique name identifier for this schedule
+            - model_id (str): MongoDB ID of the associated model
+            - model (dict): Model configuration containing:
+                - name (str): Human-readable name of the model
+                - model_config (dict, optional): Additional model-specific configuration
+            - frequency (str, optional): Execution frequency (default: '1m')
+                Format: '{number}{unit}' where unit is 's', 'm', 'h', or 'd'
+            - offset (str, optional): Schedule offset/delay (default: '0m')
+            - max_retry_policy (int, optional): Maximum retry attempts on failure (default: 1)
+            - execution_timeout_seconds (int, optional): Workflow execution timeout (default: 300)
+            - task_timeout_seconds (int, optional): Individual task timeout (default: 300)
+
+    Returns:
+        dict[str, Any]: Common configuration dictionary with standardized parameters
+            for Temporal workflow execution
+    """
+    model = config['model']
+    return {
+        'workflow_type': config['workflow_type'],
+        'schedule_name': config['schedule_name'],
+        'frequency': config.get('frequency', '1m'),
+        'offset': config.get('offset', '0m'),
+        'max_retry_policy': config.get('max_retry_policy', 1),
+        'model_id': config['model_id'],
+        'model_name': model['name'],
+        'model_config': model.get('model_config', {}),
+        'execution_timeout_seconds': config.get('execution_timeout_seconds', 300),
+        'task_timeout_seconds': config.get('task_timeout_seconds', 300),
+        'on_conflict': config.get('on_conflict', 'error'),
+        'runtime': config.get('runtime', 'legacy'),
+    }
+
+
+# Kept in sync by hand with the `chunk_period` validation in
+# sientia-dataops-laborious/laborious/activities/model_metrics.py (no shared contract).
+DRIFT_CHUNK_PERIODS = ('min', 's')
+
+
+def _drift_interval_minutes(config: dict[str, Any], frequency: Any) -> int:
+    """
+    Drift window in minutes: `interval_minutes` as configured, otherwise the schedule frequency
+    rounded up to whole minutes (min 1). Consecutive default windows do not overlap, so each
+    drift chunk is stored once.
+    """
+    if 'interval_minutes' in config:
+        return config['interval_minutes']
+    try:
+        seconds = parse_frequency(str(frequency))
+    except ValueError:
+        # create_schedules fails this schedule on its own; don't kill the tick for every model.
+        return 60
+    return max(1, ceil(seconds / 60))
+
+
+def drift(config: dict[str, Any]):
+    """
+    Build drift configuration from pipeline config.
+
+    Creates a drift detection workflow configuration with database table mappings
+    and drift metric specifications for monitoring data distribution changes.
+
+    Args:
+        config (dict[str, Any]): Pipeline configuration containing:
+            - interval_minutes (int, optional): Time window for data comparison in minutes,
+                used as-is. Default: `frequency` rounded up to whole minutes (min 1), so
+                consecutive windows do not overlap (1 with the default '1m');
+                `interval_minutes: 60` keeps the legacy window.
+            - chunk_period (str, optional): Drift bucket size, 'min' or 's' (default: 'min').
+                Any other value falls back to 'min' and create_schedule_config warns.
+            - drift_metrics (list[str], optional): Statistical metrics to compute (default:
+                ['kolmogorov_smirnov', 'jensen_shannon', 'wasserstein'])
+            - Additional fields from common_config
+
+    Returns:
+        dict[str, Any]: Drift detection configuration with source/target tables,
+            time interval, chunk period and metrics specifications. Results stored in
+            'drift_metrics' table
+    """
+    base = common_config(config)
+    configured_chunk_period = config.get('chunk_period')
+    chunk_period_valid = configured_chunk_period in DRIFT_CHUNK_PERIODS
+    output = {
+        **base,
+        'schema': 'sientia_data',
+        'source_table_name': 'laborious_data',
+        'target_table_name': 'drift_metrics',
+        'interval': _drift_interval_minutes(config, base['frequency']),
+        'chunk_period': configured_chunk_period if chunk_period_valid else 'min',
+        'drift_metrics': config.get(
+            'drift_metrics', ['kolmogorov_smirnov', 'jensen_shannon', 'wasserstein']
+        ),
+    }
+    if configured_chunk_period is not None and not chunk_period_valid:
+        # Forwarded so create_schedule_config can warn without re-reading the pipeline.
+        output['chunk_period_configured'] = configured_chunk_period
+    return output
+
+
+def chunk_period_warning(chunk_period: Any) -> str | None:
+    """
+    Warning message when a configured drift `chunk_period` is not supported by laborious,
+    otherwise None. Warn only: `drift()` already replaced it with 'min'.
+    """
+    if chunk_period is None or chunk_period in DRIFT_CHUNK_PERIODS:
+        return None
+    return (
+        f"chunk_period={chunk_period!r} is not one of {DRIFT_CHUNK_PERIODS}; falling back to 'min'"
+    )
+
+
+def simple_metrics(config: dict[str, Any]):
+    """
+    Build simple metrics configuration from pipeline config.
+
+    Creates a simple metrics computation workflow configuration for calculating
+    model performance metrics like RMSE, MSE, MAE, and R².
+
+    Args:
+        config (dict[str, Any]): Pipeline configuration containing:
+            - interval_minutes (int, optional): Computation interval in minutes (default: 60)
+            - metrics (list[str], optional): List of metrics to compute
+                (default: ['rmse', 'mse', 'mae', 'r2'])
+            - frequency_alert (str, optional): Frequency used while `adaptive_state.escalated`
+                is true; same format as `frequency`. Absent = adaptive frequency off.
+            - adaptive_state (dict, optional): `{escalated, healthy_streak}`, written by laborious.
+            - Additional fields from common_config
+
+    Returns:
+        dict[str, Any]: Simple metrics configuration with source tables (predictions and
+            actual data), target table, time interval, and metrics list. Results stored in
+            'simple_metrics' table
+    """
+    base = common_config(config)
+    frequency_alert = config.get('frequency_alert')
+    # SIENTIAPDE-1995: laborious flips `adaptive_state.escalated` on a threshold breach (and back
+    # after the window is renewed without one) and bumps `updated_at`, which makes this tick
+    # rewrite the schedule.
+    state = config.get('adaptive_state')
+    # Hand-written docs: a malformed state must not raise and kill the tick for every model.
+    if frequency_alert and isinstance(state, dict) and state.get('escalated') is True:
+        try:
+            # Temporal parses the schedule frequency on create/update; an unparseable value
+            # would fail it, so keep `frequency` (create_schedule_config still warns).
+            parse_frequency(str(frequency_alert))
+            base['frequency'] = frequency_alert
+        except ValueError:
+            pass
+    return {
+        **base,
+        'schema': 'sientia_data',
+        'predictions_table_name': 'predictions',
+        'data_table_name': 'laborious_data',
+        'target_table_name': 'simple_metrics',
+        'interval_minutes': config.get('interval_minutes', 60),
+        'metrics': config.get('metrics', ['rmse', 'mse', 'mae', 'r2']),
+        # Forwarded so create_schedule_config can warn on a value below the floor.
+        'frequency_alert': frequency_alert,
+    }
+
+
+MIN_FREQUENCY_ALERT_SECONDS = 60
+
+
+def frequency_alert_warning(frequency_alert: str | None) -> str | None:
+    """
+    Warning message when `frequency_alert` is below the 1-minute floor or does not parse,
+    otherwise None. Warn only (SIENTIAPDE-1995 decision 7): a value below the floor is kept;
+    an unparseable one is ignored by `simple_metrics()`, which keeps `frequency`.
+    """
+    if not frequency_alert:
+        return None
+    try:
+        # str(): a non-string value (e.g. a number typed into the doc) must warn, not raise
+        # AttributeError and fail the whole create_schedule_config activity.
+        seconds = parse_frequency(str(frequency_alert))
+    except ValueError:
+        return f'frequency_alert={frequency_alert!r} is not a valid frequency'
+    if seconds < MIN_FREQUENCY_ALERT_SECONDS:
+        return f'frequency_alert={frequency_alert} is below the 1m floor; kept as configured'
+    return None
+
+
+def minimal_retrain(config: dict[str, Any]):
+    """
+    Build minimal retrain configuration from pipeline config.
+
+    Args:
+        config (dict[str, Any]): Pipeline configuration containing:
+            - query (str): SQL query to retrieve training data. Should return features
+                and target variable in expected format
+            - datetime_columns (list[str], optional): Column names to parse as datetime
+                for proper temporal handling (default: [])
+            - Additional fields from common_config
+
+    Returns:
+        dict[str, Any]: Minimal retrain configuration with SQL query, database settings,
+            and datetime column specifications. Retraining logs are stored in 'log_retrain' table
+    """
+    return {
+        **common_config(config),
+        'query': config['query'],
+        'schema': 'sientia_data',
+        'table_name': 'log_retrain',
+        'datetime_columns': config.get('datetime_columns', []),
+    }
+
+
+def base_scouter(config: dict[str, Any]):
+    """
+    Build base scouter configuration shared by all scouter workflow types.
+
+    Creates the foundational configuration for OPC data collection workflows,
+    including filter policies, database settings, and data retention parameters.
+    This configuration is extended by specific scouter implementations (OPC UA, PI Web API).
+
+    Args:
+        config (dict[str, Any]): Pipeline configuration containing:
+            - filters (list[dict], optional): List of filter configurations with:
+                - filter_name (str): Name of the filter
+                - policy (str): Filter policy to apply
+            - tag_retention_minutes (int, optional): Tag retention time in minutes (default: 60)
+            - debug_data_package (bool, optional): Enable debug data package logging (default: False)
+            - fill_missing_tags (bool, optional): Fill missing tags with interpolation (default: False)
+            - Additional fields from common_config
+
+    Returns:
+        dict[str, Any]: Base scouter configuration with filters, database settings,
+            and retention policies
+    """
+
+    filters = {}
+    for f in config.get('filters', []):
+        filters[f['filter_name']] = {'policy': f['policy']}
+
+    return {
+        **common_config(config),
+        'trigger_laborious': False,
+        'filters': filters,
+        'schema': 'sientia_data',
+        'table_name': 'laborious_data',
+        'retention_time': config.get('tag_retention_minutes', 60) * 60,
+        'debug_data_package': config.get('debug_data_package', False),
+        'fill_missing_tags': config.get('fill_missing_tags', False),
+    }
+
+
+def scouter(config: dict[str, Any]):
+    """
+    Build scouter configuration from pipeline config.
+
+    Args:
+        config (dict[str, Any]): Pipeline configuration containing:
+            - schedule_name (str): Name of the schedule (used for topic generation)
+            - read_tags (list[dict]): List of tag configurations, already passed through
+              resolve_tag_names (process_schedules does it), with:
+                - tag_name (str): Name of the tag to read. A tag without one (a tag_id the
+                  tag database cannot resolve) is left out; process_slots notifies it
+                - aggr_func (str, optional): Aggregation function for data collection (default: 'lts')
+                    Common values: 'lts' (last), 'avg' (average), 'min', 'max', 'sum'
+                - data_range (list[int], optional): Valid data range [min, max] (default: [-100, 100])
+            - Additional fields from base_scouter
+
+    Returns:
+        dict[str, Any]: OPC UA scouter configuration with Kafka topic, tag mappings,
+            filters, and retention settings. Topic name follows pattern: 'raw_{schedule_name}'
+    """
+
+    tags = {}
+    for tag in config['read_tags']:
+        if not tag.get('tag_name'):
+            continue
+        tags[tag['tag_name']] = {
+            'aggr_func': tag.get('aggr_func', 'lts'),
+            'data_range': tag.get('data_range', [-100, 100]),
+        }
+
+    return {
+        **base_scouter(config),
+        'topic': f'raw_{config["schedule_name"]}',
+        'model_tags': tags,
+    }
+
+
+def pi_web_api_scouter(config: dict[str, Any]):
+    """
+    Build PI Web API scouter configuration from pipeline config.
+
+    Creates a data collection workflow configuration for OSIsoft PI servers using
+    the PI Web API REST interface. Configures tag mappings with WebIDs, aggregation
+    functions, and API query parameters including timeout management.
+
+    Args:
+        config (dict[str, Any]): Pipeline configuration containing:
+            - read_tags (list[dict]): List of tag configurations with:
+                - tag_name (str): Name of the tag
+                - webid (str): PI Web API WebID for the tag
+                - aggr_func (str, optional): Aggregation function (default: 'lts')
+                - data_range (list[int], optional): Valid data range (default: [-100, 100])
+            - pi_web_api_config (dict): PI Web API connection settings with:
+                - endpoint (str): PI Web API endpoint URL
+                - period (str, optional): Time period for data retrieval (default: '*-1d')
+                - max_count (int, optional): Maximum number of values to retrieve (default: 1)
+                - api_timeout (int, optional): API request timeout in seconds
+            - Additional fields from base_scouter
+
+    Returns:
+        dict[str, Any]: PI Web API scouter configuration with tag mappings and query settings.
+            API timeout is automatically adjusted to not exceed workflow frequency.
+    """
+    tags = {}
+    for tag, tag_config in config['read_tags'].items():
+        tags[tag] = {
+            'webid': tag_config['webid'],
+            'aggr_func': tag_config.get('aggr_func', 'lts'),
+            'data_range': tag_config.get('data_range', [-100, 100]),
+        }
+
+    base_config = base_scouter(config)
+
+    pi_web_api_config = config['pi_web_api_config']
+
+    config_timeout = pi_web_api_config.get('api_timeout', None)
+    frequency = parse_frequency(base_config['frequency'])
+
+    if config_timeout is None or config_timeout > frequency:
+        config_timeout = frequency
+
+    return {
+        **base_config,
+        'model_tags': tags,
+        'pi_web_api_query': {
+            'endpoint': pi_web_api_config['endpoint'],
+            'period': pi_web_api_config.get('period', '*-1d'),
+            'max_count': pi_web_api_config.get('max_count', 1),
+            'api_timeout': config_timeout,
+        },
+    }
+
+
+def overlap_filter_config(base_filter_config: dict[str, Any], config: list[dict[str, Any]]):
+    """
+    Merge filter configurations with base filter configuration.
+
+    Extends the base filter configuration dictionary by adding or overwriting
+    filters from the provided configuration list. Used in predictions_batch
+    workflows to combine default filters with user-defined custom filters.
+
+    Args:
+        base_filter_config (dict[str, Any]): Base filter configuration dictionary to extend.
+            Each filter entry contains 'policy' and optionally 'config' keys.
+        config (list[dict[str, Any]]): List of filter configurations to merge, each containing:
+            - filter_name (str): Name of the filter to add or update
+            - policy (str): Filter policy (e.g., 'STOP', 'CONTINUE', 'REPEAT')
+            - config (dict, optional): Additional filter-specific configuration
+
+    Returns:
+        dict[str, Any]: Extended filter configuration dictionary with merged filters.
+            Filters from config list overwrite or add to base_filter_config entries.
+    """
+    for fil in config:
+        base_filter_config[fil['filter_name']] = {
+            'policy': fil['policy'],
+            'config': fil.get('config', {}),
+        }
+
+    return base_filter_config
+
+
+def process_path_priority(path_priority: list[str]):
+    """
+    Process and normalize path priority list for filter policy execution order.
+
+    Validates and normalizes the priority list used to determine the order in which
+    filter policies are evaluated in prediction workflows. Invalid priorities are
+    removed, missing required priorities are appended, and the list is truncated to
+    exactly 3 elements.
+
+    Valid priorities define workflow behavior when filters are triggered:
+        - STOP: Halt workflow execution immediately
+        - CONTINUE: Proceed to next step despite filter trigger
+        - REPEAT: Retry the current step
+
+    Args:
+        path_priority (list[str]): User-provided list of path priorities. May contain
+            invalid values or be incomplete.
+
+    Returns:
+        list[str]: Normalized path priority list with exactly 3 elements in user-specified
+            or default order. Default order when priorities are missing: ["STOP", "CONTINUE", "REPEAT"]
+    """
+    for priority in path_priority[:]:
+        if priority not in ['STOP', 'CONTINUE', 'REPEAT']:
+            path_priority.remove(priority)
+
+    for priority in ['STOP', 'CONTINUE', 'REPEAT']:
+        if priority not in path_priority:
+            path_priority.append(priority)
+
+    return path_priority[0:3]
+
+
+def predictions_batch(config: dict[str, Any]):
+    """
+    Build predictions batch configuration from pipeline config.
+
+    Args:
+        config (dict[str, Any]): Pipeline configuration containing:
+            - query (str): SQL query to retrieve input data for predictions
+            - opc_output_config (dict, optional): The OPC write-back configuration, built by
+              process_schedules from the pipeline's write_tags ([{tag_id}]) through the tag
+              database (resolve_write_tags): {connector_id: {prediction_tags |
+              confidence_tags: {address: {data_type}}}} (default: {})
+            - datetime_columns (list[str], optional): Column names to parse as datetime (default: [])
+            - path_priority (list[str], optional): Filter policy execution order (default: ["STOP", "CONTINUE", "REPEAT"])
+            - input_filters (list[dict], optional): Input data validation filters
+            - mlflow_transform_filters (list[dict], optional): Transform stage filters
+            - mlflow_predict_filters (list[dict], optional): Prediction stage filters
+            - model_retention_minutes (int, optional): Data retention time in minutes (default: 60)
+            - save_transform (bool, optional): Save transformed data to database (default: True)
+            - predictions_storage_policy (str, optional): Prediction storage policy (default: 'lts:1')
+            - pi_web_api_output_config (dict, optional): PI Web API output configuration for write-back (default: {})
+            - Additional fields from common_config
+
+    Returns:
+        dict[str, Any]: Complete predictions batch configuration with OPC output mappings,
+            PI Web API output configuration, multi-stage filters, SQL query, and retention policies
+    """
+    path_priority = process_path_priority(
+        config.get('path_priority', ['STOP', 'CONTINUE', 'REPEAT'])
+    )
+
+    return {
+        **common_config(config),
+        'query': config['query'],
+        'datetime_columns': config.get('datetime_columns', []),
+        'schema': 'sientia_data',
+        'table_name': 'predictions',
+        'save_transform': config.get('save_transform', True),
+        'transform_table_name': 'transformed_data',
+        'retention_time': config.get('model_retention_minutes', 60) * 60,
+        'opc_output_config': config.get('opc_output_config', {}),
+        'pi_web_api_output_config': config.get('pi_web_api_output_config', {}),
+        'input_filters': overlap_filter_config(
+            {'EMPTY_DATA': {'policy': 'STOP', 'config': {}}}, config.get('input_filters', [])
+        ),
+        'mlflow_transform_filters': overlap_filter_config(
+            {
+                'EMPTY_DATA': {'policy': 'STOP', 'config': {}},
+                'API_ERROR': {'policy': 'STOP', 'config': {}},
+            },
+            config.get('mlflow_transform_filters', []),
+        ),
+        'mlflow_predict_filters': overlap_filter_config(
+            {'API_ERROR': {'policy': 'STOP', 'config': {}}},
+            config.get('mlflow_predict_filters', []),
+        ),
+        'path_priority': path_priority,
+        'predictions_storage_policy': config.get('predictions_storage_policy', 'lts:1'),
+    }

orchestrator/utils/tag_database.py

@@ -0,0 +1,310 @@
+"""
+Tag database (connectors + connector_tags): which tags the pipelines reference, the
+indexes of what was loaded, and the resolution of the OPC UA read and write tags.
+"""
+
+from typing import Any
+
+# Fields a tag database document needs to take part in a slot or a write config.
+# Documents missing any of them are left out of the index, so their tags are notified
+# as not found instead of failing the whole run.
+CONNECTOR_TAG_FIELDS = ('tag_id', 'connector_id', 'address', 'tag_name')
+
+
+CONNECTOR_FIELDS = ('connector_id', 'name', 'url')
+
+
+# security fields of an OPC UA connector: the source of its OPC security (D4)
+SECURITY_FIELDS = ('mode', 'policy')
+
+
+# Fields of a client certificate the slot carries, and of each object it names in MinIO
+CERTIFICATE_FIELDS = ('certificate_id', 'application_uri')
+CERTIFICATE_OBJECTS = ('cert', 'private_key')
+STORED_OBJECT_FIELDS = ('bucket', 'object_name')
+
+
+# connector_tags categories each use accepts
+READ_CATEGORIES = ('Read',)
+WRITE_CATEGORIES = ('Write', 'Prediction Confidence')
+
+
+# opc_output_config block each write category goes to
+WRITE_TAG_GROUPS = {'Write': 'prediction_tags', 'Prediction Confidence': 'confidence_tags'}
+
+
+# data_type values the OPC Gateway write engine accepts (contract C2)
+OPC_DATA_TYPES = ('float', 'double', 'real', 'int', 'integer', 'bool', 'boolean', 'str', 'string')
+DEFAULT_DATA_TYPE = 'float'
+
+
+# Workflow types whose write_tags reference the tag database
+WRITE_WORKFLOW_TYPES = ('predictions_batch', 'xgboost_predictions_batch')
+
+
+def is_complete(document: dict[str, Any], fields: tuple[str, ...]) -> bool:
+    return all(document.get(field) not in (None, '') for field in fields)
+
+
+def tag_id_of(tag: Any) -> int | None:
+    """
+    The tag's tag_id, or None when it is not an integer.
+
+    The connector_tags validator stores tag_id as int, so a pipeline references a tag
+    by the same int: text ('7'), bool, null or a missing tag_id is no reference.
+    """
+    tag_id = tag.get('tag_id') if isinstance(tag, dict) else None
+    return tag_id if isinstance(tag_id, int) and not isinstance(tag_id, bool) else None
+
+
+def _is_active(document: dict[str, Any]) -> bool:
+    """A document without status (an override that does not project it) counts as active."""
+    return document.get('status') in (None, 'Active')
+
+
+def pipeline_tag_refs(pipeline: dict[str, Any]) -> list[Any]:
+    """
+    The tag references of a pipeline: the read_tags of a scouter and the write_tags of a
+    predictions_batch. Entries are returned as they are (they may lack a tag_id).
+    """
+    workflow_type = pipeline.get('workflow_type')
+    if workflow_type == 'scouter':
+        return list(pipeline.get('read_tags') or [])
+    if workflow_type in WRITE_WORKFLOW_TYPES:
+        return list(pipeline.get('write_tags') or [])
+    return []
+
+
+def referenced_tag_ids(pipelines: list[dict[str, Any]]) -> list[int]:
+    """
+    The tag_ids the pipelines reference (scouter read_tags and predictions_batch
+    write_tags), for the connector_tags query.
+
+    Sorted, so the query is the same on every run. Runs in the workflow, so it never
+    raises on malformed references.
+
+    Args:
+        pipelines (list[dict[str, Any]]): Pipeline configurations
+
+    Returns:
+        list[int]: The referenced tag_ids (empty when no tag carries a tag_id)
+    """
+    ids = {tag_id_of(tag) for pipeline in pipelines for tag in pipeline_tag_refs(pipeline)}
+    return sorted(tag_id for tag_id in ids if tag_id is not None)
+
+
+def filter_by_tag_ids(query: dict[str, Any], tag_ids: list[int]) -> dict[str, Any]:
+    """
+    Restrict a connector_tags aggregation to the referenced tag_ids.
+
+    Args:
+        query (dict[str, Any]): The connector_tags query ({'collection', 'aggregation'})
+        tag_ids (list[int]): The ids from referenced_tag_ids
+
+    Returns:
+        dict[str, Any]: A new query whose aggregation starts with a tag_id $match. The
+            aggregation list is new too: aggregate_documents_in_mongodb appends a stage
+            to the list it gets.
+    """
+    return {
+        **query,
+        'aggregation': [{'$match': {'tag_id': {'$in': tag_ids}}}, *query['aggregation']],
+    }
+
+
+def index_tag_documents(connector_tags: list[dict[str, Any]]) -> dict[int, dict[str, Any]]:
+    """
+    Every loaded connector_tags document keyed by tag_id, whatever its status, category
+    or completeness.
+    """
+    indexed = {}
+    for tag in connector_tags:
+        tag_id = tag_id_of(tag)
+        if tag_id is not None:
+            indexed[tag_id] = tag
+    return indexed
+
+
+def index_connector_tags(
+    connector_tags: list[dict[str, Any]], categories: tuple[str, ...] = READ_CATEGORIES
+) -> dict[int, dict[str, Any]]:
+    """
+    Index the tag database's usable connector_tags by tag_id.
+
+    Documents that are not active, have another category, or lack an int tag_id,
+    connector_id, address or tag_name are left out.
+
+    Args:
+        connector_tags (list[dict[str, Any]]): connector_tags documents (SIENTIAPDE-1569)
+        categories (tuple[str, ...]): Accepted categories (READ_CATEGORIES for the slot
+            and the scouter names, WRITE_CATEGORIES for the write tags). A document
+            without category is accepted.
+
+    Returns:
+        dict[int, dict[str, Any]]: connector_tags documents keyed by tag_id
+    """
+    return {
+        tag_id: tag
+        for tag_id, tag in index_tag_documents(connector_tags).items()
+        if is_complete(tag, CONNECTOR_TAG_FIELDS)
+        and _is_active(tag)
+        and tag.get('category') in (None, *categories)
+    }
+
+
+def index_connectors(connectors: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
+    """
+    Index the tag database's connectors by connector_id (as text).
+
+    Documents that are not active or lack connector_id, name, url, security.mode or
+    security.policy are left out.
+
+    Args:
+        connectors (list[dict[str, Any]]): connectors documents (SIENTIAPDE-1569)
+
+    Returns:
+        dict[str, dict[str, Any]]: connectors documents keyed by str(connector_id)
+    """
+    return {
+        str(connector['connector_id']): connector
+        for connector in connectors
+        if is_complete(connector, CONNECTOR_FIELDS)
+        and _has_security(connector)
+        and _is_active(connector)
+    }
+
+
+def _has_security(connector: dict[str, Any]) -> bool:
+    security = connector.get('security')
+    return isinstance(security, dict) and is_complete(security, SECURITY_FIELDS)
+
+
+def client_certificate_id(connector: dict[str, Any]) -> Any:
+    """The connector's security.client_certificate_id (None when it has none)."""
+    security = connector.get('security')
+    return security.get('client_certificate_id') if isinstance(security, dict) else None
+
+
+def _is_usable_certificate(certificate: Any) -> bool:
+    return (
+        isinstance(certificate, dict)
+        and is_complete(certificate, CERTIFICATE_FIELDS)
+        and all(
+            isinstance(certificate.get(name), dict)
+            and is_complete(certificate[name], STORED_OBJECT_FIELDS)
+            for name in CERTIFICATE_OBJECTS
+        )
+    )
+
+
+def resolve_certificate(connector: dict[str, Any]) -> dict[str, Any] | None:
+    """
+    The client certificate a connector goes into the slot with, or None (D5).
+
+    connectors_query's $lookup puts in `certificates` the opc_certificates document the
+    connector's security.client_certificate_id points to, only when it is active and of
+    kind client. None means no certificate: the gateway connects without security. A
+    connector with a client_certificate_id and no certificate is most likely a
+    registration error, reported by the caller (see client_certificate_id).
+
+    Args:
+        connector (dict[str, Any]): A connectors document from connectors_query
+
+    Returns:
+        dict[str, Any] | None: certificate_id, application_uri, and cert and private_key
+            as {bucket, object_name}; None without a complete certificate
+    """
+    for certificate in connector.get('certificates') or []:
+        if _is_usable_certificate(certificate):
+            return {
+                **{field: certificate[field] for field in CERTIFICATE_FIELDS},
+                **{
+                    name: {field: certificate[name][field] for field in STORED_OBJECT_FIELDS}
+                    for name in CERTIFICATE_OBJECTS
+                },
+            }
+    return None
+
+
+def resolve_tag_names(
+    read_tags: list[dict[str, Any]], tags_by_id: dict[int, dict[str, Any]]
+) -> list[dict[str, Any]]:
+    """
+    Fill in the tag_name of each read tag of a scouter pipeline.
+
+    The name is the pipeline's own tag_name when it has one, else the connector_tags
+    name of its tag_id. The scouter's model_tags (scouter()) and the slot's names[]
+    (gather_read_tags()) both take the name from here, so the name the gateway writes
+    is always the one the scouter looks for (SIENTIAPDE-1974, SC1).
+
+    Args:
+        read_tags (list[dict[str, Any]]): The pipeline's read_tags
+        tags_by_id (dict[int, dict[str, Any]]): connector_tags indexed by index_connector_tags
+
+    Returns:
+        list[dict[str, Any]]: The read tags, each with tag_name set when it could be
+            resolved. A tag_id missing from tags_by_id leaves the tag without a name.
+    """
+    resolved = []
+    for tag in read_tags:
+        name = tag.get('tag_name')
+        tag_id = tag_id_of(tag)
+        if not name and tag_id is not None:
+            name = tags_by_id.get(tag_id, {}).get('tag_name')
+        resolved.append({**tag, 'tag_name': name} if name else tag)
+    return resolved
+
+
+def resolve_write_tags(
+    write_tags: list[Any],
+    tags_by_id: dict[int, dict[str, Any]],
+    connectors: dict[str, dict[str, Any]],
+) -> tuple[dict[str, Any], dict[str, set[str]]]:
+    """
+    Build the opc_output_config of a predictions_batch from its write_tags.
+
+    Each write tag is only a tag_id: the tag database gives the rest. The category says
+    where the value goes ('Write' → prediction_tags, 'Prediction Confidence' →
+    confidence_tags), the connector_id is the server the OPC Gateway writes to (it
+    resolves it in connectors), and the address and data_type come from connector_tags
+    (data_type defaults to 'float').
+
+    Args:
+        write_tags (list[Any]): The pipeline's write_tags ([{tag_id}])
+        tags_by_id (dict[int, dict[str, Any]]): Write connector_tags indexed by
+            index_connector_tags(..., WRITE_CATEGORIES)
+        connectors (dict[str, dict[str, Any]]): Active OPC UA connectors indexed by
+            index_connectors
+
+    Returns:
+        tuple: the opc_output_config ({connector_id: {prediction_tags | confidence_tags:
+            {address: {data_type}}}}) and what was left out, by kind: 'tags' (tag_id not
+            an active, complete Write / Prediction Confidence tag), 'connectors'
+            (connector_id) and 'invalid' (an entry without an int tag_id, or a tag whose
+            data_type the gateway does not accept)
+    """
+    issues: dict[str, set[str]] = {'tags': set(), 'connectors': set(), 'invalid': set()}
+    output_config: dict[str, Any] = {}
+    for entry in write_tags:
+        tag_id = tag_id_of(entry)
+        if tag_id is None:
+            issues['invalid'].add(str(entry))
+            continue
+        connector_tag = tags_by_id.get(tag_id)
+        # The category is what tells a prediction from a confidence: it cannot be missing
+        if connector_tag is None or connector_tag.get('category') not in WRITE_TAG_GROUPS:
+            issues['tags'].add(str(tag_id))
+            continue
+        connector_id = str(connector_tag['connector_id'])
+        if connector_id not in connectors:
+            issues['connectors'].add(connector_id)
+            continue
+        data_type = connector_tag.get('data_type') or DEFAULT_DATA_TYPE
+        if data_type not in OPC_DATA_TYPES:
+            issues['invalid'].add(str(tag_id))
+            continue
+        group = WRITE_TAG_GROUPS[connector_tag['category']]
+        output_config.setdefault(connector_id, {}).setdefault(group, {})[
+            connector_tag['address']
+        ] = {'data_type': data_type}
+    return output_config, issues

orchestrator/worker/worker.py

@@ -5,6 +5,7 @@ with workflow.unsafe.imports_passed_through():
     import asyncio
     import os
     import sys
+    import traceback
 
     from prometheus_client import start_http_server
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
@@ -192,13 +193,12 @@ async def main():
     try:
         await asyncio.gather(*handlers)
     except BaseException as e:
-        logger.error(f'An unhandled exception occurred: {e}', exc_info=True)
+        # sientia_do's Logger.error takes only a message: the traceback goes in it
+        logger.error(f'An unhandled exception occurred: {e}\n{traceback.format_exc()}')
         exit_code = 1
     finally:
-        if notification_handler:
-            notification_handler.shutdown()
-        if activities:
-            activities.shutdown()
+        notification_handler.shutdown()
+        activities.shutdown()
         metrics.APP_UP.labels(pod_id=POD_ID).set(0)
         sys.exit(exit_code)
 

orchestrator/workflows/orchestrator.py

@@ -7,26 +7,69 @@ with workflow.unsafe.imports_passed_through():
     from sientia_do.temporal.policies import retry_policy
 
     from orchestrator.activities.activities import Activities
-    from orchestrator.utils.orchestrator_functions import filter_by_tag_ids, referenced_tag_ids
+    from orchestrator.utils.tag_database import (
+        filter_by_tag_ids,
+        referenced_tag_ids,
+    )
 
-# Tag database (SIENTIAPDE-1569) read by the slot v2 (SIENTIAPDE-1974). The workflow
-# input may override these aggregations (connectors_query / connector_tags_query); the
-# workflow puts a tag_id $match in front of connector_tags_query. Only the fields the
-# slot uses are projected: other attributes may hold BSON types a Temporal payload
-# cannot carry.
+# MongoDB aggregation stages
+MATCH = '$match'
+PROJECT = '$project'
+LOOKUP = '$lookup'
+
+# Tag database (SIENTIAPDE-1569) read by the slot v2 and the schedules (SIENTIAPDE-1974).
+# The workflow input may override these aggregations (connectors_query /
+# connector_tags_query); the workflow puts a tag_id $match in front of
+# connector_tags_query. Every category and status of the referenced tags is loaded: the
+# slot uses the active Read tags, predictions_batch the active Write / Prediction
+# Confidence ones (status and category are filtered in the code). Only the fields used
+# are projected: other attributes may hold BSON types a Temporal payload cannot carry
+# (so the certificate dates are left out).
+# Each connector brings, in `certificates`, the opc_certificates document its
+# security.client_certificate_id points to, only when it is active and of kind client
+# (D5); resolve_certificate turns it into the slot's certificate.
 DEFAULT_CONNECTORS_QUERY = {
     'collection': 'connectors',
     'aggregation': [
-        {'$match': {'type': 'opc_ua', 'status': 'Active'}},
+        {MATCH: {'type': 'opc_ua', 'status': 'Active'}},
         {
-            '$project': {
+            LOOKUP: {
+                'from': 'opc_certificates',
+                'let': {'certificate_id': '$security.client_certificate_id'},
+                'pipeline': [
+                    {
+                        MATCH: {
+                            '$expr': {'$eq': ['$certificate_id', '$$certificate_id']},
+                            'status': 'active',
+                            'kind': 'client',
+                        }
+                    },
+                    {
+                        PROJECT: {
+                            '_id': 0,
+                            'certificate_id': 1,
+                            'application_uri': 1,
+                            'cert.bucket': 1,
+                            'cert.object_name': 1,
+                            'private_key.bucket': 1,
+                            'private_key.object_name': 1,
+                        }
+                    },
+                ],
+                'as': 'certificates',
+            }
+        },
+        {
+            PROJECT: {
                 '_id': 0,
                 'connector_id': 1,
                 'name': 1,
                 'url': 1,
-                'server_uri': 1,
-                'use_security': 1,
                 'subscription_period_ms': 1,
+                'security.mode': 1,
+                'security.policy': 1,
+                'security.client_certificate_id': 1,
+                'certificates': 1,
             }
         },
     ],
@@ -34,12 +77,20 @@ DEFAULT_CONNECTORS_QUERY = {
 DEFAULT_CONNECTOR_TAGS_QUERY = {
     'collection': 'connector_tags',
     'aggregation': [
-        {'$match': {'category': 'Read', 'status': 'Active'}},
-        {'$project': {'_id': 0, 'tag_id': 1, 'connector_id': 1, 'address': 1, 'tag_name': 1}},
+        {
+            PROJECT: {
+                '_id': 0,
+                'tag_id': 1,
+                'connector_id': 1,
+                'address': 1,
+                'tag_name': 1,
+                'category': 1,
+                'data_type': 1,
+                'status': 1,
+            }
+        },
     ],
 }
-# Runs started before the tag database loads existed replay without them
-TAG_DATABASE_PATCH = 'SIENTIAPDE-1974-tag-database'
 
 
 @workflow.defn(name='orchestrator')
@@ -60,12 +111,12 @@ class Orchestrator:
         pipelines: list[dict[str, Any]],
     ) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
         """
-        Load the tag database (SIENTIAPDE-1974) for the read tags that carry a tag_id.
+        Load the tag database (SIENTIAPDE-1974): the only source of the OPC servers and
+        tags of the read and write tags.
 
-        Only the referenced tags are loaded, with the fields the slot uses, so an
-        environment without a tag database never depends on its content (P4) and the
-        payload stays bounded by the pipelines. Nothing is loaded when no scouter read
-        tag has a tag_id, or when replaying a run started before TAG_DATABASE_PATCH.
+        Only the referenced tags are loaded, with the fields used, so the payload stays
+        bounded by the pipelines. Nothing is loaded when no scouter read tag nor
+        predictions_batch write tag has a tag_id.
 
         Args:
             input_data (dict[str, Any]): Workflow input (optional connectors_query and
@@ -77,13 +128,24 @@ class Orchestrator:
             tuple[list[dict[str, Any]], list[dict[str, Any]]]: connectors and connector_tags
         """
         tag_ids = referenced_tag_ids(pipelines)
-        # patched() first: it must run on every execution to record its marker
-        if not workflow.patched(TAG_DATABASE_PATCH) or not tag_ids:
+        if not tag_ids:
             return [], []
 
-        connectors_query = input_data.get('connectors_query', DEFAULT_CONNECTORS_QUERY)
-        connector_tags_query = input_data.get('connector_tags_query', DEFAULT_CONNECTOR_TAGS_QUERY)
+        return await self._load_connectors(
+            metadata,
+            input_data.get('connectors_query', DEFAULT_CONNECTORS_QUERY),
+            input_data.get('connector_tags_query', DEFAULT_CONNECTOR_TAGS_QUERY),
+            tag_ids,
+        )
 
+    async def _load_connectors(
+        self,
+        metadata: dict[str, Any],
+        connectors_query: dict[str, Any],
+        connector_tags_query: dict[str, Any],
+        tag_ids: list[Any],
+    ) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+        """Run a connectors query and a connector_tags query restricted to tag_ids, in parallel."""
         connectors_handler = workflow.start_local_activity_method(
             Activities.aggregate_documents_in_mongodb,
             {
@@ -97,7 +159,10 @@ class Orchestrator:
 
         connector_tags_handler = workflow.start_local_activity_method(
             Activities.aggregate_documents_in_mongodb,
-            {**metadata, 'query': filter_by_tag_ids(connector_tags_query, tag_ids)},
+            {
+                **metadata,
+                'query': filter_by_tag_ids(connector_tags_query, tag_ids),
+            },
             retry_policy=retry_policy,
             start_to_close_timeout=timedelta(seconds=60),
         )
@@ -109,9 +174,9 @@ class Orchestrator:
         """
         Execute the orchestration workflow for pipeline and slot management.
 
-        This workflow retrieves pipeline configurations and OPC server data,
-        processes schedules and slot configurations, and deploys them to
-        the appropriate services. It handles creation, updates, and deletion
+        This workflow retrieves pipeline configurations and the tag database
+        (connectors and connector_tags), processes schedules and slot configurations,
+        and deploys them to the appropriate services. It handles creation, updates, and deletion
         of schedules and slots based on current system state.
 
         Args:
@@ -119,8 +184,7 @@ class Orchestrator:
                 Required fields:
                 - schedule_name (str): Name of the orchestration schedule
                 - pipelines_query (dict[str, Any]): MongoDB query for pipeline configurations
-                - opc_servers_query (dict[str, Any]): MongoDB query for OPC server data
-                Optional fields (used only when a scouter read tag carries a tag_id):
+                Optional fields:
                 - connectors_query (dict[str, Any]): MongoDB aggregation for the tag
                   database connectors (default: DEFAULT_CONNECTORS_QUERY)
                 - connector_tags_query (dict[str, Any]): MongoDB aggregation for the tag
@@ -155,13 +219,6 @@ class Orchestrator:
             start_to_close_timeout=timedelta(seconds=60),
         )
 
-        opc_servers_handler = workflow.start_local_activity_method(
-            Activities.find_documents_in_mongodb,
-            {**metadata, 'query': input_data['opc_servers_query']},
-            retry_policy=retry_policy,
-            start_to_close_timeout=timedelta(seconds=60),
-        )
-
         orchestrated_schedules_handler = workflow.start_local_activity_method(
             Activities.find_documents_in_mongodb,
             {
@@ -199,7 +256,6 @@ class Orchestrator:
 
         orchestrated_schedules = await orchestrated_schedules_handler
         current_slot_config = await current_slot_config_handler
-        opc_servers = await opc_servers_handler
         active_ingestors = await active_ingestors_handler
 
         formatted_orchestrated_schedules_handler = workflow.start_local_activity_method(
@@ -211,7 +267,12 @@ class Orchestrator:
 
         schedules_config_handler = workflow.start_local_activity_method(
             Activities.process_schedules,
-            {**metadata, 'pipelines': pipeline_config, 'connector_tags': connector_tags},
+            {
+                **metadata,
+                'pipelines': pipeline_config,
+                'connectors': connectors,
+                'connector_tags': connector_tags,
+            },
             retry_policy=retry_policy,
             start_to_close_timeout=timedelta(seconds=60),
         )
@@ -220,7 +281,6 @@ class Orchestrator:
             Activities.process_slots,
             {
                 **metadata,
-                'opc_servers': opc_servers,
                 'connectors': connectors,
                 'connector_tags': connector_tags,
                 'active_ingestors': active_ingestors,

pipelines_sample.json

@@ -16,25 +16,22 @@
   ],
   "read_tags": [
     {
-      "server_id": "1",
+      "tag_id": 1,
       "tag_name": "Counter",
-      "tag_address": "ns=2;s=GatewayProSYS.ProSYS.Simulation.Counter",
       "aggr_func": "lts",
       "data_range": [1, 30],
       "frequency": 30000
     },
     {
-      "server_id": "1",
+      "tag_id": 2,
       "tag_name": "Square",
-      "tag_address": "ns=2;s=GatewayProSYS.ProSYS.Simulation.Square",
       "aggr_func": "lts",
       "data_range": [-2, 2],
       "frequency": 30000
     },
     {
-      "server_id": "1",
+      "tag_id": 3,
       "tag_name": "Rollout",
-      "tag_address": "ns=2;s=GatewayProSYS.ProSYS.Simulation.Rollout",
       "aggr_func": "lts",
       "data_range": [-2, 2],
       "frequency": 30000

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "orchestrator"
-version = "0.6.7"
+version = "0.9.0"
 description = "Sientia DataOps Orchestrator - ML Model Orchestration System"
 readme = "README.md"
 requires-python = ">=3.11"

requirements-local.txt

@@ -4,5 +4,5 @@ sqlalchemy
 redis
 pymongo
 jinja2
-git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.12.3
+git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.12.5
 prometheus-client

requirements.txt

@@ -4,5 +4,5 @@ sqlalchemy
 redis
 pymongo
 jinja2
-sientia_do>=1.12.3
+sientia_do>=1.12.5
 prometheus-client

tests/orchestrator/activities/test_email.py

@@ -326,3 +326,37 @@ def test_send_email(mimemultipart, mimetext, email):
     )
 
     assert email.try_send_email.call_count == 2
+
+
+@patch('orchestrator.activities.email.EmailBuilder')
+@patch('orchestrator.activities.email.smtplib')
+def test___init___without_smtp_server(smtplib, email_builder):
+    email = Email(
+        sender_email='test@test.com',
+        sender_password='test',
+        smtp_server=None,
+        smtp_port=587,
+        logger=MagicMock(),
+        notification_handler=MagicMock(),
+        metrics_controller=AsyncMock(),
+    )
+
+    smtplib.SMTP.assert_not_called()
+    assert not hasattr(email, 'server')
+
+
+@patch('orchestrator.activities.email.smtplib.SMTP')
+def test_try_send_email_reconnect_without_password(smtp, email):
+    email.sender_password = None
+    email.server.sendmail = MagicMock(side_effect=SMTPServerDisconnected('test'))
+    email.server.quit = MagicMock()
+
+    msg = MagicMock()
+
+    email.try_send_email(msg, 'test')
+
+    smtp.return_value.starttls.assert_not_called()
+    smtp.return_value.login.assert_not_called()
+    smtp.return_value.sendmail.assert_called_once_with(
+        'test@test.com', 'test', msg.as_string.return_value
+    )

tests/orchestrator/activities/test_mongo_db.py

@@ -1,14 +1,14 @@
 from datetime import UTC, datetime
-from unittest.mock import ANY, AsyncMock, MagicMock, patch
+from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
 
-from pytest import fixture
+import pytest
 from sientia_do.notifications.models import NotificationLevel
 from sientia_do.temporal.constants import DATETIME_FORMAT_MS_WITH_TZ
 
 from orchestrator.activities.mongo_db import MongoDB
 
 
-@fixture
+@pytest.fixture
 @patch('orchestrator.activities.mongo_db.MongoDBRepository')
 def mongo_db(mongo_mock):
     mongo = MongoDB(
@@ -192,6 +192,30 @@ def test_aggregate_documents_in_mongodb_success(mongo_db):
     )
 
 
+def test_aggregate_documents_in_mongodb_formats_only_datetimes(mongo_db):
+    """A timestamp field stored as text, or missing, is left as it is"""
+    mongo_db.mongo_db_repository.aggregate = MagicMock(
+        return_value=[
+            {'tag_id': 1, 'updated_at': datetime(2026, 10, 1, 12, 0, 0)},
+            {'tag_id': 2, 'updated_at': '2026-10-02 00:00:00.000+0000'},
+            {'tag_id': 3},
+        ]
+    )
+
+    result = mongo_db.aggregate_documents_in_mongodb(
+        {
+            'query': {'collection': 'connector_tags', 'aggregation': [{'$match': {}}]},
+            'timestamp_fields': ['updated_at'],
+        }
+    )
+
+    assert result == [
+        {'tag_id': 1, 'updated_at': '2026-10-01 12:00:00.000000+0000'},
+        {'tag_id': 2, 'updated_at': '2026-10-02 00:00:00.000+0000'},
+        {'tag_id': 3},
+    ]
+
+
 def test_aggregate_documents_in_mongodb_failure(mongo_db):
     input_data = {
         'collection': 'test_collection',
@@ -246,21 +270,59 @@ def test_aggregate_documents_in_mongodb_missing_aggregation(mongo_db):
 def test_update_pipelines_timestamps_success(now_mock, mongo_db):
     input_data = {
         'updated_pipelines': [
-            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
+            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True, 'config_hash': 'h1'},
+            # Updated by a run started before config_hash
             {'schedule_name': 'test2', 'namespace': 'test2', 'success': True},
         ]
     }
-    mongo_db.mongo_db_repository.update_many = MagicMock(return_value=MagicMock())
+    mongo_db.mongo_db_repository.update_one = MagicMock(return_value=1)
     mongo_db.update_pipelines_timestamps(input_data)
-    mongo_db.mongo_db_repository.update_many.assert_called_once_with(
+    assert mongo_db.mongo_db_repository.update_one.call_args_list == [
+        call(
+            'orchestrated_schedules',
+            {'schedule_name': 'test1', 'namespace': 'test1'},
+            {'$set': {'updated_at': now_mock.return_value, 'config_hash': 'h1'}},
+            {},
+        ),
+        call(
+            'orchestrated_schedules',
+            {'schedule_name': 'test2', 'namespace': 'test2'},
+            {'$set': {'updated_at': now_mock.return_value, 'config_hash': None}},
+            {},
+        ),
+    ]
+
+
+@patch('orchestrator.activities.mongo_db.now')
+@pytest.mark.parametrize(
+    'pipelines', [[], [{'schedule_name': 'test1', 'namespace': 'test1', 'success': False}]]
+)
+def test_update_pipelines_timestamps_all_failed_stamps_nothing(now_mock, mongo_db, pipelines):
+    input_data = {'updated_pipelines': pipelines}
+    mongo_db.mongo_db_repository.update_one = MagicMock()
+
+    mongo_db.update_pipelines_timestamps(input_data)
+
+    # A failed update is not stamped, so it is retried on the next tick
+    mongo_db.mongo_db_repository.update_one.assert_not_called()
+
+
+@patch('orchestrator.activities.mongo_db.now')
+def test_update_pipelines_timestamps_mixed_stamps_only_successes(now_mock, mongo_db):
+    input_data = {
+        'updated_pipelines': [
+            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True, 'config_hash': 'h1'},
+            {'schedule_name': 'test2', 'namespace': 'test2', 'success': False, 'config_hash': 'h2'},
+        ]
+    }
+    mongo_db.mongo_db_repository.update_one = MagicMock()
+
+    mongo_db.update_pipelines_timestamps(input_data)
+
+    mongo_db.mongo_db_repository.update_one.assert_called_once_with(
         'orchestrated_schedules',
-        {
-            '$or': [
-                {'schedule_name': 'test1', 'namespace': 'test1'},
-                {'schedule_name': 'test2', 'namespace': 'test2'},
-            ]
-        },
-        {'$set': {'updated_at': now_mock.return_value}},
+        {'schedule_name': 'test1', 'namespace': 'test1'},
+        {'$set': {'updated_at': now_mock.return_value, 'config_hash': 'h1'}},
         {},
     )
 
@@ -275,15 +337,17 @@ def test_update_pipelines_timestamps_failure(now_mock, mongo_db):
         **metadata,
     }
 
-    mongo_db.mongo_db_repository.update_many.side_effect = Exception('Error')
+    # The first update fails: the second is still tried, and only the failure is notified
+    mongo_db.mongo_db_repository.update_one.side_effect = [Exception('Error'), 1]
     try:
         mongo_db.update_pipelines_timestamps(input_data)
-    except Exception as e:
-        assert str(e) == 'Error'
+    except RuntimeError as e:
+        assert str(e) == 'Failed to update 1 of 2 pipelines timestamps'
+        assert mongo_db.mongo_db_repository.update_one.call_count == 2
         mongo_db.send_notification.assert_called_once_with(
             metadata=metadata['metadata'],
             notification_id='MONGODB_UPDATE_PIPELINES_ERROR',
-            message='Failed to update pipelines timestamps: Error',
+            message='Failed to update pipelines timestamps: test1/test1: Error',
             block='update_pipelines_timestamps',
             level=NotificationLevel.ERROR,
             attachment_content=ANY,
@@ -293,11 +357,37 @@ def test_update_pipelines_timestamps_failure(now_mock, mongo_db):
         raise AssertionError('Expected an exception to be raised')
 
 
+@patch('orchestrator.activities.mongo_db.now')
+def test_update_pipelines_timestamps_failures_notified_together(now_mock, mongo_db):
+    input_data = {
+        'updated_pipelines': [
+            {'schedule_name': 'test1', 'namespace': 'ns', 'success': True},
+            {'schedule_name': 'test2', 'namespace': 'ns', 'success': True},
+            {'schedule_name': 'test3', 'namespace': 'ns', 'success': True},
+        ],
+        **metadata,
+    }
+    mongo_db.mongo_db_repository.update_one.side_effect = [
+        Exception('down'),
+        1,
+        Exception('timeout'),
+    ]
+
+    with pytest.raises(RuntimeError, match='Failed to update 2 of 3 pipelines timestamps'):
+        mongo_db.update_pipelines_timestamps(input_data)
+
+    assert mongo_db.mongo_db_repository.update_one.call_count == 3
+    mongo_db.send_notification.assert_called_once()
+    assert mongo_db.send_notification.call_args.kwargs['message'] == (
+        'Failed to update pipelines timestamps: ns/test1: down; ns/test3: timeout'
+    )
+
+
 @patch('orchestrator.activities.mongo_db.now')
 def test_create_pipelines_timestamps_success(now_mock, mongo_db):
     input_data = {
         'created_pipelines': [
-            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
+            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True, 'config_hash': 'h1'},
             {'schedule_name': 'test2', 'namespace': 'test2', 'success': True},
         ]
     }
@@ -306,8 +396,18 @@ def test_create_pipelines_timestamps_success(now_mock, mongo_db):
     mongo_db.mongo_db_repository.insert_many.assert_called_once_with(
         'orchestrated_schedules',
         [
-            {'schedule_name': 'test1', 'namespace': 'test1', 'updated_at': now_mock.return_value},
-            {'schedule_name': 'test2', 'namespace': 'test2', 'updated_at': now_mock.return_value},
+            {
+                'schedule_name': 'test1',
+                'namespace': 'test1',
+                'updated_at': now_mock.return_value,
+                'config_hash': 'h1',
+            },
+            {
+                'schedule_name': 'test2',
+                'namespace': 'test2',
+                'updated_at': now_mock.return_value,
+                'config_hash': None,
+            },
         ],
         {},
     )
@@ -362,6 +462,38 @@ def test_delete_pipelines_timestamps_success(now_mock, mongo_db):
     )
 
 
+@pytest.mark.parametrize(
+    'pipelines', [[], [{'schedule_name': 'test1', 'namespace': 'test1', 'success': False}]]
+)
+def test_delete_pipelines_timestamps_all_failed_deletes_nothing(mongo_db, pipelines):
+    input_data = {'deleted_pipelines': pipelines}
+    mongo_db.mongo_db_repository.delete_many = MagicMock()
+
+    mongo_db.delete_pipelines_timestamps(input_data)
+
+    # An empty filter would wipe orchestrated_schedules; the next tick's normalize_schedules
+    # would then delete and recreate every orchestrated Temporal schedule.
+    mongo_db.mongo_db_repository.delete_many.assert_not_called()
+
+
+def test_delete_pipelines_timestamps_mixed_deletes_only_successes(mongo_db):
+    input_data = {
+        'deleted_pipelines': [
+            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
+            {'schedule_name': 'test2', 'namespace': 'test2', 'success': False},
+        ]
+    }
+    mongo_db.mongo_db_repository.delete_many = MagicMock()
+
+    mongo_db.delete_pipelines_timestamps(input_data)
+
+    mongo_db.mongo_db_repository.delete_many.assert_called_once_with(
+        'orchestrated_schedules',
+        {'$or': [{'schedule_name': 'test1', 'namespace': 'test1'}]},
+        {},
+    )
+
+
 @patch('orchestrator.activities.mongo_db.now')
 def test_delete_pipelines_timestamps_failure(now_mock, mongo_db):
     input_data = {
@@ -637,3 +769,48 @@ def test_load_latest_data_error(mongo_db):
             level=NotificationLevel.ERROR,
             attachment_content=ANY,
         )
+
+
+def test_find_documents_in_mongodb_leaves_documents_without_timestamp_field(mongo_db):
+    mongo_db.mongo_db_repository.find = MagicMock(return_value=[{'name': 'no-timestamp'}])
+
+    result = mongo_db.find_documents_in_mongodb(
+        {'query': {'collection': 'test_collection'}, 'timestamp_fields': ['timestamp']}
+    )
+
+    assert result == [{'name': 'no-timestamp'}]
+
+
+def test_aggregate_documents_in_mongodb_leaves_documents_without_timestamp_field(mongo_db):
+    mongo_db.mongo_db_repository.aggregate = MagicMock(return_value=[{'name': 'no-timestamp'}])
+
+    result = mongo_db.aggregate_documents_in_mongodb(
+        {
+            'query': {'collection': 'test_collection', 'aggregation': [{'$match': {}}]},
+            'timestamp_fields': ['timestamp'],
+        }
+    )
+
+    assert result == [{'name': 'no-timestamp'}]
+
+
+@patch('orchestrator.activities.mongo_db.now')
+def test_create_pipelines_timestamps_all_failed_inserts_nothing(_now_mock, mongo_db):
+    mongo_db.mongo_db_repository.insert_many = MagicMock()
+
+    mongo_db.create_pipelines_timestamps(
+        {'created_pipelines': [{'schedule_name': 'a', 'namespace': 'n', 'success': False}]}
+    )
+
+    mongo_db.mongo_db_repository.insert_many.assert_not_called()
+
+
+def test_create_collection_with_ttl_index_skips_pipelines_without_topic(mongo_db):
+    database = mongo_db.mongo_db_repository.database
+    database.list_collection_names = MagicMock(return_value=[])
+
+    mongo_db.create_collection_with_ttl_index(
+        {**metadata, 'pipelines': {'pi-scouter': {'model_tags': {}}, 'held': {'held': True}}}
+    )
+
+    database.create_collection.assert_not_called()

tests/orchestrator/activities/test_temporal_manager.py

@@ -3,6 +3,7 @@ from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
 
 from pytest import fixture, mark
 from sientia_do.notifications.models import NotificationLevel
+from temporalio.client import ScheduleOverlapPolicy
 
 from orchestrator.activities.temporal_manager import TemporalManager
 from orchestrator.utils.converters import parse_frequency
@@ -144,6 +145,7 @@ async def test_create_schedule(
                     'data': {'test': 'test'},
                     'execution_timeout_seconds': 100,
                     'task_timeout_seconds': 100,
+                    'config_hash': 'h1',
                 },
                 'test-schedule-invalid-frequency': {
                     'model_id': 2,
@@ -290,24 +292,29 @@ async def test_create_schedule(
         ]
     )
 
+    # config_hash goes to the report (orchestrated_schedules), not to the workflow
+    assert 'config_hash' not in input_data['schedules']['scouter']['test-schedule']
     assert report == [
         {
             'schedule_name': 'test-schedule',
             'namespace': 'scouter',
             'success': True,
             'message': 'Schedule created successfully',
+            'config_hash': 'h1',
         },
         {
             'schedule_name': 'test-schedule-invalid-frequency',
             'namespace': 'scouter',
             'success': False,
             'message': 'Invalid frequency',
+            'config_hash': None,
         },
         {
             'schedule_name': 'test-schedule-laborious',
             'namespace': 'laborious',
             'success': True,
             'message': 'Schedule created successfully',
+            'config_hash': None,
         },
     ]
 
@@ -354,6 +361,7 @@ async def test_update_schedules(
                     'workflow_type': 'scouter',
                     'frequency': '1m',
                     'data': {'test': 'test'},
+                    'config_hash': 'h1',
                 },
                 'test-schedule_no_handler': {
                     'workflow_type': 'scouter',
@@ -399,24 +407,28 @@ async def test_update_schedules(
     handler_scouter.update.assert_called_once()
     handler_laborious.update.assert_called_once()
 
+    assert 'config_hash' not in input_data['schedules']['scouter']['test-schedule']
     assert report == [
         {
             'schedule_name': 'test-schedule',
             'namespace': 'scouter',
             'success': True,
             'message': 'Schedule updated successfully',
+            'config_hash': 'h1',
         },
         {
             'schedule_name': 'test-schedule_no_handler',
             'namespace': 'scouter',
             'success': False,
             'message': 'Schedule test-schedule_no_handler not found',
+            'config_hash': None,
         },
         {
             'schedule_name': 'test-schedule-laborious',
             'namespace': 'laborious',
             'success': True,
             'message': 'Schedule updated successfully',
+            'config_hash': None,
         },
     ]
 
@@ -684,3 +696,61 @@ async def test_create_schedules_tenant_runtime_queue(
         task_timeout=timedelta(seconds=300),
         typed_search_attributes=_mock_typed_search_attributes.return_value,
     )
+
+
+@mark.asyncio
+@patch('orchestrator.activities.temporal_manager.Schedule')
+@patch('orchestrator.activities.temporal_manager.ScheduleActionStartWorkflow')
+@patch('orchestrator.activities.temporal_manager.ScheduleIntervalSpec')
+@patch('orchestrator.activities.temporal_manager.ScheduleSpec')
+@patch('orchestrator.activities.temporal_manager.TypedSearchAttributes')
+@patch('orchestrator.activities.temporal_manager.SearchAttributePair')
+async def test_create_schedules_drift_keeps_default_skip_overlap(
+    _mock_search_attribute_pair,
+    _mock_typed_search_attributes,
+    _mock_schedule_spec,
+    _mock_schedule_interval_spec,
+    _mock_schedule_action_start_workflow,
+    mock_schedule,
+    temporal_manager,
+):
+    # The laborious DriftAnalysis cache assumes no two drift runs of a model overlap.
+    input_data = {
+        'schedules': {
+            'laborious': {
+                'm1-drift-1m': {
+                    'model_id': 'm1',
+                    'model_name': 'm1',
+                    'workflow_type': 'drift',
+                    'frequency': '1m',
+                }
+            }
+        }
+    }
+    temporal_manager.temporal_clients['laborious'].create_schedule = AsyncMock()
+
+    await temporal_manager.create_schedules(input_data)
+
+    mock_schedule.assert_called_once()
+    policy = mock_schedule.call_args.kwargs.get('policy')
+    assert policy is None or policy.overlap == ScheduleOverlapPolicy.SKIP
+
+
+@mark.asyncio
+async def test_normalize_schedules_keeps_schedules_known_in_mongo(temporal_manager):
+    """An orchestrated schedule still tracked in orchestrated_schedules is not deleted"""
+    input_data = {
+        'orchestrated_schedules': {
+            'scouter': {'test-schedule-id': '2021-01-01'},
+            'laborious': {'test-schedule-id': '2021-01-01'},
+        },
+        **metadata,
+    }
+    for client in temporal_manager.temporal_clients.values():
+        client.list_schedules = AsyncMock(return_value=async_iter())
+        client.get_schedule_handle = MagicMock()
+
+    await temporal_manager.normalize_schedules(input_data)
+
+    for client in temporal_manager.temporal_clients.values():
+        client.get_schedule_handle.assert_not_called()

tests/orchestrator/utils/tag_database_fixtures.py

@@ -0,0 +1,82 @@
+"""Pipeline and tag database documents shared by the utils tests."""
+
+
+def scouter_pipeline_doc(schedule_name, model_id, read_tags, workflow_type='scouter'):
+    return {
+        'workflow_type': workflow_type,
+        'schedule_name': schedule_name,
+        'model_id': model_id,
+        'model': {'name': f'model {model_id}'},
+        'read_tags': read_tags,
+    }
+
+
+TAGS_BY_ID = {
+    7: {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1001', 'tag_name': 'Pressure'},
+    8: {'tag_id': 8, 'connector_id': 5, 'address': 'ns=3;i=1002', 'tag_name': 'Level'},
+}
+
+
+# Connector security and client certificates (SIENTIAPDE-1974)
+
+NO_SECURITY = {'mode': 'None', 'policy': 'None'}
+SECURE = {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}
+
+
+# A client certificate as connectors_query's $lookup brings it (certificates[])
+def certificate_doc(certificate_id=1, name='sientia-platform', **extra):
+    return {
+        'certificate_id': certificate_id,
+        'application_uri': 'urn:sientia:opc-ingestor',
+        'cert': {'bucket': 'opc-certificates', 'object_name': f'client/{name}/cert.pem'},
+        'private_key': {'bucket': 'opc-certificates', 'object_name': f'client/{name}/key.pem'},
+        **extra,
+    }
+
+
+# The slot's certificate for certificate_doc(): the same fields
+CERTIFICATE = certificate_doc()
+
+
+def secure_connector_fields(certificate_id=1, certificates=None):
+    """security and certificates of a connector with a client certificate."""
+    return {
+        'security': {**SECURE, 'client_certificate_id': certificate_id},
+        'certificates': [certificate_doc(certificate_id)] if certificates is None else certificates,
+    }
+
+
+# Write tags (predictions_batch write_tags through the tag database)
+
+OPC_CONNECTORS = {
+    'plant': {
+        'connector_id': 'plant',
+        'name': 'plant',
+        'url': 'opc.tcp://plant:4840',
+        'security': NO_SECURITY,
+    },
+}
+
+
+def write_tag_doc(tag_id, address, category='Write', connector_id='plant', **extra):
+    return {
+        'tag_id': tag_id,
+        'connector_id': connector_id,
+        'address': address,
+        'tag_name': f'tag-{tag_id}',
+        'category': category,
+        'status': 'Active',
+        **extra,
+    }
+
+
+def batch_pipeline_doc(schedule_name, write_tags, **fields):
+    return {
+        'workflow_type': 'predictions_batch',
+        'schedule_name': schedule_name,
+        'model_id': 'model_id',
+        'model': {'name': 'model_name'},
+        'query': 'SELECT 1',
+        'write_tags': write_tags,
+        **fields,
+    }

tests/orchestrator/utils/test_opc_slots.py

@@ -0,0 +1,450 @@
+import pytest
+
+from orchestrator.utils.opc_slots import (
+    build_tag_config,
+    gather_read_tags,
+)
+from tests.orchestrator.utils.tag_database_fixtures import (
+    CERTIFICATE,
+    NO_SECURITY,
+    SECURE,
+    TAGS_BY_ID,
+    certificate_doc,
+    scouter_pipeline_doc,
+    secure_connector_fields,
+)
+
+NO_ISSUES = {'tags': set(), 'connectors': set(), 'invalid': set(), 'certificates': set()}
+
+
+def _name(schedule_name, tag_name, model_id):
+    return {
+        'collection': f'raw_{schedule_name}',
+        'tag_name': tag_name,
+        'model_id': model_id,
+        'model_name': f'model {model_id}',
+    }
+
+
+def _read(tag_id, frequency=1000, **extra):
+    return {'tag_id': tag_id, 'frequency': frequency, **extra}
+
+
+def _connector(connector_id, name, **extra):
+    return {
+        'connector_id': connector_id,
+        'name': name,
+        'url': f'opc.tcp://{name}:4840',
+        'security': NO_SECURITY,
+        **extra,
+    }
+
+
+def _db_tag(tag_id, connector_id, address, tag_name):
+    return {
+        'tag_id': tag_id,
+        'connector_id': connector_id,
+        'address': address,
+        'tag_name': tag_name,
+    }
+
+
+connectors = {
+    '1': _connector(1, 'kepserver'),
+    '3': _connector(3, 'plant', subscription_period_ms=1500, **secure_connector_fields()),
+}
+
+
+tags_by_id = {
+    **TAGS_BY_ID,
+    10: _db_tag(10, 1, 'ns=2;s=A', 'A'),
+    11: _db_tag(11, 1, 'ns=2;s=B', 'B'),
+    12: _db_tag(12, 1, 'ns=2;s=A', 'A2'),
+    13: _db_tag(13, 3, 'ns=2;s=A', 'A on plant'),
+    14: _db_tag(14, 1, 'ns=2;s=C', 'C'),
+}
+
+
+def _gather(pipelines, connector_index=None, tag_index=None):
+    return gather_read_tags(
+        pipelines,
+        connectors if connector_index is None else connector_index,
+        tags_by_id if tag_index is None else tag_index,
+    )
+
+
+def test_gather_read_tags_multi_name():
+    """One address read by two pipelines with different names: one entry, two names"""
+    pipelines = [
+        scouter_pipeline_doc('sin', 'm1', [_read(10, 15000, tag_name='Counter')]),
+        scouter_pipeline_doc('cos', 'm2', [_read(10, 5000, tag_name='Contador')]),
+    ]
+
+    entries, issues = _gather(pipelines)
+
+    assert len(entries) == 1
+    assert entries[0]['address'] == 'ns=2;s=A'
+    assert entries[0]['frequency'] == 5000
+    assert entries[0]['names'] == [_name('cos', 'Contador', 'm2'), _name('sin', 'Counter', 'm1')]
+    assert issues == NO_ISSUES
+
+
+def test_gather_read_tags_same_name_in_two_pipelines():
+    """The same name read by two pipelines still goes to both collections"""
+    pipelines = [
+        scouter_pipeline_doc('sin', 'm1', [_read(10)]),
+        scouter_pipeline_doc('cos', 'm2', [_read(10)]),
+    ]
+
+    entries, _issues = _gather(pipelines)
+
+    assert entries[0]['names'] == [_name('cos', 'A', 'm2'), _name('sin', 'A', 'm1')]
+
+
+def test_gather_read_tags_duplicate_tag_in_one_pipeline():
+    entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(10), _read(10)])])
+
+    assert entries[0]['names'] == [_name('sin', 'A', 'm1')]
+
+
+def test_gather_read_tags_two_tag_ids_on_one_address():
+    """Two connector_tags on the same node: one entry, one name per tag"""
+    entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(10), _read(12)])])
+
+    assert len(entries) == 1
+    assert entries[0]['names'] == [_name('sin', 'A', 'm1'), _name('sin', 'A2', 'm1')]
+
+
+def test_gather_read_tags_same_address_on_two_servers():
+    entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(13), _read(10)])])
+
+    assert [(e['server']['server_id'], e['address']) for e in entries] == [
+        ('1', 'ns=2;s=A'),
+        ('3', 'ns=2;s=A'),
+    ]
+
+
+def test_gather_read_tags_with_tag_database():
+    """Address, name and server come from connector_tags + connectors"""
+    entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(7)])])
+
+    assert entries == [
+        {
+            'server': {
+                'server_id': '3',
+                'name': 'plant',
+                'url': 'opc.tcp://plant:4840',
+                'subscription_period_ms': 1500,
+                'security': SECURE,
+                'certificate': CERTIFICATE,
+                'key': 'plant',
+            },
+            'address': 'ns=3;i=1001',
+            'frequency': 1000,
+            'active_read': {'enabled': False},
+            'names': [_name('sin', 'Pressure', 'm1')],
+        }
+    ]
+
+
+def test_gather_read_tags_pipeline_name_wins_over_tag_database():
+    entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(7, tag_name='P1')])])
+
+    assert entries[0]['names'] == [_name('sin', 'P1', 'm1')]
+
+
+def test_gather_read_tags_unknown_tag_id():
+    entries, issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(404)])])
+
+    assert entries == []
+    assert issues == {**NO_ISSUES, 'tags': {'404'}}
+
+
+def test_gather_read_tags_inactive_connector():
+    """Connector 5 is not among the active connectors: its tag is left out"""
+    entries, issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(8)])])
+
+    assert entries == []
+    assert issues == {**NO_ISSUES, 'connectors': {'5'}}
+
+
+def test_gather_read_tags_without_tag_id():
+    """connectors is the only source of the servers: a read tag needs a tag_id"""
+    pipelines = [
+        scouter_pipeline_doc(
+            'sin',
+            'm1',
+            [
+                _read(10),
+                {'tag_name': 'b', 'frequency': 1000},
+                {'tag_id': '', 'tag_name': 'c', 'frequency': 1000},
+                {'tag_id': None, 'tag_name': 'd', 'frequency': 1000},
+                # tag_id is int in connector_tags: text is no reference
+                {'tag_id': '11', 'frequency': 1000},
+            ],
+        )
+    ]
+
+    entries, issues = _gather(pipelines)
+
+    assert [e['address'] for e in entries] == ['ns=2;s=A']
+    assert issues == {**NO_ISSUES, 'invalid': {'sin/b', 'sin/c', 'sin/d', 'sin/11'}}
+
+
+def test_gather_read_tags_server_name_clash():
+    """Two servers with the same name get distinct slot keys, stable by server_id"""
+    connector_index = {**connectors, '2': _connector(2, 'kepserver')}
+    tag_index = {**tags_by_id, 20: _db_tag(20, 2, 'ns=2;s=A', 'A')}
+    pipelines = [scouter_pipeline_doc('sin', 'm1', [_read(20), _read(11)])]
+
+    entries, _issues = _gather(pipelines, connector_index, tag_index)
+
+    assert [e['server']['key'] for e in entries] == ['kepserver', 'kepserver:2']
+    assert [e['server']['name'] for e in entries] == ['kepserver', 'kepserver']
+
+
+def test_gather_read_tags_active_read():
+    """Active read: on if any pipeline asks, with the lowest cycle_multiple (D3)"""
+    pipelines = [
+        scouter_pipeline_doc('p1', 'm1', [_read(10)]),
+        scouter_pipeline_doc(
+            'p2', 'm2', [_read(10, active_read={'enabled': True, 'cycle_multiple': 4})]
+        ),
+        scouter_pipeline_doc(
+            'p3', 'm3', [_read(10, active_read={'enabled': True, 'cycle_multiple': 2})]
+        ),
+        scouter_pipeline_doc(
+            'p4', 'm4', [_read(11, active_read={'enabled': False, 'cycle_multiple': 1})]
+        ),
+        scouter_pipeline_doc('p5', 'm5', [_read(14, active_read={'enabled': True})]),
+    ]
+
+    entries, _issues = _gather(pipelines)
+
+    assert [e['active_read'] for e in entries] == [
+        {'enabled': True, 'cycle_multiple': 2},
+        {'enabled': False},
+        {'enabled': True, 'cycle_multiple': 1},
+    ]
+
+
+def test_gather_read_tags_filters_non_scouter_workflows():
+    """Only scouter pipelines feed the slots"""
+    pipelines = [
+        scouter_pipeline_doc('sin', 'm1', [_read(10)]),
+        scouter_pipeline_doc('pred', 'm2', [_read(11)], workflow_type='predictions_batch'),
+        scouter_pipeline_doc('retrain', 'm3', [_read(14)], workflow_type='minimal_retrain'),
+        scouter_pipeline_doc('drift', 'm4', [_read(7)], workflow_type='drift'),
+    ]
+
+    entries, _issues = _gather(pipelines)
+
+    assert [e['address'] for e in entries] == ['ns=2;s=A']
+
+
+def test_gather_read_tags_without_read_tags():
+    pipeline = scouter_pipeline_doc('sin', 'm1', [])
+    del pipeline['read_tags']
+
+    assert _gather([pipeline]) == ([], NO_ISSUES)
+
+
+def test_build_tag_config():
+    """Slot v2 (contract C1): security, certificate and subscription period per server"""
+    pipelines = [
+        scouter_pipeline_doc('sin', 'm1', [_read(10, 1000), _read(11, '300'), _read(7, 100)])
+    ]
+    entries, _issues = _gather(pipelines)
+
+    result = build_tag_config(entries)
+
+    assert result == {
+        'kepserver': {
+            'server_id': '1',
+            'name': 'kepserver',
+            'url': 'opc.tcp://kepserver:4840',
+            # No client certificate: no security (D5)
+            'security': NO_SECURITY,
+            'certificate': None,
+            # No subscription_period_ms on the connector: half its lowest tag frequency
+            'subscription_period_ms': 150,
+            'tags': {
+                'ns=2;s=A': {
+                    'frequency': 1000,
+                    'active_read': {'enabled': False},
+                    'names': [_name('sin', 'A', 'm1')],
+                },
+                'ns=2;s=B': {
+                    'frequency': 300,
+                    'active_read': {'enabled': False},
+                    'names': [_name('sin', 'B', 'm1')],
+                },
+            },
+        },
+        'plant': {
+            'server_id': '3',
+            'name': 'plant',
+            'url': 'opc.tcp://plant:4840',
+            'subscription_period_ms': 1500,
+            'security': SECURE,
+            'certificate': CERTIFICATE,
+            'tags': {
+                'ns=3;i=1001': {
+                    'frequency': 100,
+                    'active_read': {'enabled': False},
+                    'names': [_name('sin', 'Pressure', 'm1')],
+                },
+            },
+        },
+    }
+
+
+def test_build_tag_config_empty():
+    assert build_tag_config([]) == {}
+
+
+def test_gather_read_tags_invalid_read_tags():
+    """Unusable read tags are reported and skipped; the others still make the slot"""
+    pipelines = [
+        scouter_pipeline_doc('sin', 'm1', [_read(10), {'tag_id': 11}, _read(7, frequency='fast')])
+    ]
+
+    entries, issues = _gather(pipelines)
+
+    assert [e['address'] for e in entries] == ['ns=2;s=A']
+    assert issues['invalid'] == {'sin/11', 'sin/7'}
+
+
+def test_gather_read_tags_frequency_type_does_not_depend_on_order():
+    """15000 and '15000' give the same int frequency in any pipeline order (O6)"""
+    as_int = scouter_pipeline_doc('sin', 'm1', [_read(10, 15000)])
+    as_text = scouter_pipeline_doc('cos', 'm2', [_read(10, '15000')])
+
+    first, _ = _gather([as_int, as_text])
+    second, _ = _gather([as_text, as_int])
+
+    assert first[0]['frequency'] == second[0]['frequency'] == 15000
+    assert isinstance(first[0]['frequency'], int)
+    assert isinstance(second[0]['frequency'], int)
+
+
+def test_gather_read_tags_server_key_never_collides():
+    """A server literally named 'x:3' does not share its key with server 3"""
+    connector_index = {
+        '1': _connector(1, 'x:3'),
+        '2': _connector(2, 'x'),
+        '3': _connector(3, 'x'),
+    }
+    tag_index = {
+        1: _db_tag(1, 1, 'ns=2;s=A', 'a'),
+        2: _db_tag(2, 2, 'ns=2;s=B', 'b'),
+        3: _db_tag(3, 3, 'ns=2;s=C', 'c'),
+    }
+    pipelines = [scouter_pipeline_doc('sin', 'm1', [_read(1), _read(2), _read(3)])]
+
+    entries, _issues = _gather(pipelines, connector_index, tag_index)
+    slot = build_tag_config(entries)
+
+    keys = [e['server']['key'] for e in entries]
+    assert len(set(keys)) == 3
+    assert {server['server_id'] for server in slot.values()} == {'1', '2', '3'}
+    assert all(len(server['tags']) == 1 for server in slot.values())
+
+
+def test_gather_read_tags_invalid_cycle_multiple_does_not_fail_the_slot():
+    """A bad active_read is reported as invalid instead of failing every pipeline's slot"""
+    pipelines = [
+        scouter_pipeline_doc(
+            'sin',
+            'm1',
+            [
+                _read(10),
+                _read(11, active_read={'enabled': True, 'cycle_multiple': 'x'}),
+                _read(7, active_read={'enabled': True, 'cycle_multiple': None}),
+                # Not a dict: treated as no active read
+                _read(14, active_read=True),
+            ],
+        )
+    ]
+
+    entries, issues = _gather(pipelines)
+
+    assert [e['address'] for e in entries] == ['ns=2;s=A', 'ns=2;s=C']
+    assert entries[1]['active_read'] == {'enabled': False}
+    assert issues['invalid'] == {'sin/11', 'sin/7'}
+
+
+def test_gather_read_tags_skips_incomplete_pipelines():
+    """A pipeline without schedule_name, model_id or model.name is reported, not raised"""
+    no_model = scouter_pipeline_doc('no-model', 'm1', [_read(10)])
+    del no_model['model']
+    no_name = scouter_pipeline_doc('no-name', 'm2', [_read(10)])
+    no_name['model'] = {}
+    no_model_id = scouter_pipeline_doc('no-id', None, [_read(10)])
+    no_schedule = scouter_pipeline_doc(None, 'm4', [_read(10)])
+    ok = scouter_pipeline_doc('ok', 'm5', [_read(11)])
+
+    entries, issues = _gather([no_model, no_name, no_model_id, no_schedule, ok])
+
+    assert [e['address'] for e in entries] == ['ns=2;s=B']
+    assert issues['invalid'] == {'no-model', 'no-name', 'no-id', 'None'}
+
+
+def _plant_with(**fields):
+    return {'1': connectors['1'], '3': {**connectors['3'], **fields}}
+
+
+@pytest.mark.parametrize(
+    'fields',
+    [
+        # $lookup found nothing: missing id, inactive or not a client certificate
+        pytest.param(secure_connector_fields(certificates=[]), id='not-found'),
+        pytest.param(
+            secure_connector_fields(certificates=[certificate_doc(application_uri='')]),
+            id='no-application-uri',
+        ),
+        pytest.param(
+            secure_connector_fields(
+                certificates=[{**certificate_doc(), 'private_key': {'bucket': 'b'}}]
+            ),
+            id='key-without-object',
+        ),
+    ],
+)
+def test_gather_read_tags_registered_certificate_missing(fields):
+    """An id without a usable certificate: in the slot without a certificate, with a warning (D5)"""
+    entries, issues = _gather(
+        [scouter_pipeline_doc('sin', 'm1', [_read(7)])], _plant_with(**fields)
+    )
+
+    assert entries[0]['server']['certificate'] is None
+    assert entries[0]['server']['security'] == SECURE
+    assert issues == {**NO_ISSUES, 'certificates': {'3'}}
+
+
+@pytest.mark.parametrize('certificate_id', [None, 'absent'])
+def test_gather_read_tags_without_certificate_id(certificate_id):
+    """No client_certificate_id: no certificate and nothing to report (D5)"""
+    security = {**NO_SECURITY, 'client_certificate_id': certificate_id}
+    if certificate_id == 'absent':
+        security = NO_SECURITY
+    entries, issues = _gather(
+        [scouter_pipeline_doc('sin', 'm1', [_read(7)])],
+        _plant_with(security=security, certificates=[]),
+    )
+
+    assert entries[0]['server']['certificate'] is None
+    assert issues == NO_ISSUES
+
+
+def test_gather_read_tags_reports_a_certificate_once_per_connector():
+    entries, issues = _gather(
+        [
+            scouter_pipeline_doc('sin', 'm1', [_read(7)]),
+            scouter_pipeline_doc('cos', 'm2', [_read(7)]),
+        ],
+        _plant_with(**secure_connector_fields(certificates=[])),
+    )
+
+    assert len(entries) == 1
+    assert issues['certificates'] == {'3'}

tests/orchestrator/utils/test_pipeline_configs.py

@@ -0,0 +1,742 @@
+from unittest.mock import call, patch
+
+from pytest import mark
+
+from orchestrator.utils.pipeline_configs import (
+    base_scouter,
+    chunk_period_warning,
+    common_config,
+    drift,
+    frequency_alert_warning,
+    minimal_retrain,
+    overlap_filter_config,
+    pi_web_api_scouter,
+    predictions_batch,
+    process_path_priority,
+    scouter,
+    simple_metrics,
+)
+
+
+def test_common_config():
+    config = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'model_id': 'test_model_id',
+        'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
+    }
+    result = common_config(config)
+    expected = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'frequency': '1m',
+        'offset': '0m',
+        'max_retry_policy': 1,
+        'model_id': 'test_model_id',
+        'model_name': 'test_model_name',
+        'model_config': {'test_config': 'test_config'},
+        'on_conflict': 'error',
+        'execution_timeout_seconds': 300,
+        'task_timeout_seconds': 300,
+        'runtime': 'legacy',
+    }
+    assert result == expected
+
+
+def test_common_config_preserves_explicit_runtime():
+    config = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'model_id': 'test_model_id',
+        'model': {'name': 'test_model_name'},
+        'runtime': 'tenant-x',
+    }
+    assert common_config(config)['runtime'] == 'tenant-x'
+
+
+def test_drift():
+    config = {
+        'workflow_type': 'drift',
+        'schedule_name': 'test_schedule',
+        'model_id': 'test_model_id',
+        'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
+        'interval_minutes': 120,
+        'drift_metrics': ['kolmogorov_smirnov', 'jensen_shannon'],
+    }
+    result = drift(config)
+    expected = {
+        'workflow_type': 'drift',
+        'schedule_name': 'test_schedule',
+        'frequency': '1m',
+        'offset': '0m',
+        'max_retry_policy': 1,
+        'model_id': 'test_model_id',
+        'model_name': 'test_model_name',
+        'model_config': {'test_config': 'test_config'},
+        'on_conflict': 'error',
+        'schema': 'sientia_data',
+        'source_table_name': 'laborious_data',
+        'target_table_name': 'drift_metrics',
+        'interval': 120,
+        'chunk_period': 'min',
+        'drift_metrics': ['kolmogorov_smirnov', 'jensen_shannon'],
+        'execution_timeout_seconds': 300,
+        'task_timeout_seconds': 300,
+        'runtime': 'legacy',
+    }
+    assert result == expected
+
+
+def _drift_config(**fields):
+    return {
+        'workflow_type': 'drift',
+        'schedule_name': 'm1-drift',
+        'model_id': 'm1',
+        'model': {'name': 'm1'},
+        **fields,
+    }
+
+
+def test_drift_default_window_is_frequency():
+    result = drift(_drift_config())
+
+    assert result['frequency'] == '1m'
+    assert result['interval'] == 1
+
+
+@mark.parametrize(
+    ('frequency', 'interval'),
+    [('1h', 60), ('30s', 1), ('90s', 2)],
+)
+def test_drift_window_derived_from_frequency(frequency, interval):
+    assert drift(_drift_config(frequency=frequency))['interval'] == interval
+
+
+def test_drift_explicit_interval_minutes_wins():
+    result = drift(_drift_config(frequency='1m', interval_minutes=60))
+
+    assert result['interval'] == 60
+
+
+def test_drift_invalid_frequency_falls_back_to_legacy_window():
+    result = drift(_drift_config(frequency='abc'))
+
+    assert result['frequency'] == 'abc'
+    assert result['interval'] == 60
+
+
+def test_drift_chunk_period_defaults_to_min():
+    result = drift(_drift_config())
+
+    assert result['chunk_period'] == 'min'
+    assert 'chunk_period_configured' not in result
+
+
+def test_drift_chunk_period_seconds_kept():
+    result = drift(_drift_config(chunk_period='s'))
+
+    assert result['chunk_period'] == 's'
+    assert 'chunk_period_configured' not in result
+
+
+@mark.parametrize('chunk_period', ['hour', 5])
+def test_drift_invalid_chunk_period_falls_back_to_min(chunk_period):
+    result = drift(_drift_config(chunk_period=chunk_period))
+
+    assert result['chunk_period'] == 'min'
+    assert result['chunk_period_configured'] == chunk_period
+
+
+@mark.parametrize('chunk_period', [None, 'min', 's'])
+def test_chunk_period_warning_none_when_absent_or_valid(chunk_period):
+    assert chunk_period_warning(chunk_period) is None
+
+
+@mark.parametrize('chunk_period', ['hour', 5])
+def test_chunk_period_warning_message_when_invalid(chunk_period):
+    warning = chunk_period_warning(chunk_period)
+
+    assert warning is not None
+    assert repr(chunk_period) in warning
+
+
+def test_simple_metrics():
+    config = {
+        'workflow_type': 'simple_metrics',
+        'schedule_name': 'test_schedule',
+        'model_id': 'test_model_id',
+        'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
+        'interval_minutes': 120,
+        'metrics': ['rmse', 'mse'],
+    }
+    result = simple_metrics(config)
+    expected = {
+        'workflow_type': 'simple_metrics',
+        'schedule_name': 'test_schedule',
+        'frequency': '1m',
+        'offset': '0m',
+        'max_retry_policy': 1,
+        'model_id': 'test_model_id',
+        'model_name': 'test_model_name',
+        'model_config': {'test_config': 'test_config'},
+        'on_conflict': 'error',
+        'schema': 'sientia_data',
+        'predictions_table_name': 'predictions',
+        'data_table_name': 'laborious_data',
+        'target_table_name': 'simple_metrics',
+        'interval_minutes': 120,
+        'metrics': ['rmse', 'mse'],
+        'frequency_alert': None,
+        'execution_timeout_seconds': 300,
+        'task_timeout_seconds': 300,
+        'runtime': 'legacy',
+    }
+    assert result == expected
+
+
+def _simple_metrics_config(**fields):
+    return {
+        'workflow_type': 'simple_metrics',
+        'schedule_name': 'm1-simple_metrics-60m',
+        'model_id': 'm1',
+        'model': {'name': 'm1'},
+        'frequency': '1h',
+        **fields,
+    }
+
+
+def test_simple_metrics_escalated_uses_frequency_alert():
+    config = _simple_metrics_config(
+        frequency_alert='5m', adaptive_state={'escalated': True, 'healthy_streak': 0}
+    )
+    result = simple_metrics(config)
+    assert result['frequency'] == '5m'
+    assert result['frequency_alert'] == '5m'
+
+
+def test_simple_metrics_not_escalated_keeps_frequency():
+    config = _simple_metrics_config(
+        frequency_alert='5m', adaptive_state={'escalated': False, 'healthy_streak': 2}
+    )
+    assert simple_metrics(config)['frequency'] == '1h'
+
+
+def test_simple_metrics_escalated_without_frequency_alert_keeps_frequency():
+    # Removing frequency_alert turns adaptive mode off even if a stale state says escalated.
+    config = _simple_metrics_config(adaptive_state={'escalated': True, 'healthy_streak': 0})
+    assert simple_metrics(config)['frequency'] == '1h'
+
+
+def test_simple_metrics_frequency_alert_without_state_keeps_frequency():
+    assert simple_metrics(_simple_metrics_config(frequency_alert='5m'))['frequency'] == '1h'
+
+
+def test_simple_metrics_escalated_unparseable_frequency_alert_keeps_frequency():
+    # An unparseable frequency would make the Temporal schedule create/update fail.
+    config = _simple_metrics_config(frequency_alert='5min', adaptive_state={'escalated': True})
+    assert simple_metrics(config)['frequency'] == '1h'
+
+
+def test_simple_metrics_escalated_below_floor_frequency_alert_is_applied():
+    # Decision 7: below the floor only warns, the value is kept.
+    config = _simple_metrics_config(frequency_alert='30s', adaptive_state={'escalated': True})
+    assert simple_metrics(config)['frequency'] == '30s'
+
+
+@mark.parametrize('adaptive_state', ['yes', {'escalated': 'false'}])
+def test_simple_metrics_malformed_adaptive_state_keeps_frequency(adaptive_state):
+    config = _simple_metrics_config(frequency_alert='5m', adaptive_state=adaptive_state)
+    assert simple_metrics(config)['frequency'] == '1h'
+
+
+def test_minimal_retrain():
+    config = {
+        'workflow_type': 'minimal_retrain',
+        'schedule_name': 'test_schedule',
+        'model_id': 'test_model_id',
+        'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
+        'query': 'select * from sientia_data.laborious_data order by "timestamp" desc limit 30;',
+        'datetime_columns': ['timestamp'],
+    }
+    result = minimal_retrain(config)
+    expected = {
+        'workflow_type': 'minimal_retrain',
+        'schedule_name': 'test_schedule',
+        'frequency': '1m',
+        'offset': '0m',
+        'max_retry_policy': 1,
+        'model_id': 'test_model_id',
+        'model_name': 'test_model_name',
+        'model_config': {'test_config': 'test_config'},
+        'on_conflict': 'error',
+        'query': 'select * from sientia_data.laborious_data order by "timestamp" desc limit 30;',
+        'schema': 'sientia_data',
+        'table_name': 'log_retrain',
+        'datetime_columns': ['timestamp'],
+        'execution_timeout_seconds': 300,
+        'task_timeout_seconds': 300,
+        'runtime': 'legacy',
+    }
+    assert result == expected
+
+
+def test_scouter():
+    config = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'model_id': 'test_model_id',
+        'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
+        'filters': [{'filter_name': 'test_filter_name', 'policy': 'test_policy'}],
+        'read_tags': [
+            {'tag_name': 'test_tag_name', 'aggr_func': 'test_aggr_func', 'data_range': [1, 2]}
+        ],
+        'tag_retention_minutes': 10,
+        'execution_timeout_seconds': 300,
+        'task_timeout_seconds': 300,
+        'runtime': 'legacy',
+    }
+    result = scouter(config)
+    expected = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'frequency': '1m',
+        'offset': '0m',
+        'max_retry_policy': 1,
+        'model_id': 'test_model_id',
+        'model_name': 'test_model_name',
+        'model_config': {'test_config': 'test_config'},
+        'on_conflict': 'error',
+        'topic': 'raw_test_schedule',
+        'trigger_laborious': False,
+        'filters': {'test_filter_name': {'policy': 'test_policy'}},
+        'schema': 'sientia_data',
+        'table_name': 'laborious_data',
+        'retention_time': 10 * 60,
+        'model_tags': {'test_tag_name': {'aggr_func': 'test_aggr_func', 'data_range': [1, 2]}},
+        'debug_data_package': False,
+        'execution_timeout_seconds': 300,
+        'task_timeout_seconds': 300,
+        'fill_missing_tags': False,
+        'runtime': 'legacy',
+    }
+    assert result == expected
+
+
+def test_overlap_filter_config():
+    config = [
+        {'filter_name': 'test_filter_name', 'policy': 'test_policy'},
+        {'filter_name': 'test_filter_name2', 'policy': 'test_policy2'},
+    ]
+    result = overlap_filter_config({'test_filter_name': {'policy': 'test_policy'}}, config)
+    expected = {
+        'test_filter_name': {'policy': 'test_policy', 'config': {}},
+        'test_filter_name2': {'policy': 'test_policy2', 'config': {}},
+    }
+    assert result == expected
+
+
+def test_process_path_priority():
+    config = ['OTHER', 'STOP', 'CONTINUE']
+    result = process_path_priority(config)
+    expected = ['STOP', 'CONTINUE', 'REPEAT']
+    assert result == expected
+
+
+@patch(
+    'orchestrator.utils.pipeline_configs.overlap_filter_config',
+    return_value={'test_filter_name': {'policy': 'test_policy', 'config': {}}},
+)
+@patch(
+    'orchestrator.utils.pipeline_configs.process_path_priority',
+    return_value=['STOP', 'CONTINUE', 'REPEAT'],
+)
+def test_predictions_batch(mock_process_path_priority, mock_overlap_filter_config):
+    config = {
+        'schedule_name': 'test_schedule',
+        'workflow_type': 'predictions_batch',
+        'model_id': 'test_model_id',
+        'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
+        'query': 'test_query',
+        'write_tags': [{'tag_id': 77}, {'tag_id': 78}],
+        # Resolved from write_tags by process_schedules
+        'opc_output_config': {
+            'test_server_id': {
+                'prediction_tags': {'test_addr': {'data_type': 'float'}},
+                'confidence_tags': {'test_addr': {'data_type': 'float'}},
+            }
+        },
+        'pi_web_api_output_config': {
+            'endpoint': 'test_endpoint',
+            'prediction_tags': {
+                'tag_1': 'webid_1',
+            },
+            'confidence_tags': {
+                'tag_2': 'webid_2',
+            },
+        },
+        'input_filters': [{'filter_name': 'test_filter_name', 'policy': 'test_policy'}],
+        'mlflow_transform_filters': [{'filter_name': 'test_filter_name', 'policy': 'test_policy'}],
+        'mlflow_predict_filters': [{'filter_name': 'test_filter_name', 'policy': 'test_policy'}],
+        'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
+        'datetime_columns': ['timestamp'],
+        'predictions_storage_policy': 'erl:1',
+    }
+
+    result = predictions_batch(config)
+
+    mock_overlap_filter_config.assert_has_calls(
+        [call({'EMPTY_DATA': {'policy': 'STOP', 'config': {}}}, config['input_filters'])]
+    )
+    mock_overlap_filter_config.assert_has_calls(
+        [
+            call(
+                {
+                    'EMPTY_DATA': {'policy': 'STOP', 'config': {}},
+                    'API_ERROR': {'policy': 'STOP', 'config': {}},
+                },
+                config['mlflow_transform_filters'],
+            )
+        ]
+    )
+    mock_overlap_filter_config.assert_has_calls(
+        [call({'API_ERROR': {'policy': 'STOP', 'config': {}}}, config['mlflow_predict_filters'])]
+    )
+    mock_process_path_priority.assert_called_once_with(config['path_priority'])
+
+    expected = {
+        'workflow_type': 'predictions_batch',
+        'schedule_name': 'test_schedule',
+        'frequency': '1m',
+        'offset': '0m',
+        'max_retry_policy': 1,
+        'model_id': 'test_model_id',
+        'model_name': 'test_model_name',
+        'model_config': {'test_config': 'test_config'},
+        'on_conflict': 'error',
+        'query': 'test_query',
+        'schema': 'sientia_data',
+        'table_name': 'predictions',
+        'save_transform': True,
+        'transform_table_name': 'transformed_data',
+        'retention_time': 60 * 60,
+        'opc_output_config': {
+            'test_server_id': {
+                'prediction_tags': {'test_addr': {'data_type': 'float'}},
+                'confidence_tags': {'test_addr': {'data_type': 'float'}},
+            }
+        },
+        'pi_web_api_output_config': {
+            'endpoint': 'test_endpoint',
+            'prediction_tags': {
+                'tag_1': 'webid_1',
+            },
+            'confidence_tags': {
+                'tag_2': 'webid_2',
+            },
+        },
+        'input_filters': {'test_filter_name': {'policy': 'test_policy', 'config': {}}},
+        'mlflow_transform_filters': {'test_filter_name': {'policy': 'test_policy', 'config': {}}},
+        'mlflow_predict_filters': {'test_filter_name': {'policy': 'test_policy', 'config': {}}},
+        'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
+        'datetime_columns': ['timestamp'],
+        'predictions_storage_policy': 'erl:1',
+        'execution_timeout_seconds': 300,
+        'task_timeout_seconds': 300,
+        'runtime': 'legacy',
+    }
+    assert result == expected
+
+
+def test_scouter_skips_tags_without_name():
+    """A tag_id the tag database cannot resolve has no name and stays out of model_tags"""
+    config = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'sin',
+        'model_id': 'm1',
+        'model': {'name': 'model m1'},
+        'read_tags': [{'tag_name': 'a'}, {'tag_id': 404}],
+    }
+
+    assert list(scouter(config)['model_tags']) == ['a']
+
+
+def test_base_scouter():
+    config = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'model_id': 'test_model_id',
+        'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
+        'filters': [
+            {'filter_name': 'test_filter_name', 'policy': 'test_policy'},
+            {'filter_name': 'test_filter_name2', 'policy': 'test_policy2'},
+        ],
+        'tag_retention_minutes': 30,
+        'debug_data_package': True,
+        'fill_missing_tags': True,
+    }
+    result = base_scouter(config)
+    expected = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'frequency': '1m',
+        'offset': '0m',
+        'max_retry_policy': 1,
+        'model_id': 'test_model_id',
+        'model_name': 'test_model_name',
+        'model_config': {'test_config': 'test_config'},
+        'on_conflict': 'error',
+        'trigger_laborious': False,
+        'filters': {
+            'test_filter_name': {'policy': 'test_policy'},
+            'test_filter_name2': {'policy': 'test_policy2'},
+        },
+        'schema': 'sientia_data',
+        'table_name': 'laborious_data',
+        'retention_time': 30 * 60,
+        'debug_data_package': True,
+        'execution_timeout_seconds': 300,
+        'task_timeout_seconds': 300,
+        'fill_missing_tags': True,
+        'runtime': 'legacy',
+    }
+    assert result == expected
+
+
+def test_pi_web_api_scouter():
+    config = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'model_id': 'test_model_id',
+        'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
+        'filters': [{'filter_name': 'test_filter_name', 'policy': 'test_policy'}],
+        'read_tags': {
+            'test_tag_name': {
+                'webid': 'test_webid',
+                'aggr_func': 'test_aggr_func',
+                'data_range': [1, 2],
+            }
+        },
+        'tag_retention_minutes': 10,
+        'pi_web_api_config': {
+            'endpoint': 'https://test-endpoint.com',
+            'period': '*-2d',
+            'max_count': 5,
+            'api_timeout': 30,
+        },
+        'frequency': '1m',
+    }
+    result = pi_web_api_scouter(config)
+    expected = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'frequency': '1m',
+        'offset': '0m',
+        'max_retry_policy': 1,
+        'model_id': 'test_model_id',
+        'model_name': 'test_model_name',
+        'model_config': {'test_config': 'test_config'},
+        'on_conflict': 'error',
+        'trigger_laborious': False,
+        'filters': {'test_filter_name': {'policy': 'test_policy'}},
+        'schema': 'sientia_data',
+        'table_name': 'laborious_data',
+        'retention_time': 10 * 60,
+        'model_tags': {
+            'test_tag_name': {
+                'webid': 'test_webid',
+                'aggr_func': 'test_aggr_func',
+                'data_range': [1, 2],
+            }
+        },
+        'debug_data_package': False,
+        'execution_timeout_seconds': 300,
+        'task_timeout_seconds': 300,
+        'fill_missing_tags': False,
+        'pi_web_api_query': {
+            'endpoint': 'https://test-endpoint.com',
+            'period': '*-2d',
+            'max_count': 5,
+            'api_timeout': 30,
+        },
+        'runtime': 'legacy',
+    }
+    assert result == expected
+
+
+def test_pi_web_api_scouter_with_timeout_greater_than_frequency():
+    config = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'model_id': 'test_model_id',
+        'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
+        'filters': [],
+        'read_tags': {
+            'test_tag_name': {
+                'webid': 'test_webid',
+            }
+        },
+        'tag_retention_minutes': 10,
+        'pi_web_api_config': {
+            'endpoint': 'https://test-endpoint.com',
+            'api_timeout': 120,
+        },
+        'frequency': '1m',
+    }
+    result = pi_web_api_scouter(config)
+    expected = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'frequency': '1m',
+        'offset': '0m',
+        'max_retry_policy': 1,
+        'model_id': 'test_model_id',
+        'model_name': 'test_model_name',
+        'model_config': {'test_config': 'test_config'},
+        'on_conflict': 'error',
+        'trigger_laborious': False,
+        'filters': {},
+        'schema': 'sientia_data',
+        'table_name': 'laborious_data',
+        'retention_time': 10 * 60,
+        'model_tags': {
+            'test_tag_name': {'webid': 'test_webid', 'aggr_func': 'lts', 'data_range': [-100, 100]}
+        },
+        'debug_data_package': False,
+        'execution_timeout_seconds': 300,
+        'task_timeout_seconds': 300,
+        'fill_missing_tags': False,
+        'pi_web_api_query': {
+            'endpoint': 'https://test-endpoint.com',
+            'period': '*-1d',
+            'max_count': 1,
+            'api_timeout': 60,
+        },
+        'runtime': 'legacy',
+    }
+    assert result == expected
+
+
+def test_pi_web_api_scouter_with_no_timeout():
+    config = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'model_id': 'test_model_id',
+        'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
+        'filters': [],
+        'read_tags': {
+            'test_tag_name': {
+                'webid': 'test_webid',
+            }
+        },
+        'tag_retention_minutes': 10,
+        'pi_web_api_config': {
+            'endpoint': 'https://test-endpoint.com',
+        },
+        'frequency': '30s',
+    }
+    result = pi_web_api_scouter(config)
+    expected = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'test_schedule',
+        'frequency': '30s',
+        'offset': '0m',
+        'max_retry_policy': 1,
+        'model_id': 'test_model_id',
+        'model_name': 'test_model_name',
+        'model_config': {'test_config': 'test_config'},
+        'on_conflict': 'error',
+        'trigger_laborious': False,
+        'filters': {},
+        'schema': 'sientia_data',
+        'table_name': 'laborious_data',
+        'retention_time': 10 * 60,
+        'model_tags': {
+            'test_tag_name': {'webid': 'test_webid', 'aggr_func': 'lts', 'data_range': [-100, 100]}
+        },
+        'debug_data_package': False,
+        'execution_timeout_seconds': 300,
+        'task_timeout_seconds': 300,
+        'fill_missing_tags': False,
+        'pi_web_api_query': {
+            'endpoint': 'https://test-endpoint.com',
+            'period': '*-1d',
+            'max_count': 1,
+            'api_timeout': 30,
+        },
+        'runtime': 'legacy',
+    }
+    assert result == expected
+
+
+@mark.parametrize(
+    'frequency_alert, warns',
+    [
+        (None, False),
+        ('5m', False),
+        ('1m', False),  # the floor itself is allowed
+        ('60s', False),
+        ('30s', True),
+        ('5min', True),  # unparseable
+        (5, True),  # not a string: must warn, never raise
+    ],
+)
+def test_frequency_alert_warning(frequency_alert, warns):
+    assert (frequency_alert_warning(frequency_alert) is not None) is warns
+
+
+def test_pi_web_api_scouter_caps_api_timeout_at_frequency_only_when_larger():
+    config = {
+        'workflow_type': 'pi_web_api_scouter',
+        'schedule_name': 'pi_schedule',
+        'model_id': 'model_id',
+        'model': {'name': 'model_name'},
+        'frequency': '1m',
+        'read_tags': {'TI-101': {'webid': 'F1DP-X'}},
+        'pi_web_api_config': {
+            'endpoint': '/streamsets/recorded',
+            'max_count': 5,
+            'api_timeout': 10,
+        },
+    }
+
+    result = pi_web_api_scouter(config)
+
+    assert result['model_tags'] == {
+        'TI-101': {'webid': 'F1DP-X', 'aggr_func': 'lts', 'data_range': [-100, 100]}
+    }
+    assert result['pi_web_api_query'] == {
+        'endpoint': '/streamsets/recorded',
+        'period': '*-1d',
+        'max_count': 5,
+        'api_timeout': 10,
+    }
+
+
+def test_predictions_batch_copies_the_resolved_opc_output_config():
+    opc_output_config = {'plant': {'prediction_tags': {'ns=2;s=P1': {'data_type': 'int'}}}}
+    config = {
+        'workflow_type': 'predictions_batch',
+        'schedule_name': 'batch',
+        'model_id': 'model_id',
+        'model': {'name': 'model_name'},
+        'query': 'SELECT 1',
+        'write_tags': [{'tag_id': 77}],
+        'opc_output_config': opc_output_config,
+    }
+
+    result = predictions_batch(config)
+
+    assert result['opc_output_config'] == opc_output_config
+    assert 'write_tags' not in result
+
+
+def test_predictions_batch_without_write_tags_has_no_opc_output():
+    config = {
+        'workflow_type': 'predictions_batch',
+        'schedule_name': 'batch',
+        'model_id': 'model_id',
+        'model': {'name': 'model_name'},
+        'query': 'SELECT 1',
+    }
+
+    assert predictions_batch(config)['opc_output_config'] == {}

tests/orchestrator/utils/test_tag_database.py

@@ -0,0 +1,279 @@
+import pytest
+
+from orchestrator.utils.tag_database import (
+    WRITE_CATEGORIES,
+    client_certificate_id,
+    filter_by_tag_ids,
+    index_connector_tags,
+    index_connectors,
+    index_tag_documents,
+    referenced_tag_ids,
+    resolve_certificate,
+    resolve_tag_names,
+    resolve_write_tags,
+    tag_id_of,
+)
+from tests.orchestrator.utils.tag_database_fixtures import (
+    CERTIFICATE,
+    NO_SECURITY,
+    OPC_CONNECTORS,
+    TAGS_BY_ID,
+    batch_pipeline_doc,
+    certificate_doc,
+    scouter_pipeline_doc,
+    secure_connector_fields,
+    write_tag_doc,
+)
+
+
+def test_index_connector_tags():
+    tags = [
+        {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1', 'tag_name': 'a'},
+        # tag_id is int in connector_tags: a text id is no tag
+        {'tag_id': '8', 'connector_id': 3, 'address': 'ns=3;i=2', 'tag_name': 'b'},
+    ]
+
+    assert index_connector_tags(tags) == {7: tags[0]}
+
+
+def test_resolve_tag_names():
+    read_tags = [
+        {'tag_id': 7},
+        {'tag_id': 7, 'tag_name': 'own name'},
+        {'tag_id': '7'},
+        {'tag_id': 404},
+        {'tag_name': 'no tag_id'},
+    ]
+
+    assert resolve_tag_names(read_tags, TAGS_BY_ID) == [
+        {'tag_id': 7, 'tag_name': 'Pressure'},
+        {'tag_id': 7, 'tag_name': 'own name'},
+        {'tag_id': '7'},
+        {'tag_id': 404},
+        {'tag_name': 'no tag_id'},
+    ]
+
+
+def test_referenced_tag_ids():
+    """Int ids only, scouter pipelines only; text, empty or null tag_id ignored"""
+    pipelines = [
+        scouter_pipeline_doc(
+            'sin', 'm1', [{'tag_id': 7}, {'tag_id': '12'}, {'tag_id': ''}, {'tag_id': None}]
+        ),
+        scouter_pipeline_doc(
+            'cos', 'm2', [{'tag_id': 'abc'}, {'tag_name': 'a', 'frequency': 1000}]
+        ),
+        scouter_pipeline_doc('pred', 'm3', [{'tag_id': 99}], workflow_type='predictions_batch'),
+    ]
+
+    # Sorted, so the query is the same on every run
+    assert referenced_tag_ids(pipelines) == [7]
+
+
+def test_referenced_tag_ids_without_tag_ids():
+    pipeline = scouter_pipeline_doc('sin', 'm1', [{'tag_name': 'a', 'frequency': 1000}])
+    no_read_tags = scouter_pipeline_doc('cos', 'm2', [])
+    del no_read_tags['read_tags']
+
+    assert referenced_tag_ids([pipeline, no_read_tags]) == []
+
+
+def test_filter_by_tag_ids():
+    """A tag_id $match goes in front, on a new aggregation list"""
+    query = {'collection': 'connector_tags', 'aggregation': [{'$project': {'_id': 0}}]}
+
+    result = filter_by_tag_ids(query, [7, 8])
+
+    assert result == {
+        'collection': 'connector_tags',
+        'aggregation': [{'$match': {'tag_id': {'$in': [7, 8]}}}, {'$project': {'_id': 0}}],
+    }
+    assert result['aggregation'] is not query['aggregation']
+    assert query['aggregation'] == [{'$project': {'_id': 0}}]
+
+
+def test_index_skips_incomplete_documents():
+    """A malformed tag database document is left out instead of failing the run"""
+    tags = [
+        {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1', 'tag_name': 'a'},
+        {'connector_id': 3, 'address': 'ns=3;i=2', 'tag_name': 'no id'},
+        {'tag_id': 9, 'connector_id': 3, 'address': '', 'tag_name': 'no address'},
+    ]
+    connector_list = [
+        {'connector_id': 3, 'name': 'plant', 'url': 'opc.tcp://p:4840', 'security': NO_SECURITY},
+        {'connector_id': 4, 'name': 'no url', 'security': NO_SECURITY},
+        {'name': 'no id', 'url': 'opc.tcp://x:4840', 'security': NO_SECURITY},
+        # security.mode and security.policy are the source of the OPC security (D4)
+        {'connector_id': 5, 'name': 'no security', 'url': 'opc.tcp://x:4840'},
+        {
+            'connector_id': 6,
+            'name': 'no mode',
+            'url': 'opc.tcp://x:4840',
+            'security': {'policy': 'None'},
+        },
+        {
+            'connector_id': 7,
+            'name': 'no policy',
+            'url': 'opc.tcp://x:4840',
+            'security': {'mode': 'None'},
+        },
+    ]
+
+    assert list(index_connector_tags(tags)) == [7]
+    assert list(index_connectors(connector_list)) == ['3']
+
+
+def test_tag_id_of_accepts_only_int():
+    """connector_tags stores tag_id as int: anything else is no reference"""
+    assert tag_id_of({'tag_id': 7}) == 7
+    assert tag_id_of({'tag_id': '7'}) is None
+    assert tag_id_of({'tag_id': 'abc'}) is None
+    assert tag_id_of({'tag_id': 7.0}) is None
+    assert tag_id_of({'tag_id': True}) is None
+    assert tag_id_of({'tag_id': ''}) is None
+    assert tag_id_of({'tag_id': None}) is None
+    assert tag_id_of({}) is None
+    assert tag_id_of('ns=2;s=raw') is None
+
+
+def test_referenced_tag_ids_includes_write_tags():
+    pipelines = [
+        scouter_pipeline_doc('sin', 'm1', [{'tag_id': 7}]),
+        batch_pipeline_doc('pred', [{'tag_id': 77}, {'tag_id': 78}, {'server_id': '1'}]),
+        batch_pipeline_doc('xgb', [{'tag_id': 79}], workflow_type='xgboost_predictions_batch'),
+        batch_pipeline_doc('retrain', [{'tag_id': 99}], workflow_type='minimal_retrain'),
+    ]
+
+    assert referenced_tag_ids(pipelines) == [7, 77, 78, 79]
+
+
+def test_index_connector_tags_filters_status_and_category():
+    tags = [
+        write_tag_doc(1, 'ns=2;s=R', category='Read'),
+        write_tag_doc(2, 'ns=2;s=W'),
+        write_tag_doc(3, 'ns=2;s=C', category='Prediction Confidence'),
+        write_tag_doc(4, 'ns=2;s=I', status='Inactive'),
+        # An override that projects neither status nor category
+        {'tag_id': 5, 'connector_id': 'plant', 'address': 'ns=2;s=N', 'tag_name': 'n'},
+    ]
+
+    assert list(index_connector_tags(tags)) == [1, 5]
+    assert list(index_connector_tags(tags, WRITE_CATEGORIES)) == [2, 3, 5]
+    # Every document, whatever its status or category
+    assert list(index_tag_documents(tags)) == [1, 2, 3, 4, 5]
+
+
+def test_index_connectors_leaves_out_inactive():
+    connectors = [
+        {**OPC_CONNECTORS['plant'], 'status': 'Active'},
+        {**OPC_CONNECTORS['plant'], 'connector_id': 'old', 'status': 'Inactive'},
+    ]
+
+    assert list(index_connectors(connectors)) == ['plant']
+
+
+def test_resolve_write_tags():
+    tags = index_connector_tags(
+        [
+            write_tag_doc(77, 'ns=2;s=P1'),
+            write_tag_doc(78, 'ns=2;s=C1', category='Prediction Confidence', data_type='double'),
+            write_tag_doc(79, 'ns=2;s=P2', connector_id='other', data_type='int'),
+        ],
+        WRITE_CATEGORIES,
+    )
+    connectors = {**OPC_CONNECTORS, 'other': {**OPC_CONNECTORS['plant'], 'connector_id': 'other'}}
+
+    output_config, issues = resolve_write_tags(
+        [{'tag_id': 77}, {'tag_id': 78}, {'tag_id': 79}], tags, connectors
+    )
+
+    assert output_config == {
+        'plant': {
+            'prediction_tags': {'ns=2;s=P1': {'data_type': 'float'}},
+            'confidence_tags': {'ns=2;s=C1': {'data_type': 'double'}},
+        },
+        'other': {'prediction_tags': {'ns=2;s=P2': {'data_type': 'int'}}},
+    }
+    assert not any(issues.values())
+
+
+def test_resolve_write_tags_issues():
+    tags = index_connector_tags(
+        [
+            write_tag_doc(1, 'ns=2;s=R', category='Read'),
+            write_tag_doc(2, 'ns=2;s=X', connector_id='gone'),
+            write_tag_doc(3, 'ns=2;s=B', data_type='Double'),
+            write_tag_doc(4, 'ns=2;s=I', status='Inactive'),
+            # No category: a write tag cannot tell prediction from confidence
+            {'tag_id': 5, 'connector_id': 'plant', 'address': 'ns=2;s=N', 'tag_name': 'n'},
+            write_tag_doc(6, 'ns=2;s=OK'),
+        ],
+        WRITE_CATEGORIES,
+    )
+    write_tags = [
+        {'tag_id': 1},
+        {'tag_id': 2},
+        {'tag_id': 3},
+        {'tag_id': 4},
+        {'tag_id': 5},
+        {'tag_id': 404},
+        # Every write tag is a tag_id
+        {'tag_name': 'L'},
+        {'tag_id': None},
+        # tag_id is int: a text id is invalid
+        {'tag_id': '6'},
+        'ns=2;s=raw',
+        {'tag_id': 6},
+    ]
+
+    output_config, issues = resolve_write_tags(write_tags, tags, OPC_CONNECTORS)
+
+    assert output_config == {'plant': {'prediction_tags': {'ns=2;s=OK': {'data_type': 'float'}}}}
+    assert issues['tags'] == {'1', '4', '5', '404'}
+    assert issues['connectors'] == {'gone'}
+    assert issues['invalid'] == {
+        '3',
+        "{'tag_name': 'L'}",
+        "{'tag_id': '6'}",
+        "{'tag_id': None}",
+        'ns=2;s=raw',
+    }
+
+
+def test_resolve_certificate():
+    """Only the fields the gateway needs: dates and other fields stay out of the slot"""
+    connector = {
+        **OPC_CONNECTORS['plant'],
+        **secure_connector_fields(
+            certificates=[certificate_doc(thumbprint_sha1='ab', not_after=None)]
+        ),
+    }
+
+    assert resolve_certificate(connector) == CERTIFICATE
+    assert client_certificate_id(connector) == 1
+
+
+@pytest.mark.parametrize(
+    'certificates',
+    [
+        pytest.param(None, id='no-lookup'),
+        pytest.param([], id='not-found'),
+        pytest.param([certificate_doc(application_uri=None)], id='no-application-uri'),
+        pytest.param([{**certificate_doc(), 'cert': None}], id='no-cert'),
+        pytest.param(
+            [{**certificate_doc(), 'private_key': {'object_name': 'k'}}], id='key-no-bucket'
+        ),
+        pytest.param(['not a document'], id='not-a-document'),
+    ],
+)
+def test_resolve_certificate_without_a_usable_certificate(certificates):
+    connector = {**OPC_CONNECTORS['plant'], **secure_connector_fields()}
+    connector['certificates'] = certificates
+
+    assert resolve_certificate(connector) is None
+
+
+def test_client_certificate_id_without_one():
+    assert client_certificate_id(OPC_CONNECTORS['plant']) is None
+    assert client_certificate_id({'connector_id': 'x'}) is None
+    assert client_certificate_id({'security': None}) is None

tests/orchestrator/worker/test_worker.py

@@ -3,7 +3,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
 
 from pytest import mark
 
-from orchestrator.worker.worker import main
+from orchestrator.worker.worker import main, start_prometheus_server
 from orchestrator.workflows.alerts import Alerts
 from orchestrator.workflows.orchestrator import Orchestrator
 from orchestrator.workflows.reports import Reports
@@ -47,3 +47,43 @@ async def test_main_starts_three_workers_for_expected_workflows(
     assert main_workflows == [Orchestrator, Alerts, Reports]
 
     assert worker_mock.run.call_count == 3
+
+
+@mark.asyncio
+@patch('orchestrator.worker.worker.Runtime')
+@patch('orchestrator.worker.worker.sys')
+@patch('orchestrator.worker.worker.start_http_server')
+@patch('orchestrator.worker.worker.NotificationHandler')
+@patch('orchestrator.worker.worker.Activities')
+@patch('orchestrator.worker.worker.prepare_worker')
+@patch('orchestrator.worker.worker.client.Client.connect', new_callable=AsyncMock)
+async def test_main_exits_with_error_when_a_worker_fails(
+    _connect_mock,
+    prepare_worker_mock,
+    activities_mock,
+    notification_handler_mock,
+    _start_http_server,
+    sys_mock,
+    _runtime_mock,
+):
+    """A worker failure shuts everything down and exits with code 1."""
+    activities_instance = activities_mock.return_value
+    activities_instance.connect_to_temporal = AsyncMock()
+
+    worker_mock = MagicMock()
+    worker_mock.run = AsyncMock(side_effect=RuntimeError('worker crashed'))
+    prepare_worker_mock.return_value = worker_mock
+
+    await main()
+
+    notification_handler_mock.return_value.shutdown.assert_called_once()
+    activities_instance.shutdown.assert_called_once()
+    sys_mock.exit.assert_called_once_with(1)
+
+
+@patch('orchestrator.worker.worker.os._exit')
+@patch('orchestrator.worker.worker.start_http_server', side_effect=OSError('port in use'))
+def test_start_prometheus_server_exits_on_failure(_start_http_server, exit_mock):
+    start_prometheus_server()
+
+    exit_mock.assert_called_once_with(1)

tests/orchestrator/workflows/test_orchestrator.py

@@ -1,4 +1,4 @@
-from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
+from unittest.mock import ANY, AsyncMock, call, patch
 
 from pytest import fixture, mark
 
@@ -6,7 +6,6 @@ from orchestrator.activities.activities import Activities
 from orchestrator.workflows.orchestrator import (
     DEFAULT_CONNECTOR_TAGS_QUERY,
     DEFAULT_CONNECTORS_QUERY,
-    TAG_DATABASE_PATCH,
     Orchestrator,
 )
 
@@ -30,10 +29,8 @@ metadata = {
 @patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[7, '7'])
 @patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
 async def test_run(workflow_mock, referenced_tag_ids_mock, orchestrator):
-    workflow_mock.patched = MagicMock(return_value=True)
     input_data = {
         'pipelines_query': 'SELECT * FROM bucket',
-        'opc_servers_query': 'SELECT * FROM servers',
         'schedule_name': 'test-schedule-name',
     }
 
@@ -54,16 +51,13 @@ async def test_run(workflow_mock, referenced_tag_ids_mock, orchestrator):
         ]
     )
 
-    workflow_mock.start_local_activity_method.assert_has_calls(
-        [
-            call(
-                Activities.find_documents_in_mongodb,
-                {**metadata, 'query': input_data['opc_servers_query']},
-                retry_policy=ANY,
-                start_to_close_timeout=ANY,
-            )
-        ]
-    )
+    # connectors is the only source of the OPC servers: opc-servers is never loaded
+    assert not [
+        c
+        for c in workflow_mock.start_local_activity_method.call_args_list
+        if c.args[0] is Activities.find_documents_in_mongodb
+        and c.args[1]['query'] != {'collection': 'orchestrated_schedules'}
+    ]
 
     workflow_mock.start_local_activity_method.assert_has_calls(
         [
@@ -116,7 +110,6 @@ async def test_run(workflow_mock, referenced_tag_ids_mock, orchestrator):
         ]
     )
 
-    workflow_mock.patched.assert_called_once_with(TAG_DATABASE_PATCH)
     referenced_tag_ids_mock.assert_called_once_with(
         workflow_mock.start_local_activity_method.return_value
     )
@@ -162,6 +155,7 @@ async def test_run(workflow_mock, referenced_tag_ids_mock, orchestrator):
                 {
                     **metadata,
                     'pipelines': workflow_mock.start_local_activity_method.return_value,
+                    'connectors': workflow_mock.start_local_activity_method.return_value,
                     'connector_tags': workflow_mock.start_local_activity_method.return_value,
                 },
                 retry_policy=ANY,
@@ -176,7 +170,6 @@ async def test_run(workflow_mock, referenced_tag_ids_mock, orchestrator):
                 Activities.process_slots,
                 {
                     **metadata,
-                    'opc_servers': workflow_mock.start_local_activity_method.return_value,
                     'connectors': workflow_mock.start_local_activity_method.return_value,
                     'connector_tags': workflow_mock.start_local_activity_method.return_value,
                     'active_ingestors': workflow_mock.start_local_activity_method.return_value,
@@ -423,7 +416,6 @@ def _process_slots_input(workflow_mock):
 @patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
 async def test_run_with_tag_database_queries_from_input(workflow_mock, _ids, orchestrator):
     """The input overrides the default tag database aggregations (SIENTIAPDE-1974, D9)"""
-    workflow_mock.patched = MagicMock(return_value=True)
     connectors_query = {'collection': 'connectors', 'aggregation': [{'$match': {'type': 'opc_ua'}}]}
     connector_tags_query = {
         'collection': 'connector_tags',
@@ -431,7 +423,6 @@ async def test_run_with_tag_database_queries_from_input(workflow_mock, _ids, orc
     }
     input_data = {
         'pipelines_query': 'SELECT * FROM bucket',
-        'opc_servers_query': 'SELECT * FROM servers',
         'connectors_query': connectors_query,
         'connector_tags_query': connector_tags_query,
         'schedule_name': 'test-schedule-name',
@@ -457,37 +448,79 @@ async def test_run_with_tag_database_queries_from_input(workflow_mock, _ids, orc
 @patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
 async def test_run_without_tag_ids_skips_tag_database(workflow_mock, _ids, orchestrator):
     """No read tag with tag_id: the tag database is not queried at all (P4)"""
-    workflow_mock.patched = MagicMock(return_value=True)
     input_data = {
         'pipelines_query': 'SELECT * FROM bucket',
-        'opc_servers_query': 'SELECT * FROM servers',
         'schedule_name': 'test-schedule-name',
     }
 
     await orchestrator.run(input_data)
 
-    workflow_mock.patched.assert_called_once_with(TAG_DATABASE_PATCH)
     assert _tag_database_calls(workflow_mock) == []
     slots_input = _process_slots_input(workflow_mock)
     assert slots_input['connectors'] == []
     assert slots_input['connector_tags'] == []
 
 
+def _process_schedules_input(workflow_mock):
+    return next(
+        c.args[1]
+        for c in workflow_mock.start_local_activity_method.call_args_list
+        if c.args[0] is Activities.process_schedules
+    )
+
+
+def test_default_connector_tags_query_loads_every_category_and_status():
+    """
+    The status / category filters are applied in the code. updated_at is not loaded: a
+    tag change reaches the schedules through their config_hash.
+    """
+    stages = DEFAULT_CONNECTOR_TAGS_QUERY['aggregation']
+    assert all('$match' not in stage for stage in stages)
+    projected = stages[-1]['$project']
+    for field in ('category', 'data_type', 'status'):
+        assert projected[field] == 1
+    assert 'updated_at' not in projected
+
+
 @mark.asyncio
-@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[7])
+@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[77])
 @patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
-async def test_run_replay_before_tag_database_patch(workflow_mock, _ids, orchestrator):
-    """A run started before the patch replays without the tag database loads"""
-    workflow_mock.patched = MagicMock(return_value=False)
+async def test_run_hands_the_tag_database_to_process_schedules(workflow_mock, _ids, orchestrator):
+    """The write tags of process_schedules need the connectors as well as the tags"""
     input_data = {
         'pipelines_query': 'SELECT * FROM bucket',
-        'opc_servers_query': 'SELECT * FROM servers',
         'schedule_name': 'test-schedule-name',
     }
 
     await orchestrator.run(input_data)
 
-    assert _tag_database_calls(workflow_mock) == []
-    slots_input = _process_slots_input(workflow_mock)
-    assert slots_input['connectors'] == []
-    assert slots_input['connector_tags'] == []
+    loaded = workflow_mock.start_local_activity_method.return_value
+    schedules_input = _process_schedules_input(workflow_mock)
+    assert schedules_input['connectors'] is loaded
+    assert schedules_input['connector_tags'] is loaded
+    assert 'pi_connectors' not in schedules_input
+
+
+@mark.asyncio
+@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[])
+@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
+async def test_run_without_changes_skips_reports_and_timestamps(workflow_mock, _ids, orchestrator):
+    """Nothing created, updated or deleted: no report and no pipeline timestamp activity"""
+    workflow_mock.start_activity_method = AsyncMock(return_value=[])
+    input_data = {
+        'pipelines_query': 'SELECT * FROM bucket',
+        'schedule_name': 'test-schedule-name',
+    }
+
+    await orchestrator.run(input_data)
+
+    started = [c.args[0] for c in workflow_mock.start_activity_method.call_args_list]
+    assert started == [
+        Activities.normalize_schedules,
+        Activities.create_collection_with_ttl_index,
+        Activities.delete_slots,
+        Activities.update_slots,
+        Activities.delete_schedules,
+        Activities.create_schedules,
+        Activities.update_schedules,
+    ]

tests/orchestrator/workflows/test_orchestrator_replay.py

@@ -0,0 +1,27 @@
+"""
+Replay a real Orchestrator history against the current workflow code.
+
+orchestrator_tag_database.json was recorded on the e2e stack (real Temporal, Mongo and
+Redis), with the connectors / connector_tags loads of read and write tags by tag_id.
+
+A NondeterminismError here means a change to the workflow breaks runs in flight.
+"""
+
+from pathlib import Path
+
+import pytest
+from temporalio.client import WorkflowHistory
+from temporalio.worker import Replayer
+
+from orchestrator.workflows.orchestrator import Orchestrator
+
+HISTORIES = Path(__file__).parent / 'histories'
+
+
+@pytest.mark.asyncio
+async def test_replay_recorded_history():
+    history = WorkflowHistory.from_json(
+        'orchestrator-history', (HISTORIES / 'orchestrator_tag_database.json').read_text()
+    )
+
+    await Replayer(workflows=[Orchestrator]).replay_workflow(history)