SIENTIAPDE-1974

SIENTIAPDE-1974-connectors-pi-web-api: Implement OPC UA client certificate handling and update connector security model. Connectors now define security using security.mode and security.policy. Client certificate details (MinIO paths) are fetched from the opc_certificates collection via a MongoDB $lookup and included in the OPC slot. A warning is issued if a connector's client_certificate_id does not resolve to an active client certificate. Documentation (README, CLAUDE.md) and tests have been updated.

.github/workflows/deploy.yml

@@ -1,10 +1,15 @@
 name: Deploy DataOps Worker
 
 on:
-  pull_request:
-    types: [closed]
-    branches: [main, 'release/**', 'feature/**']
   workflow_dispatch:
+    inputs:
+      chart_ref:
+        description: >
+          sientia-helm-chart branch to bump GITHUB_BRANCH on AND deploy — the literal
+          deploy target. Validated via git ls-remote; a branch that does not exist in
+          sientia-helm-chart is a hard error, not a silent fallback to main.
+        required: true
+        type: string
 
 permissions:
   contents: write
@@ -14,10 +19,10 @@ permissions:
 jobs:
   deploy:
     name: Deploy
-    if: github.event_name == 'workflow_dispatch' || github.event.pull_request.merged == true
     uses: Aignosi/github_workflow_templates/.github/workflows/reusable-deploy-dataops.yml@main
     with:
-      branch_name: ${{ github.event.pull_request.head.ref }}
+      branch_name: ${{ github.ref_name }}
+      chart_ref: ${{ inputs.chart_ref }}
       project_name: 'orchestrator'
       chart_name: 'orchestrator-worker'
       env_list_path: '[".env"]'

.github/workflows/quality-gate.yml

@@ -5,7 +5,6 @@ on:
     branches:
       - main
       - 'release/**'
-      - 'feature/**'
   pull_request:
     branches:
       - main

.github/workflows/require-chart-ref-label.yml

@@ -1,16 +0,0 @@
-name: Require chart-ref label
-
-on:
-  pull_request:
-    types: [opened, synchronize, reopened, labeled, unlabeled]
-    branches:
-      - main
-      - 'release/**'
-      - 'feature/**'
-
-permissions:
-  pull-requests: write
-
-jobs:
-  require-chart-ref-label:
-    uses: Aignosi/github_workflow_templates/.github/workflows/require-chart-ref-label.yml@main

.github/workflows/version-bump.yml

@@ -0,0 +1,17 @@
+name: Version Bump
+
+permissions:
+  contents: write
+
+on:
+  pull_request:
+    types: [closed]
+    branches: [main, 'release/**', 'feature/**']
+
+jobs:
+  bump:
+    if: github.event.pull_request.merged == true
+    uses: Aignosi/github_workflow_templates/.github/workflows/reusable-version-bump.yml@main
+    with:
+      update_version_in: '["pyproject"]'
+    secrets: inherit

.gitignore

@@ -49,4 +49,5 @@ git_log
 .env
 
 openspec/
-.cursor/
+.cursor/
+.claude/

CLAUDE.md

@@ -45,9 +45,10 @@ CI quality gate (`.github/workflows/quality-gate.yml`) calls the shared reusable
 
 ### Layering
 - **`orchestrator/worker/worker.py`** — entrypoint (`python -m orchestrator.worker.worker`). Builds a single `Activities` instance, connects to Temporal, starts Prometheus (`HTTP_METRICS_PORT`) and Temporal SDK metrics (`HTTP_SDK_METRICS_PORT`) servers, then runs three `Worker`s concurrently via `asyncio.gather`, each on its own task queue: `orchestrator-queue`, `alerts-queue`, `reports-queue`. `prepare_worker.py` derives the queue name from the workflow class name (`camel_to_snake`) and reads per-queue concurrency/poller tuning from env vars prefixed with the workflow name (e.g. `ORCHESTRATOR_MAX_CONCURRENT_ACTIVITIES`).
-- **`orchestrator/workflows/`** — Temporal workflow definitions: `orchestrator.py` (pipeline/slot/schedule orchestration), `alerts.py` (real-time ERROR notifications), `reports.py` (scheduled all-level reports). `alerts.py` and `reports.py` both delegate to the same two subworkflows in `workflows/subworkflows/`: `load_notification_package.py` (load notifications + receiver configs from Mongo, incremental via Redis timestamp) and `process_notifications.py` (build HTML email, send, log to Postgres). The difference between alerts and reports is which filter activity they call (`filter_notification_alerts` vs `filter_notification_reports`) and their base data filter (`level: ERROR` by default vs all levels) — `load_notification_package.py`'s `_compute_query_time_filter` can widen either beyond that default to the union of active receiver groups' own `levels` (SIENTIAPDE-2011); the ERROR-only default only holds when no active group narrows further.
+- **`orchestrator/workflows/`** — Temporal workflow definitions: `orchestrator.py` (pipeline/slot/schedule orchestration), `alerts.py` (real-time ERROR notifications), `reports.py` (scheduled all-level reports). `alerts.py` and `reports.py` both delegate to the same two subworkflows in `workflows/subworkflows/`: `load_notification_package.py` (load notifications + receiver configs from Mongo, incremental via Redis timestamp) and `process_notifications.py` (build HTML email, send, log to Postgres). The difference between alerts and reports is which filter activity they call (`filter_notification_alerts` vs `filter_notification_reports`) and their base data filter (`level: ERROR` by default vs all levels) — `load_notification_package.py`'s `_compute_query_time_filter` can widen either beyond that default to the union of active receiver groups' own `levels` (SIENTIAPDE-2011); the ERROR-only default only holds when no active group narrows further. Receiver groups also support a `model_ids` allow-list (SIENTIAPDE-2149), independent of the pre-existing `ignore_models_id` deny-list — both apply together (AND) when a group configures both, and `model_ids` follows the same union-at-query-time / precise-per-group-slice split as `levels`. `load_latest_data`'s Mongo query is capped at `NOTIFICATION_QUEUE_BATCH_SIZE` documents per run (default `0`, meaning unlimited — unchanged from pre-SIENTIAPDE-2013 behavior unless a deployment opts in), sorted by `timestamp` ascending so a capped run always returns the oldest pending batch first (SIENTIAPDE-2013) — a hard cap with no drain loop, since the incremental Redis cursor picks up any remainder on the next run.
 - **`orchestrator/activities/activities.py`** — `Activities` is a single class combining every activity mixin via multiple inheritance: `TemporalManager, SlotManager, Formatters, MongoDB, Email, Postgres` (Postgres comes from `sientia_do.temporal.activities.postgres`, the external `sientia-dataops-library` dependency). One instance is constructed in `worker.py` and its bound methods are handed to each `Worker` as the activity list — activities are plain methods on this shared object, not separate classes. Each mixin's `__init__` takes its own config dict plus shared `logger`, `notification_handler`, `metrics_controller`. `Activities.shutdown()` closes every mixin's connections.
-- **`orchestrator/utils/`** — `connectors_config.py` builds each config dict from env vars (Temporal/Redis/MongoDB/Postgres/Email); `email_builder.py` renders Jinja2 templates (`utils/templates/email_template.html`, `general_template.html`); `orchestrator_functions.py` transforms pipeline documents into pipeline-type-specific configs (`scouter`, `pi_web_api_scouter`, `predictions_batch`, `minimal_retrain`, `drift`, `simple_metrics`); `converters.py` handles Temporal schedule frequency parsing.
+- **`orchestrator/utils/`** — `connectors_config.py` builds each config dict from env vars (Temporal/Redis/MongoDB/Postgres/Email); `email_builder.py` renders Jinja2 templates (`utils/templates/email_template.html`, `general_template.html`); `pipeline_configs.py` transforms pipeline documents into pipeline-type-specific configs (`scouter`, `pi_web_api_scouter`, `predictions_batch`, `minimal_retrain`, `drift`, `simple_metrics`); `tag_database.py` indexes the tag database (`connectors` + `connector_tags`) and resolves tags by `tag_id` (scouter read tag names and the `predictions_batch` write tags → `opc_output_config` keyed by `connector_id`, via `resolve_write_tags`; tag changes reach the schedules through `config_hash`, see below); `opc_slots.py` builds the OPC slot v2 the gateway reads (`gather_read_tags` / `build_tag_config`, contract C1: one entry per address with a `names[]` list, and per server `security` {mode, policy} plus the `certificate` from `tag_database.resolve_certificate`; see the README section "OPC slots"). The certificate comes from `opc_certificates` through the `$lookup` in `DEFAULT_CONNECTORS_QUERY` (only `active` + `client`); no certificate means the gateway connects without security, and a `client_certificate_id` without one is a `WARNING` (SIENTIAPDE-1974, D5). `resolve_tag_names` (in `tag_database.py`) is the single source of tag names for both the scouter's `model_tags` and the slot's `names` (SIENTIAPDE-1974 SC1); `converters.py` handles Temporal schedule frequency parsing.
+- **Schedule redeploys** — `orchestrated_schedules` stores, per schedule, the last deploy time (`updated_at`) and the `config_hash` it was deployed with (`formatters.schedule_config_hash`: SHA-256 of the built schedule without `updated_at`). `compare_config_timestamps` redeploys when the pipeline's `updated_at` is newer (manual force) or the hash differs (any content change, including tag database edits/deletes made without bumping `updated_at`). `create_schedules`/`update_schedules` pop `config_hash` before sending the schedule to Temporal and return it in their report for `create_pipelines_timestamps`/`update_pipelines_timestamps`.
 
 ### Temporal-specific conventions (all workflow/activity files follow this)
 - Every workflow and activity-consuming module wraps non-deterministic/side-effecting imports in `with workflow.unsafe.imports_passed_through(): ...` right after `from temporalio import workflow`, because Temporal's sandbox restricts what workflow code can import directly. When adding a new workflow file, follow the existing pattern (see `orchestrator/workflows/orchestrator.py`, `orchestrator/activities/activities.py`) rather than importing at module top level.

README.md

@@ -97,13 +97,13 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
 - **Infrastructure Management**: Creates, updates, and deletes workflow schedules
 
 **Execution Flow**:
-1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, and active ingestors
+1. **Parallel Data Loading**: Concurrently loads pipelines, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`, the only source of the OPC servers and tags of the slot and the schedules)
 2. **Parallel Processing**: Formats orchestrated schedules and processes new schedules and slots
 3. **Parallel Config Creation**: Creates schedule and slot action configurations, normalizes schedules, and creates collections with TTL indexes
 4. **Parallel Operations**: Executes slot deletions, slot updates, schedule deletions, schedule creations, and schedule updates concurrently
 5. **Parallel Reports & Timestamps**: Generates orchestration reports and updates MongoDB timestamps for created/updated/deleted pipelines
 
-**Input Parameters**:
+**Input Parameters** (minimal: each pipeline must already carry `model.name`; see "Workflow Configuration" for the `$lookup` on `models` that adds it):
 ```json
 {
   "schedule_name": "hourly_orchestration",
@@ -113,21 +113,153 @@ The **Orchestrator** workflow is the main coordination workflow that manages pip
       {"$match": {"active": true}},
       {"$sort": {"updated_at": -1}}
     ]
-  },
-  "opc_servers_query": {
-    "collection": "opc_servers",
-    "filters": {"active": true}
   }
 }
 ```
 
+Optional: `connectors_query` and `connector_tags_query` override the tag database aggregations. The defaults live in the code (`orchestrator/workflows/orchestrator.py`), so the schedule input does not need them:
+
+```json
+{
+  "connectors_query": {
+    "collection": "connectors",
+    "aggregation": [
+      {"$match": {"type": "opc_ua", "status": "Active"}},
+      {"$lookup": {"from": "opc_certificates",
+                   "let": {"certificate_id": "$security.client_certificate_id"},
+                   "pipeline": [
+                     {"$match": {"$expr": {"$eq": ["$certificate_id", "$$certificate_id"]},
+                                 "status": "active", "kind": "client"}},
+                     {"$project": {"_id": 0, "certificate_id": 1, "application_uri": 1,
+                                   "cert.bucket": 1, "cert.object_name": 1,
+                                   "private_key.bucket": 1, "private_key.object_name": 1}}
+                   ],
+                   "as": "certificates"}},
+      {"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "subscription_period_ms": 1,
+                    "security.mode": 1, "security.policy": 1,
+                    "security.client_certificate_id": 1, "certificates": 1}}
+    ]
+  },
+  "connector_tags_query": {
+    "collection": "connector_tags",
+    "aggregation": [
+      {"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1,
+                    "category": 1, "data_type": 1, "status": 1}}
+    ]
+  }
+}
+```
+
+The workflow puts `{"$match": {"tag_id": {"$in": [...]}}}` in front of `connector_tags_query`, so it loads only the tags the scouter read tags and `predictions_batch` write tags reference. When no pipeline references a `tag_id`, the tag database is not queried. An override should keep a `$project` of these fields: any other attribute may hold a BSON type a Temporal payload cannot carry (such as `ObjectId`).
+
+The referenced tags are loaded whatever their category and status: the code uses only the active ones (`Read` for the slot and the scouter names, `Write` / `Prediction Confidence` for the write tags), and a change to a referenced tag that reaches a schedule redeploys it through its `config_hash` (see "Redeploy on changes" below). A document without `status` counts as active.
+
+#### OPC slots (slot v2, contract C1)
+
+`process_slots` writes one slot per active ingestor to Redis (`slot:opc_tags:{n}`). The OPC Gateway reads it (SIENTIAPDE-1974). Each server carries its connection fields, its security and the MinIO objects of its client certificate, and each tag address carries the list of names it is written under:
+
+```json
+{
+  "kepserver": {
+    "server_id": "1", "name": "kepserver", "url": "opc.tcp://host:49320",
+    "subscription_period_ms": 7500.0,
+    "security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256"},
+    "certificate": {
+      "certificate_id": 1, "application_uri": "urn:sientia:opc-ingestor",
+      "cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-cert.pem"},
+      "private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/opc-ingestor-key.pem"}
+    },
+    "tags": {
+      "ns=2;s=Channel1.Device1.Tag1": {
+        "frequency": 15000,
+        "active_read": {"enabled": false},
+        "names": [
+          {"collection": "raw_model_a", "tag_name": "Counter",  "model_id": "1", "model_name": "model a"},
+          {"collection": "raw_model_b", "tag_name": "Contador", "model_id": "2", "model_name": "model b"}
+        ]
+      }
+    }
+  }
+}
+```
+
+Each scouter `read_tags` entry references its tag by `tag_id` (`{"tag_id": 7, "frequency": 15000}`). `connectors` is the only source of the OPC servers:
+
+| Field | Source |
+|---|---|
+| Address | `connector_tags.address` |
+| Server (`server_id`, `name`, `url`, `subscription_period_ms`, `security.mode`, `security.policy`) | The tag's `connector_tags.connector_id` in `connectors` (`server_id` = `connector_id`) |
+| `certificate` | The `opc_certificates` document of the connector's `security.client_certificate_id`, when it is `active` and of kind `client` (`connectors_query`'s `$lookup`) |
+| Name | The read tag's `tag_name`, else the `connector_tags` one |
+
+Rules:
+- **One entry per address.** An address read by N pipelines is read once and written to N collections: one `names` item per pipeline (`collection` = `raw_{schedule_name}`).
+- **Same name source as the scouter.** `resolve_tag_names` names both the slot's `names` and the scouter's `model_tags`, so the gateway always writes the name the scouter looks for.
+- **`frequency`:** the lowest among the pipelines of the address, as an integer. Every read tag needs one.
+- **`tag_id`:** required, an integer like in `connector_tags`. A read tag whose `tag_id` is missing, `null` or not an integer (`"7"` included) is left out and notified as invalid.
+- **`subscription_period_ms`:** the connector's value. A connector without one gets half its lowest tag frequency in the slot.
+- **`security`:** the connector's `mode` and `policy`, the source of the OPC security (`use_security` and `server_uri` no longer exist). A connector without them is left out like an incomplete one.
+- **`certificate`:** `certificate_id`, `application_uri` (the SAN URI of the certificate, used as the client application URI) and the `bucket`/`object_name` of `cert` and `private_key`; the gateway downloads them from MinIO. The certificate dates, `sha256` and audit fields stay out of the slot. There is no default certificate: `null` means the gateway connects without security, and a server that requires a certificate refuses the connection. A connector whose `client_certificate_id` is set but has no active `client` certificate (missing id, `retired`, another kind or an incomplete document) still goes into the slot with `certificate: null`, and is reported (see below).
+- **`active_read`:** off by default. It is on when any pipeline sets `read_tags[].active_read.enabled`, with the lowest `cycle_multiple`.
+- **Server keys:** two different connectors with the same name get distinct slot keys (`name`, then `name:server_id`).
+- **Slot division:** the addresses are split across the slots, sorted by server and address, and the names of an address always stay in the same slot.
+- **Unresolved tags:** a read tag that cannot be resolved is left out of the slot, and the rest of the slot is still written. It is notified once per run:
+  - `ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the `tag_id` is not an active Read tag, or its document lacks `connector_id`, `address` or `tag_name`;
+  - `ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the connector is missing or inactive, or lacks `name`, `url`, `security.mode` or `security.policy`;
+  - `ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION`: the read tag has no integer `tag_id`, no integer `frequency`, or an enabled `active_read` without an integer `cycle_multiple`; or its pipeline lacks `schedule_name`, `model_id` or `model.name` (the value is then the `schedule_name`).
+  - `ORCHESTRATOR_CERTIFICATE_NOT_FOUND_DURING_SLOT_CONFIGURATION` (`WARNING`): the connector's `client_certificate_id` has no active `client` certificate, so its tags go into the slot without a certificate. Most likely a registration error.
+- **Only changed slots are written.** The slot is built in a stable order, so unchanged pipelines produce an identical slot, and the gateway does not re-subscribe.
+
+#### Write tags (predictions_batch, through the tag database)
+
+A `predictions_batch` (or `xgboost_predictions_batch`) references its write tags by `tag_id` only, exactly like the read tags. Everything else comes from the tag database, so the pipeline does not say which server, address or type a tag has:
+
+```json
+{"workflow_type": "predictions_batch", "write_tags": [{"tag_id": 101}, {"tag_id": 102}]}
+```
+
+`process_schedules` turns them into the `opc_output_config` the Laborious already reads:
+
+```json
+{"plant": {
+  "prediction_tags": {"ns=3;i=2001": {"data_type": "double"}},
+  "confidence_tags": {"ns=3;i=2002": {"data_type": "float"}}}}
+```
+
+| Field | Source |
+|---|---|
+| Server key | `connector_tags.connector_id`. The Laborious publishes it as `server_id` on NATS, and the OPC Gateway resolves it in `connectors` |
+| `prediction_tags` / `confidence_tags` | `connector_tags.category`: `Write` / `Prediction Confidence` |
+| Address | `connector_tags.address` |
+| `data_type` | `connector_tags.data_type` (`float`, `double`, `real`, `int`, `integer`, `bool`, `boolean`, `str`, `string`), else `float` |
+
+A write tag that cannot be resolved is left out, the rest of the pipeline is still scheduled, and it is notified once per run (values are `schedule_name/tag_id`):
+- `ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the `tag_id` is not an active `Write` / `Prediction Confidence` tag, or its document lacks `connector_id`, `address`, `tag_name` or `category`;
+- `ORCHESTRATOR_WRITE_TAG_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the tag's connector is missing, inactive or incomplete in `connectors`;
+- `ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION`: the entry has no integer `tag_id`, or the tag's `data_type` is not one the gateway accepts.
+
+#### Redeploy on changes (`updated_at` and `config_hash`)
+
+`orchestrated_schedules` keeps, per schedule, when it was last deployed (`updated_at`) and the `config_hash` it was deployed with: a SHA-256 of the schedule as the Orchestrator builds it (the workflow input, frequency and offset, with tags and connectors already resolved), without `updated_at`. A schedule is redeployed when either:
+
+- the pipeline's `updated_at` is newer than its last deploy. Bumping `updated_at` by hand forces a redeploy even when nothing else changed; or
+- its `config_hash` differs from the stored one. Any change that reaches the schedule redeploys it, whether or not someone bumped an `updated_at`: a pipeline edited by hand, a tag renamed, moved, retyped, deactivated or deleted in `connector_tags`, a connector changed, or a new Orchestrator version that builds the schedule differently.
+
+A change that does not reach the schedule (e.g. a tag's `description`) does not redeploy. The slot is compared by content and follows on its own. A pipeline without `updated_at` is still redeployed on every run. Schedules deployed before `config_hash` existed have none stored, so the first run of this version redeploys all of them once and records it.
+
+#### Deploying
+
+1. Run the migrations that create the tag database and add `data_type` to the `connector_tags` validator (`sientia-dataops-database-migrations`).
+2. Every scouter `read_tags` and every `predictions_batch` / `xgboost_predictions_batch` `write_tags` entry is a `tag_id` of `connector_tags`, as an integer. Anything else is left out and notified.
+3. Deploy this version. Its first run redeploys every schedule once, to record their `config_hash` (see above).
+
 #### Architecture
 
 ```mermaid
 flowchart LR
     subgraph "Parallel Data Loading"
         A[1. aggregate_documents_in_mongodb]
-        B[2. find_documents_in_mongodb<br/>OPC Servers]
+        B[2. aggregate_documents_in_mongodb<br/>connectors + connector_tags]
         C[3. find_documents_in_mongodb<br/>Orchestrated Schedules]
         D[4. load_opc_slots]
         E[5. load_active_ingestors]
@@ -556,6 +688,112 @@ python -m orchestrator.worker.worker
 | `LOG_LEVEL` | Application logging level | `INFO` | No |
 | `KAFKA_BOOTSTRAP_SERVERS` | Kafka bootstrap servers | - | No |
 
+### MongoDB Collections
+
+| Collection | Read / written by | Purpose |
+|---|---|---|
+| `pipelines` | Orchestrator (`pipelines_query`) | One document per Temporal schedule (`schedule_name`, `workflow_type`, `model_id`, `active`, `frequency`, ...) |
+| `models` | Orchestrator (`$lookup` in `pipelines_query`) | Gives each pipeline its `model` (`name`, `active`, `model_config`) |
+| `connectors` | Orchestrator (`connectors_query`) | Tag database: the OPC UA servers (`connector_id`, `type`, `name`, `url`, `security`, `subscription_period_ms`, `status`) |
+| `connector_tags` | Orchestrator (`connector_tags_query`) | Tag database: the tags (`tag_id`, `connector_id`, `address`, `tag_name`, `category`, `data_type`, `status`) |
+| `opc_certificates` | Orchestrator (`$lookup` in `connectors_query`) | The client certificate of each connector (`security.client_certificate_id` → `certificate_id`), only when `active` and of kind `client`: `application_uri` and the MinIO objects of `cert` and `private_key` go into the slot |
+| `orchestrated_schedules` | Orchestrator (read and written) | Per deployed schedule: `schedule_name`, `namespace`, `updated_at` (last deploy) and `config_hash` |
+| `raw_{schedule_name}` | Orchestrator (created) | One per scouter `topic`, where the OPC Gateway writes its tags. Created with a TTL index on `inserted_at` (`MONGODB_TTL_INDEX_HOURS`) |
+| `notification_queue` | Alerts / Reports (read) | Notifications (`notification_id`, `trigger`, `level`, `model_id`, `timestamp`, ...), loaded incrementally by `timestamp` |
+| `receiver_groups` | Alerts / Reports (read, `active: true`) | Who receives what (see below) |
+
+#### Tags in a pipeline
+
+Pipelines reference tags only by `tag_id` (an integer, as in `connector_tags`). Servers, addresses and data types come from the tag database, never from the pipeline.
+
+Scouter (`scouter` namespace), `workflow_type: "scouter"`:
+
+```json
+{
+  "schedule_name": "scouter-linreg",
+  "workflow_type": "scouter",
+  "model_id": "1",
+  "active": true,
+  "frequency": "1m",
+  "read_tags": [
+    {
+      "tag_id": 7,
+      "frequency": 15000,
+      "tag_name": "Pressure",
+      "aggr_func": "avg",
+      "data_range": [0, 100],
+      "active_read": {"enabled": true, "cycle_multiple": 2}
+    }
+  ]
+}
+```
+
+| Field | Required | Meaning |
+|---|---|---|
+| `tag_id` | Yes | An active `Read` tag of `connector_tags` |
+| `frequency` | Yes | Read period in ms, an integer. The slot uses the lowest among the pipelines of the address |
+| `tag_name` | No | Overrides the `connector_tags` name: the name the gateway writes and the scouter looks for |
+| `aggr_func` | No | Default `lts` |
+| `data_range` | No | Default `[-100, 100]` |
+| `active_read` | No | Off by default. When `enabled`, `cycle_multiple` must be an integer (default `1`) |
+
+Laborious (`laborious` namespace): only `predictions_batch` and `xgboost_predictions_batch` reference tags; `minimal_retrain`, `drift` and `simple_metrics` have none.
+
+```json
+{
+  "schedule_name": "pred-linreg",
+  "workflow_type": "predictions_batch",
+  "model_id": "1",
+  "active": true,
+  "frequency": "1m",
+  "query": "SELECT ...",
+  "write_tags": [{"tag_id": 101}, {"tag_id": 102}]
+}
+```
+
+Each `write_tags` entry is only a `tag_id`, of an active `Write` (prediction) or `Prediction Confidence` tag. The Orchestrator builds `opc_output_config` from them (see "Write tags" above) and overwrites any `opc_output_config` stored in the pipeline.
+
+The `connector_tags` documents those pipelines reference:
+
+```json
+{"tag_id": 7,   "connector_id": "plant", "address": "ns=3;i=1001", "tag_name": "Pressure", "category": "Read", "status": "Active"}
+{"tag_id": 101, "connector_id": "plant", "address": "ns=3;i=2001", "tag_name": "PRED-101", "category": "Write", "data_type": "double", "status": "Active"}
+{"tag_id": 102, "connector_id": "plant", "address": "ns=3;i=2002", "tag_name": "CONF-101", "category": "Prediction Confidence", "status": "Active"}
+```
+
+`pi_web_api_scouter` (`scouter` namespace) does not use the tag database: its `read_tags` is an object keyed by tag name, and it is not part of the OPC slot:
+
+```json
+{
+  "workflow_type": "pi_web_api_scouter",
+  "read_tags": {"Pressure": {"webid": "F1DP...", "aggr_func": "lts", "data_range": [0, 100]}},
+  "pi_web_api_config": {"endpoint": "https://pi/piwebapi", "period": "*-1d", "max_count": 1, "api_timeout": 30}
+}
+```
+
+#### Receiver groups
+
+```json
+{
+  "group_name": "admin_team",
+  "active": true,
+  "members": ["admin@example.com"],
+  "contents": ["core_alerts", "persistent_alerts", "reports"],
+  "levels": ["ERROR", "WARNING"],
+  "model_ids": ["1"],
+  "ignore_models_id": ["2"],
+  "ignore": ["SOME_NOTIFICATION_ID"]
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `contents` | What the group gets: `core_alerts` / `persistent_alerts` (Alerts workflow), `reports` (Reports workflow) |
+| `levels` | Optional allow-list of notification levels. Absent = every level the workflow loads |
+| `model_ids` | Optional allow-list of `model_id`s |
+| `ignore_models_id` | Optional deny-list of `model_id`s (applies together with `model_ids`) |
+| `ignore` | Optional deny-list of `notification_id`s |
+
 ### Workflow Configuration
 
 Temporal input configuration sample:
@@ -583,7 +821,7 @@ Temporal input configuration sample:
             },
             {
                 "$addFields": {
-                    "models": {
+                    "model": {
                         "$arrayElemAt": [
                             "$model_docs",
                             0
@@ -593,7 +831,7 @@ Temporal input configuration sample:
             },
             {
                 "$match": {
-                    "models.active": True
+                    "model.active": True
                 }
             },
             {
@@ -603,15 +841,70 @@ Temporal input configuration sample:
             }
         ]
     },
-    "opc_servers_query": {
-        "collection": "opc-servers",
-        "filters": {
-
-        }
+    "connectors_query": {
+        "collection": "connectors",
+        "aggregation": [
+            {"$match": {"type": "opc_ua", "status": "Active"}},
+            {"$lookup": {"from": "opc_certificates",
+                         "let": {"certificate_id": "$security.client_certificate_id"},
+                         "pipeline": [
+                           {"$match": {"$expr": {"$eq": ["$certificate_id", "$$certificate_id"]},
+                                       "status": "active", "kind": "client"}},
+                           {"$project": {"_id": 0, "certificate_id": 1, "application_uri": 1,
+                                         "cert.bucket": 1, "cert.object_name": 1,
+                                         "private_key.bucket": 1, "private_key.object_name": 1}}
+                         ],
+                         "as": "certificates"}},
+            {"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "subscription_period_ms": 1,
+                          "security.mode": 1, "security.policy": 1,
+                          "security.client_certificate_id": 1, "certificates": 1}}
+        ]
+    },
+    "connector_tags_query": {
+        "collection": "connector_tags",
+        "aggregation": [
+            {"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1,
+                          "category": 1, "data_type": 1, "status": 1}}
+        ]
     }
 }
 ```
 
+`connectors_query` and `connector_tags_query` are optional: the values above are the defaults in the code (see "OPC slots" and "Write tags" above).
+
+The `$lookup` must end in a `model` field (singular): every pipeline type reads `model.name` (and `model.model_config`, when present), so a pipeline without `model` fails to build its schedule.
+
+#### Simple metrics: parallel windows & adaptive frequency
+
+One `pipelines` doc per window, all with the same `model_id`. Name them `<model>-simple_metrics-<interval_minutes>m`; each doc becomes its own Temporal schedule. Thresholds are shared through `model.model_config`. The adaptive state is per doc.
+
+```json
+{
+    "schedule_name": "model-x-simple_metrics-60m",
+    "workflow_type": "simple_metrics",
+    "model_id": "model-x",
+    "active": true,
+    "interval_minutes": 60,
+    "frequency": "1h",
+    "frequency_alert": "5m"
+}
+```
+
+| Field | Meaning | Default |
+|---|---|---|
+| `frequency_alert` | Frequency while escalated, same format as `frequency`. Absent = adaptive mode off. Below `1m` (or unparseable), the orchestrator sends a WARNING each time it creates or updates the schedule. A value below `1m` is kept; an unparseable value is ignored (the schedule keeps `frequency`) | — |
+| `adaptive_state` | `{escalated, healthy_streak}`, written **only** by laborious (with `updated_at` as a BSON datetime when `escalated` flips) | `{false, 0}` |
+| `model.model_config.simple_metrics_thresholds` | `{<metric>_max, <metric>_min}` shared by all windows | — |
+| `model.model_config.simple_metrics_min_samples` | Windows with fewer rows neither alert nor escalate, and count toward recovery | — |
+
+Escalation, recovery and the per-model alert are implemented in laborious (SIENTIAPDE-2189, SIENTIAPDE-2191); the orchestrator only picks `frequency_alert` while `adaptive_state.escalated` is true.
+
+Rules:
+- **Escalation:** laborious escalates on the first breach.
+- **Recovery:** after `min(12, ceil(interval_minutes / frequency_alert))` consecutive runs without one. Empty windows count as runs without a breach.
+- **Alerts** are per model (trigger `<model_id>-simple_metrics`), so several windows seeing the same breach produce one alert.
+- **Delay:** the schedule switches on the orchestrator tick after laborious flips the state, at most one tick later.
+
 #### Alerts Workflow
 
 ```json
@@ -792,7 +1085,9 @@ orchestrator/
 ├── utils/                   # Utility functions
 │   ├── connectors_config.py # Database configuration
 │   ├── email_builder.py    # Email template generation
-│   ├── orchestrator_functions.py # Pipeline utilities
+│   ├── pipeline_configs.py # Schedule input builders per pipeline type
+│   ├── tag_database.py     # Tag database indexes, OPC read names and write tags
+│   ├── opc_slots.py        # OPC slot v2 (contract C1)
 │   ├── converters.py       # Data type conversion utilities
 │   └── templates/          # Email HTML templates
 │       ├── email_template.html    # Report email template

e2e/conftest.py

@@ -92,9 +92,7 @@ async def register_namespace_if_missing(env: WorkflowEnvironment, namespace: str
     raise TimeoutError(f'Namespace {namespace} not ready within 5s')
 
 
-async def ensure_namespace_search_attributes(
-    env: WorkflowEnvironment, namespace: str
-) -> None:
+async def ensure_namespace_search_attributes(env: WorkflowEnvironment, namespace: str) -> None:
     """
     Register the orchestrator search attributes on a namespace, if missing.
 
@@ -330,6 +328,7 @@ async def test_activities(
             'connection_string': f'mongodb://localhost:{mongo_port}',
             'database_name': E2E_DATABASE,
             'ttl_index_seconds': 3600,
+            'notification_queue_batch_size': 0,
         },
         email_config={
             'sender_email': 'e2e@example.com',

e2e/helpers.py

@@ -7,6 +7,7 @@ import json
 import re
 import uuid
 from datetime import UTC, datetime, timedelta
+from email.message import EmailMessage, Message
 from pathlib import Path
 from typing import Any
 
@@ -24,8 +25,8 @@ MONGO_COLLECTIONS = (
     'receiver_groups',
     'orchestrated_schedules',
     'pipelines',
-    'opc_servers',
-    'opc-servers',
+    'connectors',
+    'connector_tags',
 )
 
 DATETIME_FORMAT_MS_WITH_TZ = '%Y-%m-%d %H:%M:%S.%f%z'
@@ -142,19 +143,25 @@ def seed_pipelines(
         client.close()
 
 
-def seed_opc_servers(
+def seed_tag_database(
     mongo_uri: str,
     database: str,
-    servers: list[dict[str, Any]],
-    *,
-    collection: str = 'opc_servers',
+    connectors: list[dict[str, Any]],
+    connector_tags: list[dict[str, Any]],
+    opc_certificates: list[dict[str, Any]] | None = None,
 ) -> None:
-    """Insert OPC server documents into the test Mongo database."""
+    """
+    Insert tag database documents (connectors, connector_tags and the opc_certificates
+    the connectors point to) into the test Mongo database.
+    """
     client = MongoClient(mongo_uri)
     try:
-        coll = client[database][collection]
-        if servers:
-            coll.insert_many(servers)
+        if connectors:
+            client[database]['connectors'].insert_many(connectors)
+        if connector_tags:
+            client[database]['connector_tags'].insert_many(connector_tags)
+        if opc_certificates:
+            client[database]['opc_certificates'].insert_many(opc_certificates)
     finally:
         client.close()
 
@@ -346,6 +353,7 @@ def default_receiver_group(
     levels: list[str] | None = None,
     contents: list[str] | None = None,
     ignore_models_id: list[str] | None = None,
+    model_ids: list[str] | None = None,
 ) -> dict[str, Any]:
     """Minimal active receiver_groups document."""
     return {
@@ -355,4 +363,42 @@ def default_receiver_group(
         'levels': levels or ['ERROR', 'WARNING', 'INFO'],
         'contents': contents or ['core_alerts', 'persistent_alerts', 'reports'],
         'ignore_models_id': ignore_models_id or [],
+        'model_ids': model_ids or [],
     }
+
+
+def extract_html_body(message: EmailMessage | Message) -> str:
+    """
+    Return the text/html portion of an email message as a decoded string.
+
+    Walks every part looking for the first text/html payload, decoding it
+    according to the part's transfer encoding and charset. Falls back to
+    the message's own ``get_content``/raw payload when no HTML part is
+    present so callers can still inspect plain-text reports. Shared between
+    the Alerts and Reports e2e suites, both of which send mail through the
+    same ``build_email_html``/``send_email`` activities.
+
+    Args:
+        message: Captured email message returned by the test SMTP server.
+
+    Return:
+        str: HTML body content, or an empty string when nothing decodable
+            is found.
+    """
+    if message.is_multipart():
+        for part in message.walk():
+            if part.get_content_type() != 'text/html':
+                continue
+            payload = part.get_payload(decode=True)
+            if payload is None:
+                continue
+            charset = part.get_content_charset() or 'utf-8'
+            return payload.decode(charset, errors='replace')
+    payload = message.get_payload(decode=True)
+    if payload is not None:
+        charset = message.get_content_charset() or 'utf-8'
+        return payload.decode(charset, errors='replace')
+    try:
+        return message.get_content()
+    except (AttributeError, KeyError):
+        return str(message.get_payload() or '')

e2e/scenario_inputs/orchestrator_conflict.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -22,13 +18,16 @@
       "frequency": "1m",
       "offset": "0m",
       "query": "SELECT 1",
-      "write_tags": [
-        {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
-      ]
+      "write_tags": [{"tag_id": 101}]
     }
   ],
-  "opc_servers": [
-    {"id": "srv-1", "active": true}
+  "connectors": [
+    {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+     "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+  ],
+  "connector_tags": [
+    {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+     "category": "Write", "status": "Active", "updated_at": "@now-1d"}
   ],
   "active_ingestors": ["heartbeat:ingestor:1"]
 }

e2e/scenario_inputs/orchestrator_create_only.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -22,13 +18,16 @@
       "frequency": "1m",
       "offset": "0m",
       "query": "SELECT 1",
-      "write_tags": [
-        {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
-      ]
+      "write_tags": [{"tag_id": 101}]
     }
   ],
-  "opc_servers": [
-    {"id": "srv-1", "active": true}
+  "connectors": [
+    {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+     "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+  ],
+  "connector_tags": [
+    {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+     "category": "Write", "status": "Active", "updated_at": "@now-1d"}
   ],
   "active_ingestors": ["heartbeat:ingestor:1"]
 }

e2e/scenario_inputs/orchestrator_delete_only.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -24,8 +20,5 @@
       "interval_minutes": 60
     }
   ],
-  "opc_servers": [
-    {"id": "srv-1", "active": true}
-  ],
   "active_ingestors": ["heartbeat:ingestor:1"]
 }

e2e/scenario_inputs/orchestrator_drift_window.json

@@ -0,0 +1,33 @@
+{
+  "workflow_input": {
+    "schedule_name": "orchestrator-e2e-drift-window",
+    "pipelines_query": {
+      "collection": "pipelines",
+      "aggregation": [{"$match": {"active": true}}]
+    }
+  },
+  "pipelines": [
+    {
+      "schedule_name": "model-dr-drift-default",
+      "workflow_type": "drift",
+      "model_id": "model-dr",
+      "model": {"name": "Model model-dr"},
+      "active": true,
+      "updated_at": "@now",
+      "frequency": "1m",
+      "chunk_period": "hour"
+    },
+    {
+      "schedule_name": "model-dr-drift-explicit",
+      "workflow_type": "drift",
+      "model_id": "model-dr",
+      "model": {"name": "Model model-dr"},
+      "active": true,
+      "updated_at": "@now",
+      "frequency": "1m",
+      "interval_minutes": 60,
+      "chunk_period": "s"
+    }
+  ],
+  "active_ingestors": ["heartbeat:ingestor:1"]
+}

e2e/scenario_inputs/orchestrator_empty.json

@@ -4,13 +4,8 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [],
-  "opc_servers": [],
   "active_ingestors": []
 }

e2e/scenario_inputs/orchestrator_happy_path.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -22,9 +18,7 @@
       "frequency": "1m",
       "offset": "0m",
       "query": "SELECT 1",
-      "write_tags": [
-        {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
-      ]
+      "write_tags": [{"tag_id": 101}]
     },
     {
       "schedule_name": "drift-gpu",
@@ -39,8 +33,13 @@
       "interval_minutes": 60
     }
   ],
-  "opc_servers": [
-    {"id": "srv-1", "active": true, "name": "opc-1"}
+  "connectors": [
+    {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+     "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+  ],
+  "connector_tags": [
+    {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+     "category": "Write", "status": "Active", "updated_at": "@now-1d"}
   ],
   "active_ingestors": ["heartbeat:ingestor:1"]
 }

e2e/scenario_inputs/orchestrator_noop.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -22,13 +18,16 @@
       "frequency": "1m",
       "offset": "0m",
       "query": "SELECT 1",
-      "write_tags": [
-        {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
-      ]
+      "write_tags": [{"tag_id": 101}]
     }
   ],
-  "opc_servers": [
-    {"id": "srv-1", "active": true}
+  "connectors": [
+    {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+     "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+  ],
+  "connector_tags": [
+    {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+     "category": "Write", "status": "Active", "updated_at": "@now-1d"}
   ],
   "active_ingestors": ["heartbeat:ingestor:1"],
   "orchestrated_schedules": [

e2e/scenario_inputs/orchestrator_simple_metrics_windows.json

@@ -0,0 +1,40 @@
+{
+  "workflow_input": {
+    "schedule_name": "orchestrator-e2e-sm-windows",
+    "pipelines_query": {
+      "collection": "pipelines",
+      "aggregation": [{"$match": {"active": true}}]
+    }
+  },
+  "pipelines": [
+    {
+      "schedule_name": "model-sm-simple_metrics-60m",
+      "workflow_type": "simple_metrics",
+      "model_id": "model-sm",
+      "model": {
+        "name": "Model model-sm",
+        "model_config": {"target": "y", "simple_metrics_thresholds": {"rmse_max": 1.0}}
+      },
+      "active": true,
+      "updated_at": "@now",
+      "frequency": "1h",
+      "frequency_alert": "5m",
+      "interval_minutes": 60
+    },
+    {
+      "schedule_name": "model-sm-simple_metrics-1440m",
+      "workflow_type": "simple_metrics",
+      "model_id": "model-sm",
+      "model": {
+        "name": "Model model-sm",
+        "model_config": {"target": "y", "simple_metrics_thresholds": {"rmse_max": 1.0}}
+      },
+      "active": true,
+      "updated_at": "@now",
+      "frequency": "1d",
+      "frequency_alert": "1h",
+      "interval_minutes": 1440
+    }
+  ],
+  "active_ingestors": ["heartbeat:ingestor:1"]
+}

e2e/scenario_inputs/orchestrator_tag_database.json

@@ -0,0 +1,71 @@
+{
+  "workflow_input": {
+    "schedule_name": "orchestrator-e2e-tag-database",
+    "pipelines_query": {
+      "collection": "pipelines",
+      "aggregation": [{"$match": {"active": true}}]
+    }
+  },
+  "connectors": [
+    {
+      "connector_id": "plant",
+      "type": "opc_ua",
+      "name": "plant",
+      "url": "opc.tcp://plant:4840",
+      "subscription_period_ms": 500,
+      "security": {"mode": "SignAndEncrypt", "policy": "Basic256Sha256", "client_certificate_id": 1},
+      "status": "Active",
+      "updated_at": "@now-1d"
+    }
+  ],
+  "opc_certificates": [
+    {"certificate_id": 1, "kind": "client", "name": "sientia-platform", "status": "active",
+     "application_uri": "urn:sientia:opc-ingestor",
+     "cert": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/cert.pem", "sha256": null},
+     "private_key": {"bucket": "opc-certificates", "object_name": "client/sientia-platform/key.pem", "sha256": null},
+     "thumbprint_sha1": null, "not_before": "@now-1d", "not_after": "@now+365d",
+     "created_at": "@now-1d", "updated_at": "@now-1d", "created_by": "e2e", "updated_by": "e2e"},
+    {"certificate_id": 2, "kind": "client", "name": "retired", "status": "retired",
+     "application_uri": "urn:sientia:old",
+     "cert": {"bucket": "opc-certificates", "object_name": "client/retired/cert.pem"},
+     "private_key": {"bucket": "opc-certificates", "object_name": "client/retired/key.pem"},
+     "created_at": "@now-1d", "updated_at": "@now-1d", "created_by": "e2e", "updated_by": "e2e"}
+  ],
+  "connector_tags": [
+    {"tag_id": 7, "connector_id": "plant", "address": "ns=3;i=1001", "tag_name": "Pressure", "category": "Read", "status": "Active", "updated_at": "@now-1d"},
+    {"tag_id": 101, "connector_id": "plant", "address": "ns=3;i=2001", "tag_name": "PRED-101", "category": "Write", "data_type": "double", "status": "Active", "updated_at": "@now-1d"},
+    {"tag_id": 102, "connector_id": "plant", "address": "ns=3;i=2002", "tag_name": "CONF-101", "category": "Prediction Confidence", "status": "Active", "updated_at": "@now-1d"},
+    {"tag_id": 103, "connector_id": "plant", "address": "ns=3;i=2003", "tag_name": "PRED-OLD", "category": "Write", "status": "Inactive", "updated_at": "@now-1d"}
+  ],
+  "pipelines": [
+    {
+      "schedule_name": "opc-tagdb",
+      "workflow_type": "scouter",
+      "model_id": "model-opc",
+      "model": {"name": "Model model-opc"},
+      "active": true,
+      "updated_at": "@now-2d",
+      "frequency": "1m",
+      "read_tags": [{"tag_id": 7, "frequency": 1000, "aggr_func": "avg"}]
+    },
+    {
+      "schedule_name": "pred-tagdb",
+      "workflow_type": "predictions_batch",
+      "model_id": "model-opc",
+      "model": {"name": "Model model-opc"},
+      "active": true,
+      "updated_at": "@now-2d",
+      "frequency": "1m",
+      "query": "SELECT 1",
+      "write_tags": [
+        {"tag_id": 101},
+        {"tag_id": 102},
+        {"tag_id": 7},
+        {"tag_id": 103},
+        {"tag_id": 404},
+        {"tag_name": "PRED-NO-ID"}
+      ]
+    }
+  ],
+  "active_ingestors": ["heartbeat:ingestor:1"]
+}

e2e/scenario_inputs/orchestrator_ttl_index.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -22,24 +18,17 @@
       "frequency": "1m",
       "offset": "0m",
       "read_tags": [
-        {
-          "server_id": "srv-1",
-          "tag_name": "Read1",
-          "tag_address": "ns=2;s=Read1",
-          "aggr_func": "lts",
-          "frequency": 1000
-        }
+        {"tag_id": 1, "aggr_func": "lts", "frequency": 1000}
       ]
     }
   ],
-  "opc_servers": [
-    {
-      "id": "srv-1",
-      "active": true,
-      "server_name": "opc-1",
-      "url": "opc.tcp://localhost:4840",
-      "uri": "urn:opcfoundation:UA:DemoServer"
-    }
+  "connectors": [
+    {"connector_id": "opc-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://localhost:4840",
+     "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+  ],
+  "connector_tags": [
+    {"tag_id": 1, "connector_id": "opc-1", "address": "ns=2;s=Read1", "tag_name": "Read1",
+     "category": "Read", "status": "Active"}
   ],
   "active_ingestors": ["heartbeat:ingestor:1"]
 }

e2e/scenario_inputs/orchestrator_update_only.json

@@ -4,10 +4,6 @@
     "pipelines_query": {
       "collection": "pipelines",
       "aggregation": [{"$match": {"active": true}}]
-    },
-    "opc_servers_query": {
-      "collection": "opc_servers",
-      "filters": {"active": true}
     }
   },
   "pipelines": [
@@ -22,13 +18,16 @@
       "frequency": "5m",
       "offset": "0m",
       "query": "SELECT 1",
-      "write_tags": [
-        {"server_id": "srv-1", "type": "prediction", "addr": "ns=2;s=Tag1"}
-      ]
+      "write_tags": [{"tag_id": 101}]
     }
   ],
-  "opc_servers": [
-    {"id": "srv-1", "active": true}
+  "connectors": [
+    {"connector_id": "srv-1", "type": "opc_ua", "name": "opc-1", "url": "opc.tcp://opc-1:4840",
+     "security": {"mode": "None", "policy": "None"}, "status": "Active"}
+  ],
+  "connector_tags": [
+    {"tag_id": 101, "connector_id": "srv-1", "address": "ns=2;s=Tag1", "tag_name": "PRED-1",
+     "category": "Write", "status": "Active", "updated_at": "@now-1d"}
   ],
   "active_ingestors": ["heartbeat:ingestor:1"],
   "orchestrated_schedules": [

e2e/scenarios.md

@@ -6,6 +6,11 @@ Functional reference for orchestrator E2E scenarios. Tests live under `e2e/`, us
 pytest e2e/ --override-ini testpaths=e2e -m e2e
 ```
 
+**Every section below maps 1:1 to an actual test function** (named in each section, or as a bullet
+under a `Source:` heading) — this file describes what the suite verifies today, not planned or
+aspirational coverage. Adding a scenario here without a matching test (or vice versa) is a doc bug;
+fix the mismatch instead of leaving it for later.
+
 ## Execution context
 
 - MongoDB, Redis, PostgreSQL: testcontainers (session-scoped).
@@ -21,36 +26,74 @@ Source: `e2e/test_orchestrator_main_workflow.py`
 
 ### 1.1.1 Happy path
 
-Pipelines in Mongo → schedules created in correct namespace/task queue, Redis slots written, `orchestrated_schedules` updated.
-
-### 1.2.1 No-op tick
-
-Mongo, Redis, and Temporal already match desired state → no new schedules or slot writes.
+Pipelines in Mongo → both Temporal schedules created in the `laborious` namespace, one
+`orchestrated_schedules` entry per pipeline, and the `predictions_batch` write tag (`tag_id`)
+resolved into `opc_output_config` through the tag database.
 
 ### 1.3.1 Create-only
 
-New pipeline only → schedules created, timestamps inserted.
-
-### 1.3.2 Update-only
-
-Existing pipeline with newer `updated_at` → schedule updated in Temporal.
+New pipeline only → Temporal schedule created for it.
 
 ### 1.3.3 Delete-only
 
-Pipeline removed from Mongo → schedule deleted from Temporal.
-
-### 1.4.1 Conflict ordering
-
-Pipeline update and slot delete on same OPC server → slot insert before delete (production ordering).
+Pipeline removed from Mongo → previously-created Temporal schedule deleted.
 
 ### 1.5.1 Empty pipelines
 
-No active pipelines → orphan schedules removed, no new orchestration writes.
+No active pipelines → no `orchestrated_schedules` documents written, even with a pre-existing
+unrelated Temporal schedule present.
 
 ### 1.6.1 TTL index bootstrap
 
 First run creates TTL index on notification collection used by scouter pipelines.
 
+### 1.7.1 Simple metrics parallel windows
+
+Two `simple_metrics` pipelines for the same `model_id` (60 min and 1440 min windows) → one
+Temporal schedule per window, each with its own `frequency` as interval.
+
+### 1.7.2 Adaptive frequency escalation/recovery round trip
+
+Laborious-style write (`adaptive_state.escalated=true` + BSON-datetime `updated_at`) on one
+window → next run switches only that schedule to `frequency_alert`; the recovery write switches it
+back to `frequency`.
+
+### 1.8.1 Drift window and chunk_period
+
+Two `drift` pipelines on `frequency='1m'`: one without `interval_minutes` and with an invalid
+`chunk_period='hour'` → workflow input `interval=1` (frequency, no overlap),
+`chunk_period='min'` and exactly one `DRIFT_CHUNK_PERIOD_INVALID` notification; one with
+`interval_minutes=60` and `chunk_period='s'` → both forwarded unchanged, no warning
+(SIENTIAPDE-2212).
+
+### 1.9 OPC tag database: read and write tags by tag_id
+
+Scenario input: `orchestrator_tag_database.json` (one `opc_ua` connector, its `connector_tags` of
+every category, one inactive, an OPC scouter reading by `tag_id` and a `predictions_batch` writing by
+`tag_id`, with a Read tag, an inactive tag, an unknown `tag_id` and an entry without `tag_id` among its
+write tags).
+
+#### 1.9.1 Write tags through the tag database
+
+`write_tags: [{tag_id}]` → `opc_output_config` keyed by `connector_id`, address and `data_type`
+from `connector_tags` (`float` when absent), `Write` → `prediction_tags`, `Prediction Confidence` →
+`confidence_tags`. The Read, inactive and unknown tags are left out and notified once
+(`ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`); the entry without `tag_id` is notified
+(`ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION`).
+
+#### 1.9.2 A tag change redeploys its pipelines
+
+A second run with nothing changed redeploys nothing (`orchestrated_schedules` is not stamped
+again). Then renaming the read tag, moving the write tag and deleting the confidence tag in
+`connector_tags` by hand (no `updated_at` bumped anywhere, pipelines untouched) → the next run
+redeploys both through their `config_hash`: the scouter `model_tags` and the slot `names` take
+the new name, the `opc_output_config` the new address without the confidence tag, and
+`orchestrated_schedules` records the new `config_hash`.
+
+#### 1.9.3 OPC slot from the tag database
+
+The slot v2 written to Redis holds only the active Read tag, with the connector's server fields.
+
 ---
 
 ## 2. Alerts workflow
@@ -65,18 +108,20 @@ ERROR notification → one SMTP message, one `log_report` row, Redis cache key.
 
 Second run with same data and cache seeded → no extra email or log row.
 
-### A.1.3 Persistent escalation
-
-Alert past `notification_ttl` with cache cleared → new email sent.
-
 ### A.2.1 Group filtering
 
-Receiver group `levels` / `ignore_models_id` honored. The query-time half
-(union of `levels`, intersection of `ignore_models_id` across active groups)
-is covered at the subworkflow level — see `LoadNotificationPackage` below.
-The per-group half (each group only receiving its own precise slice) is not
-yet implemented in production code (SIENTIAPDE-2012) and so is not yet
-exercised here.
+Receiver group `levels` / `ignore_models_id` honored, both halves now covered:
+the query-time union/intersection across active groups (SIENTIAPDE-2011, see
+`LoadNotificationPackage` below) and the per-group precise slice
+(SIENTIAPDE-2012) — a group only receives notifications matching its own
+`levels`/`ignore_models_id`, even when the query-time filter widened the fetch
+to satisfy another active group's broader configuration.
+
+### A.2.2 Model_id allow-list
+
+Receiver group `model_ids` (SIENTIAPDE-2149) honored: a group configuring it only
+receives notifications from those models, even when another active group with
+no such restriction widens the query-time fetch to include every model.
 
 ### A.3.1 Empty queue
 
@@ -109,7 +154,10 @@ No notifications → no SMTP, no Postgres row.
 - No prior Redis timestamp → all matching notifications returned, max timestamp stored.
 - Prior timestamp → only newer notifications returned.
 - Query-time filter (SIENTIAPDE-2011) → union of active groups' `levels`, intersection of their `ignore_models_id`; a model ignored by only one of several active groups is not excluded from the fetch.
+- Model_id allow-list union at query time (SIENTIAPDE-2149) → when every active group sets `model_ids`, the fetch is restricted to the union of every group's allow-list; a model outside that union is never fetched at all, even for a group that would otherwise want it via a different axis.
 - Empty Mongo → no Redis timestamp write.
+- Batch size cap (SIENTIAPDE-2013) → with 5 pending notifications and a batch size of 2, each run returns exactly the next-oldest 2 (sorted ascending by timestamp), and the Redis cursor advances only to what was actually returned, so three successive runs drain the backlog in order instead of skipping anything.
+- Batch size 0 means unlimited (SIENTIAPDE-2013) → the configured default (`0`) does not cap the query; 12 pending notifications all come back in one run, verified against a real MongoDB instance rather than assumed from pymongo's documented convention.
 
 ### ProcessNotifications — `e2e/test_subworkflow_process_notifications.py`
 

e2e/test_alerts_main_workflow.py

@@ -12,6 +12,7 @@ from e2e.helpers import (
     count_log_report_rows,
     default_notification,
     default_receiver_group,
+    extract_html_body,
     fetch_log_report,
     load_scenario_input,
     make_workflow_id,
@@ -92,6 +93,162 @@ async def test_scenario_a_1_2_duplicate_suppressed(
     assert count_log_report_rows(postgres_engine, 'Alerts') == 0
 
 
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_a_2_1_per_group_levels_and_ignore_models_id(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    smtp_server,
+):
+    """
+    A.2.1 (SIENTIAPDE-2012): each receiver group gets only its own precise
+    slice of notifications, on top of SIENTIAPDE-2011's query-time fetch.
+
+    Two active groups widen the query-time union beyond Alerts' ERROR-only
+    default: group-errors (levels=['ERROR']) and group-warnings
+    (levels=['WARNING'], ignore_models_id=['model-x']). Because the query-time
+    filter only unions/intersects across ALL active groups, both ERROR and
+    WARNING notifications get fetched for every group, and model-x is not
+    excluded at the query level either (group-errors has no ignore_models_id
+    restriction, so the intersection is empty). The per-group filter added in
+    this subtask is what actually narrows each group back down to what it
+    configured.
+    """
+    seed_receiver_groups(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_receiver_group(
+                group_name='group-errors', contents=['core_alerts'], levels=['ERROR']
+            ),
+            default_receiver_group(
+                group_name='group-warnings',
+                contents=['core_alerts'],
+                levels=['WARNING'],
+                ignore_models_id=['model-x'],
+            ),
+        ],
+    )
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(
+                notification_id='err-a', level='ERROR', model_name='model-a', model_id='model-a'
+            ),
+            default_notification(
+                notification_id='warn-a',
+                level='WARNING',
+                model_name='model-a',
+                model_id='model-a',
+                timestamp=datetime(2024, 6, 1, 12, 1, 0, tzinfo=UTC),
+            ),
+            default_notification(
+                notification_id='warn-x',
+                level='WARNING',
+                model_name='model-x',
+                model_id='model-x',
+                timestamp=datetime(2024, 6, 1, 12, 2, 0, tzinfo=UTC),
+            ),
+        ],
+    )
+
+    await start_and_await_workflow(
+        temporal_env.client,
+        Alerts.run,
+        load_scenario_input('alerts_happy_path.json'),
+        make_workflow_id('alerts-2012'),
+    )
+
+    # One email per group -- each group ended up with exactly one notification.
+    assert len(smtp_server.messages) == 2
+    bodies = [extract_html_body(message) for message in smtp_server.messages]
+    combined = '\n'.join(bodies)
+
+    # group-errors (levels=['ERROR']): received err-a, never warn-a/warn-x
+    # despite both being fetched by the widened query-time union.
+    errors_body = next(b for b in bodies if 'Errors detected:' in b)
+    assert 'Warnings detected:' not in errors_body
+
+    # group-warnings (levels=['WARNING'], ignore_models_id=['model-x']):
+    # received warn-a only -- warn-x was fetched (query-time intersection is
+    # empty, since group-errors doesn't ignore any model) but dropped here by
+    # the per-group ignore_models_id check.
+    warnings_body = next(b for b in bodies if 'Warnings detected:' in b)
+    assert 'Errors detected:' not in warnings_body
+    assert 'model-a' in combined
+    assert 'model-x' not in combined
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_a_2_2_model_ids_allow_list(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    smtp_server,
+):
+    """
+    A.2.2 (SIENTIAPDE-2149): a group configuring `model_ids` only receives
+    notifications from those models, on top of the existing query-time/per-group
+    filtering (SIENTIAPDE-2011/2012).
+
+    group-allow (model_ids=['model-a']) coexists with group-everything (no
+    model_ids restriction). Because the query-time filter only restricts to an
+    allow-list union when EVERY active group has one, group-everything's lack
+    of a model_ids restriction means the query itself fetches both models --
+    the per-group filter added by this ticket is what narrows group-allow back
+    down to model-a only.
+    """
+    seed_receiver_groups(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_receiver_group(
+                group_name='group-allow', contents=['core_alerts'], model_ids=['model-a']
+            ),
+            default_receiver_group(group_name='group-everything', contents=['core_alerts']),
+        ],
+    )
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(
+                notification_id='err-a', level='ERROR', model_name='model-a', model_id='model-a'
+            ),
+            default_notification(
+                notification_id='err-b',
+                level='ERROR',
+                model_name='model-b',
+                model_id='model-b',
+                timestamp=datetime(2024, 6, 1, 12, 1, 0, tzinfo=UTC),
+            ),
+        ],
+    )
+
+    await start_and_await_workflow(
+        temporal_env.client,
+        Alerts.run,
+        load_scenario_input('alerts_happy_path.json'),
+        make_workflow_id('alerts-2149'),
+    )
+
+    # One email per group -- group-allow got only err-a, group-everything got both.
+    assert len(smtp_server.messages) == 2
+    bodies = [extract_html_body(message) for message in smtp_server.messages]
+
+    # Distinguish the two emails by content rather than order: only
+    # group-everything's email mentions model-b.
+    allow_body = next(b for b in bodies if 'model-b' not in b)
+    everything_body = next(b for b in bodies if 'model-b' in b)
+    assert 'model-a' in allow_body
+    assert 'model-a' in everything_body
+
+
 @pytest.mark.e2e
 @pytest.mark.asyncio
 async def test_scenario_a_3_1_empty_queue(

e2e/test_orchestrator_main_workflow.py

@@ -1,5 +1,8 @@
 """E2E tests for the Orchestrator main workflow."""
 
+import json
+from datetime import UTC, datetime, timedelta
+
 import pytest
 from pymongo import MongoClient
 from redis import Redis
@@ -12,9 +15,9 @@ from e2e.helpers import (
     load_scenario_input,
     make_workflow_id,
     seed_active_ingestors,
-    seed_opc_servers,
     seed_orchestrated_schedules,
     seed_pipelines,
+    seed_tag_database,
     start_and_await_workflow,
 )
 from orchestrator.workflows.orchestrator import Orchestrator
@@ -26,13 +29,20 @@ async def _schedule_ids(host: str, namespace: str) -> list[str]:
     return [schedule.id async for schedule in await client.list_schedules()]
 
 
+async def _schedule_every(host: str, schedule_id: str) -> timedelta:
+    """Return the interval of a laborious-namespace schedule."""
+    client = await Client.connect(host, namespace='laborious')
+    description = await client.get_schedule_handle(schedule_id).describe()
+    return description.schedule.spec.intervals[0].every
+
+
 def _apply_orchestrator_seeds(
     scenario: dict,
     mongo_uri: str,
     redis_client: Redis,
 ) -> None:
     """
-    Seed Mongo and Redis with the pipelines/opc_servers/ingestors declared in a scenario.
+    Seed Mongo and Redis with the pipelines/tag database/ingestors of a scenario.
 
     Args:
         scenario: Scenario payload returned by ``load_scenario_input``.
@@ -40,7 +50,13 @@ def _apply_orchestrator_seeds(
         redis_client: Redis client connected to the test instance.
     """
     seed_pipelines(mongo_uri, E2E_DATABASE, scenario.get('pipelines', []))
-    seed_opc_servers(mongo_uri, E2E_DATABASE, scenario.get('opc_servers', []))
+    seed_tag_database(
+        mongo_uri,
+        E2E_DATABASE,
+        scenario.get('connectors', []),
+        scenario.get('connector_tags', []),
+        scenario.get('opc_certificates', []),
+    )
     seed_active_ingestors(redis_client, scenario.get('active_ingestors', []))
     if scenario.get('orchestrated_schedules'):
         seed_orchestrated_schedules(
@@ -75,14 +91,19 @@ async def test_scenario_1_1_1_happy_path(
     try:
         tracked = list(mongo[E2E_DATABASE]['orchestrated_schedules'].find())
         names = {doc['schedule_name'] for doc in tracked}
-        assert names, f'Expected orchestrated_schedules rows, got {tracked}'
+        assert names == {'pred-legacy', 'drift-gpu'}, f'Unexpected orchestrated_schedules {tracked}'
     finally:
         mongo.close()
 
     laborious_schedules = await _schedule_ids(temporal_host, 'laborious')
-    assert 'pred-legacy' in laborious_schedules or 'drift-gpu' in laborious_schedules, (
+    assert {'pred-legacy', 'drift-gpu'} <= set(laborious_schedules), (
         f'Expected Temporal schedules in laborious, got {laborious_schedules}'
     )
+    # The write tag comes from the tag database, keyed by the connector the gateway writes to
+    pred = await _schedule_input(temporal_host, 'pred-legacy')
+    assert pred['opc_output_config'] == {
+        'srv-1': {'prediction_tags': {'ns=2;s=Tag1': {'data_type': 'float'}}}
+    }
 
 
 @pytest.mark.e2e
@@ -224,3 +245,326 @@ async def test_scenario_1_6_1_ttl_index_bootstrap(
         assert any('expireAfterSeconds' in info for info in indexes.values())
     finally:
         mongo.close()
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_7_1_simple_metrics_parallel_windows(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    temporal_host: str,
+    redis_client: Redis,
+):
+    """1.7.1: One pipelines doc per window gives one schedule per window, each with its own interval."""
+    scenario = load_scenario_input('orchestrator_simple_metrics_windows')
+    _apply_orchestrator_seeds(scenario, mongo_uri, redis_client)
+
+    await start_and_await_workflow(
+        temporal_env.client,
+        Orchestrator.run,
+        scenario['workflow_input'],
+        make_workflow_id('orchestrator-sm-windows'),
+    )
+
+    short, long = 'model-sm-simple_metrics-60m', 'model-sm-simple_metrics-1440m'
+    assert await _schedule_every(temporal_host, short) == timedelta(hours=1)
+    assert await _schedule_every(temporal_host, long) == timedelta(days=1)
+
+
+def _laborious_adaptive_write(mongo_uri: str, schedule_name: str, escalated: bool) -> None:
+    """Replicates laborious `update_simple_metrics_adaptive_state` on an escalated flip."""
+    mongo = MongoClient(mongo_uri)
+    try:
+        mongo[E2E_DATABASE]['pipelines'].update_one(
+            {'schedule_name': schedule_name},
+            {
+                '$set': {
+                    'adaptive_state': {'escalated': escalated, 'healthy_streak': 0},
+                    # BSON datetime, as laborious writes it; a string would break the tick.
+                    'updated_at': datetime.now(UTC),
+                }
+            },
+        )
+    finally:
+        mongo.close()
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_7_2_simple_metrics_adaptive_frequency_round_trip(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    temporal_host: str,
+    redis_client: Redis,
+):
+    """1.7.2: Escalation written by laborious switches only that window to frequency_alert; recovery switches it back."""
+    scenario = load_scenario_input('orchestrator_simple_metrics_windows')
+    _apply_orchestrator_seeds(scenario, mongo_uri, redis_client)
+    short, long = 'model-sm-simple_metrics-60m', 'model-sm-simple_metrics-1440m'
+
+    async def run(tag: str) -> None:
+        await start_and_await_workflow(
+            temporal_env.client,
+            Orchestrator.run,
+            scenario['workflow_input'],
+            make_workflow_id(f'orchestrator-sm-adaptive-{tag}'),
+        )
+
+    await run('create')
+    assert await _schedule_every(temporal_host, short) == timedelta(hours=1)
+
+    _laborious_adaptive_write(mongo_uri, short, escalated=True)
+    await run('escalate')
+    assert await _schedule_every(temporal_host, short) == timedelta(minutes=5)
+    assert await _schedule_every(temporal_host, long) == timedelta(days=1)
+
+    _laborious_adaptive_write(mongo_uri, short, escalated=False)
+    await run('recover')
+    assert await _schedule_every(temporal_host, short) == timedelta(hours=1)
+
+
+async def _schedule_input(host: str, schedule_id: str, namespace: str = 'laborious') -> dict:
+    """Return the decoded workflow input of a schedule (laborious namespace by default)."""
+    client = await Client.connect(host, namespace=namespace)
+    description = await client.get_schedule_handle(schedule_id).describe()
+    action = description.schedule.action
+    assert isinstance(action, ScheduleActionStartWorkflow)
+    [input_data] = await client.data_converter.decode(action.args)
+    return input_data
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_8_1_drift_window_and_chunk_period(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    temporal_host: str,
+    redis_client: Redis,
+    notification_handler,
+):
+    """1.8.1: Drift window defaults to frequency; invalid chunk_period warns and falls back."""
+    scenario = load_scenario_input('orchestrator_drift_window')
+    _apply_orchestrator_seeds(scenario, mongo_uri, redis_client)
+
+    await start_and_await_workflow(
+        temporal_env.client,
+        Orchestrator.run,
+        scenario['workflow_input'],
+        make_workflow_id('orchestrator-drift-window'),
+    )
+
+    default = await _schedule_input(temporal_host, 'model-dr-drift-default')
+    assert default['interval'] == 1
+    assert default['chunk_period'] == 'min'
+    assert default['chunk_period_configured'] == 'hour'
+
+    explicit = await _schedule_input(temporal_host, 'model-dr-drift-explicit')
+    assert explicit['interval'] == 60
+    assert explicit['chunk_period'] == 's'
+    assert 'chunk_period_configured' not in explicit
+
+    warnings = list(
+        notification_handler.mongo_collection.find(
+            {'notification_id': 'DRIFT_CHUNK_PERIOD_INVALID'}
+        )
+    )
+    assert len(warnings) == 1
+    assert warnings[0]['model_id'] == 'model-dr'
+
+
+# 1.9 OPC tag database: read and write tags by tag_id (SIENTIAPDE-1974)
+
+
+async def _run_tag_database_scenario(temporal_env: WorkflowEnvironment, tag: str) -> None:
+    await start_and_await_workflow(
+        temporal_env.client,
+        Orchestrator.run,
+        load_scenario_input('orchestrator_tag_database')['workflow_input'],
+        make_workflow_id(f'orchestrator-tagdb-{tag}'),
+    )
+
+
+def _notifications(notification_handler, notification_id: str) -> list[dict]:
+    return list(notification_handler.mongo_collection.find({'notification_id': notification_id}))
+
+
+def _opc_slots(redis_client: Redis) -> dict[str, dict]:
+    return {
+        key if isinstance(key, str) else key.decode(): json.loads(redis_client.get(key))
+        for key in redis_client.keys('slot:opc_tags:*')
+    }
+
+
+def _orchestrated_schedules(mongo_uri: str) -> dict[str, dict]:
+    """orchestrated_schedules by schedule_name: updated_at (last deploy) and config_hash."""
+    mongo = MongoClient(mongo_uri)
+    try:
+        return {
+            doc['schedule_name']: {
+                'updated_at': doc['updated_at'],
+                'config_hash': doc.get('config_hash'),
+            }
+            for doc in mongo[E2E_DATABASE]['orchestrated_schedules'].find()
+        }
+    finally:
+        mongo.close()
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_9_1_write_tags_through_tag_database(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    temporal_host: str,
+    redis_client: Redis,
+    notification_handler,
+):
+    """1.9.1: write_tags by tag_id become the opc_output_config; the rest is notified."""
+    _apply_orchestrator_seeds(
+        load_scenario_input('orchestrator_tag_database'), mongo_uri, redis_client
+    )
+
+    await _run_tag_database_scenario(temporal_env, 'write')
+
+    pred = await _schedule_input(temporal_host, 'pred-tagdb')
+    assert pred['opc_output_config'] == {
+        'plant': {
+            'prediction_tags': {'ns=3;i=2001': {'data_type': 'double'}},
+            'confidence_tags': {'ns=3;i=2002': {'data_type': 'float'}},
+        }
+    }
+
+    not_found = _notifications(
+        notification_handler, 'ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION'
+    )
+    assert len(not_found) == 1
+    # A Read tag, an inactive tag and an unknown tag_id
+    for value in ('pred-tagdb/7', 'pred-tagdb/103', 'pred-tagdb/404'):
+        assert value in not_found[0]['message']
+    invalid = _notifications(
+        notification_handler, 'ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION'
+    )
+    assert len(invalid) == 1
+    assert 'PRED-NO-ID' in invalid[0]['message']
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_9_2_tag_change_redeploys_pipelines(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    temporal_host: str,
+    redis_client: Redis,
+):
+    """
+    1.9.2: changing or deleting a tag in connector_tags, without touching any updated_at,
+    redeploys the pipelines that reference it (config_hash); an unchanged run does not.
+    """
+    _apply_orchestrator_seeds(
+        load_scenario_input('orchestrator_tag_database'), mongo_uri, redis_client
+    )
+    await _run_tag_database_scenario(temporal_env, 'before-change')
+
+    scouter = await _schedule_input(temporal_host, 'opc-tagdb', namespace='scouter')
+    assert set(scouter['model_tags']) == {'Pressure'}
+    deployed = _orchestrated_schedules(mongo_uri)
+    assert all(doc.get('config_hash') for doc in deployed.values()), deployed
+
+    # Nothing changed: no redeploy, so orchestrated_schedules is not stamped again
+    await _run_tag_database_scenario(temporal_env, 'unchanged')
+    assert _orchestrated_schedules(mongo_uri) == deployed
+
+    # As an edit by hand in Compass: rename the read tag, move a write tag and delete the
+    # confidence tag, with no updated_at bumped anywhere
+    mongo = MongoClient(mongo_uri)
+    try:
+        tags = mongo[E2E_DATABASE]['connector_tags']
+        tags.update_one({'tag_id': 7}, {'$set': {'tag_name': 'Pressao'}})
+        tags.update_one({'tag_id': 101}, {'$set': {'address': 'ns=3;i=3001'}})
+        tags.delete_one({'tag_id': 102})
+    finally:
+        mongo.close()
+
+    await _run_tag_database_scenario(temporal_env, 'after-change')
+
+    scouter = await _schedule_input(temporal_host, 'opc-tagdb', namespace='scouter')
+    assert set(scouter['model_tags']) == {'Pressao'}
+    pred = await _schedule_input(temporal_host, 'pred-tagdb')
+    assert pred['opc_output_config'] == {
+        'plant': {'prediction_tags': {'ns=3;i=3001': {'data_type': 'double'}}}
+    }
+    redeployed = _orchestrated_schedules(mongo_uri)
+    for name in ('opc-tagdb', 'pred-tagdb'):
+        assert redeployed[name]['config_hash'] != deployed[name]['config_hash']
+        assert redeployed[name]['updated_at'] > deployed[name]['updated_at']
+    # The slot names the read tag like the scouter does (SC1)
+    [slot] = _opc_slots(redis_client).values()
+    [name] = slot['plant']['tags']['ns=3;i=1001']['names']
+    assert name['tag_name'] == 'Pressao'
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_9_3_opc_slot_from_tag_database(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    redis_client: Redis,
+):
+    """1.9.3: the slot v2 holds only the active Read tags of the tag database."""
+    _apply_orchestrator_seeds(
+        load_scenario_input('orchestrator_tag_database'), mongo_uri, redis_client
+    )
+
+    await _run_tag_database_scenario(temporal_env, 'slot')
+
+    assert _opc_slots(redis_client) == {
+        'slot:opc_tags:1': {
+            'plant': {
+                'server_id': 'plant',
+                'name': 'plant',
+                'url': 'opc.tcp://plant:4840',
+                'subscription_period_ms': 500,
+                # The active client certificate of client_certificate_id, without its
+                # dates, sha256 or audit fields (D5)
+                'security': {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'},
+                'certificate': {
+                    'certificate_id': 1,
+                    'application_uri': 'urn:sientia:opc-ingestor',
+                    'cert': {
+                        'bucket': 'opc-certificates',
+                        'object_name': 'client/sientia-platform/cert.pem',
+                    },
+                    'private_key': {
+                        'bucket': 'opc-certificates',
+                        'object_name': 'client/sientia-platform/key.pem',
+                    },
+                },
+                'tags': {
+                    'ns=3;i=1001': {
+                        'frequency': 1000,
+                        'active_read': {'enabled': False},
+                        'names': [
+                            {
+                                'collection': 'raw_opc-tagdb',
+                                'tag_name': 'Pressure',
+                                'model_id': 'model-opc',
+                                'model_name': 'Model model-opc',
+                            }
+                        ],
+                    }
+                },
+            }
+        }
+    }

e2e/test_reports_main_workflow.py

@@ -1,7 +1,6 @@
 """E2E tests for the Reports main workflow."""
 
 from datetime import UTC, datetime
-from email.message import EmailMessage, Message
 
 import pytest
 from temporalio.testing import WorkflowEnvironment
@@ -11,6 +10,7 @@ from e2e.helpers import (
     count_log_report_rows,
     default_notification,
     default_receiver_group,
+    extract_html_body,
     load_scenario_input,
     make_workflow_id,
     seed_notifications,
@@ -20,41 +20,6 @@ from e2e.helpers import (
 from orchestrator.workflows.reports import Reports
 
 
-def _extract_html_body(message: EmailMessage | Message) -> str:
-    """
-    Return the text/html portion of an email message as a decoded string.
-
-    Walks every part looking for the first text/html payload, decoding it
-    according to the part's transfer encoding and charset. Falls back to
-    the message's own ``get_content``/raw payload when no HTML part is
-    present so callers can still inspect plain-text reports.
-
-    Args:
-        message: Captured email message returned by the test SMTP server.
-
-    Return:
-        str: HTML body content, or an empty string when nothing decodable
-            is found.
-    """
-    if message.is_multipart():
-        for part in message.walk():
-            if part.get_content_type() != 'text/html':
-                continue
-            payload = part.get_payload(decode=True)
-            if payload is None:
-                continue
-            charset = part.get_content_charset() or 'utf-8'
-            return payload.decode(charset, errors='replace')
-    payload = message.get_payload(decode=True)
-    if payload is not None:
-        charset = message.get_content_charset() or 'utf-8'
-        return payload.decode(charset, errors='replace')
-    try:
-        return message.get_content()
-    except (AttributeError, KeyError):
-        return str(message.get_payload() or '')
-
-
 @pytest.mark.e2e
 @pytest.mark.asyncio
 async def test_scenario_r_1_1_happy_path(
@@ -99,7 +64,7 @@ async def test_scenario_r_1_1_happy_path(
     )
 
     assert len(smtp_server.messages) == 1
-    body = _extract_html_body(smtp_server.messages[0])
+    body = extract_html_body(smtp_server.messages[0])
     assert 'Errors detected:' in body
     assert 'Warnings detected:' in body
     assert 'Infos detected:' in body
@@ -138,7 +103,7 @@ async def test_scenario_r_1_2_error_section_only(
     )
 
     assert smtp_server.messages, 'Expected at least one report email'
-    body = _extract_html_body(smtp_server.messages[0])
+    body = extract_html_body(smtp_server.messages[0])
     assert 'Errors detected:' in body
     assert 'Warnings detected:' not in body
     assert 'Infos detected:' not in body

e2e/test_subworkflow_load_notification_package.py

@@ -205,6 +205,57 @@ async def test_load_package_query_time_filter_union_and_intersection(
     assert ids == {'error-included', 'warning-included', 'modely-included'}
 
 
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_load_package_query_time_filter_model_ids_union(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    redis_client: Redis,
+):
+    """
+    SIENTIAPDE-2149: query-time filter unions active groups' `model_ids`
+    allow-list when every active group sets one, excluding any model outside
+    that union from the fetch entirely -- not just from a single group's
+    per-group slice.
+
+    group_a wants only model-a, group_b wants only model-b -- both restrict
+    levels to ERROR so only the model_ids axis is under test. model-c is
+    outside the union of both allow-lists and must never be fetched.
+    """
+    seed_receiver_groups(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_receiver_group(group_name='group_a', levels=['ERROR'], model_ids=['model-a']),
+            default_receiver_group(group_name='group_b', levels=['ERROR'], model_ids=['model-b']),
+        ],
+    )
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(notification_id='a-included', level='ERROR', model_id='model-a'),
+            default_notification(notification_id='b-included', level='ERROR', model_id='model-b'),
+            default_notification(notification_id='c-excluded', level='ERROR', model_id='model-c'),
+        ],
+    )
+
+    input_data = load_scenario_input('subworkflow_load_notification_package.json')
+    _build_metadata(input_data)
+
+    result = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-model-ids-union'),
+    )
+
+    ids = {n['notification_id'] for n in result['notification_package']}
+    assert ids == {'a-included', 'b-included'}
+
+
 @pytest.mark.e2e
 @pytest.mark.asyncio
 async def test_load_package_empty_mongo_no_redis_write(
@@ -228,3 +279,114 @@ async def test_load_package_empty_mongo_no_redis_write(
     )
 
     assert redis_client.get('notification_last_timestamp:Alerts') is None
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_load_package_batch_size_caps_and_returns_oldest_first(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    redis_client: Redis,
+    test_activities,
+):
+    """
+    SIENTIAPDE-2013: a capped run returns the oldest pending documents first,
+    and the incremental cursor picks up the remainder on the next run.
+
+    Five notifications exist, but the batch size is capped at 2. Each run must
+    return exactly the next-oldest 2 (never an arbitrary subset), and the
+    Redis cursor must advance only to the max timestamp of what was actually
+    returned -- not the true max across all 5 -- or the skipped documents
+    would never be fetched again.
+    """
+    test_activities.notification_queue_batch_size = 2
+
+    seed_receiver_groups(mongo_uri, E2E_DATABASE, [default_receiver_group()])
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(
+                notification_id=f'n{i}',
+                timestamp=datetime(2024, 6, 1, 12, i, 0, tzinfo=UTC),
+            )
+            for i in range(1, 6)
+        ],
+    )
+
+    input_data = load_scenario_input('subworkflow_load_notification_package.json')
+    _build_metadata(input_data)
+
+    first_run = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-batch-1'),
+    )
+    assert [n['notification_id'] for n in first_run['notification_package']] == ['n1', 'n2']
+
+    second_run = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-batch-2'),
+    )
+    assert [n['notification_id'] for n in second_run['notification_package']] == ['n3', 'n4']
+
+    third_run = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-batch-3'),
+    )
+    assert [n['notification_id'] for n in third_run['notification_package']] == ['n5']
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_load_package_batch_size_zero_means_unlimited(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    test_activities,
+):
+    """
+    SIENTIAPDE-2013: notification_queue_batch_size=0 (the config default) must
+    NOT cap the query. This exercises pymongo's own documented convention --
+    `Collection.find(..., limit=0)` is equivalent to no limit at all -- against
+    a real MongoDB instance rather than trusting the convention on faith.
+
+    12 notifications is deliberately more than any other test in this file
+    seeds, so a regression that silently caps at some smaller default (e.g. if
+    `limit=0` were ever misread as "return nothing", or pymongo's own behavior
+    changed) would fail loudly here instead of passing by coincidence.
+    """
+    test_activities.notification_queue_batch_size = 0
+
+    seed_receiver_groups(mongo_uri, E2E_DATABASE, [default_receiver_group()])
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(
+                notification_id=f'n{i}',
+                timestamp=datetime(2024, 6, 1, 12, i, 0, tzinfo=UTC),
+            )
+            for i in range(1, 13)
+        ],
+    )
+
+    input_data = load_scenario_input('subworkflow_load_notification_package.json')
+    _build_metadata(input_data)
+
+    result = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-batch-unlimited'),
+    )
+
+    assert len(result['notification_package']) == 12

orchestrator/activities/activities.py

@@ -84,6 +84,7 @@ class Activities(TemporalManager, SlotManager, Formatters, MongoDB, Email, Postg
             connection_string=mongodb_config['connection_string'],
             database_name=mongodb_config['database_name'],
             ttl_index_seconds=mongodb_config['ttl_index_seconds'],
+            notification_queue_batch_size=mongodb_config['notification_queue_batch_size'],
             logger=logger,
             notification_handler=notification_handler,
             metrics_controller=metrics_controller,

orchestrator/activities/email.py

@@ -177,14 +177,13 @@ class Email(SientiaMonitoring):
             self.logger.error(f'SMTP server disconnected: {e}')
             self.logger.info(f'Reconnecting to {self.smtp_server}:{self.smtp_port}')
 
-            if self.server:
-                try:
-                    self.server.quit()
-                except SMTPServerDisconnected as e:
-                    self.logger.info(f'Server already disconnected: {e}')
-                except Exception as e:
-                    self.logger.error(f'Failed to quit server: {e}')
-                    raise e
+            try:
+                self.server.quit()
+            except SMTPServerDisconnected as e:
+                self.logger.info(f'Server already disconnected: {e}')
+            except Exception as e:
+                self.logger.error(f'Failed to quit server: {e}')
+                raise e
 
             self.server = smtplib.SMTP(self.smtp_server, self.smtp_port, timeout=20)
             if self.sender_password:

orchestrator/activities/formatters.py

@@ -1,6 +1,7 @@
 from temporalio import activity, workflow
 
 with workflow.unsafe.imports_passed_through():
+    import hashlib
     import json
     from collections.abc import Callable
     from logging import Logger
@@ -17,19 +18,95 @@ with workflow.unsafe.imports_passed_through():
         now,
     )
 
-    from orchestrator.utils.orchestrator_functions import (
-        build_tag_config,
+    from orchestrator.utils.opc_slots import build_tag_config, gather_read_tags
+    from orchestrator.utils.pipeline_configs import (
+        chunk_period_warning,
         drift,
-        gather_read_tags,
+        frequency_alert_warning,
         minimal_retrain,
         pi_web_api_scouter,
         predictions_batch,
         scouter,
         simple_metrics,
     )
+    from orchestrator.utils.tag_database import (
+        WRITE_CATEGORIES,
+        WRITE_WORKFLOW_TYPES,
+        index_connector_tags,
+        index_connectors,
+        resolve_tag_names,
+        resolve_write_tags,
+    )
 
 topic_separator = '\n ========== \n'
 
+# Schedule fields left out of config_hash: updated_at is compared on its own
+CONFIG_HASH_EXCLUDED_FIELDS = ('updated_at', 'config_hash')
+
+# Read tags left out of the slots, by gather_read_tags issue kind:
+# (kind, notification_id, message, level)
+SLOT_ISSUES = (
+    (
+        'tags',
+        'ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+        'Tags {} not found among the active, complete Read tags of connector_tags',
+        NotificationLevel.ERROR,
+    ),
+    (
+        'connectors',
+        'ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+        'Connectors {} not found, not active or incomplete in connectors',
+        NotificationLevel.ERROR,
+    ),
+    (
+        'invalid',
+        'ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION',
+        'Read tags {} have no integer tag_id, no integer frequency or an active_read without an '
+        'integer cycle_multiple; or their pipeline lacks schedule_name, model_id or model.name',
+        NotificationLevel.ERROR,
+    ),
+    (
+        'certificates',
+        'ORCHESTRATOR_CERTIFICATE_NOT_FOUND_DURING_SLOT_CONFIGURATION',
+        'Connectors {} have a security.client_certificate_id without an active client '
+        'certificate in opc_certificates: they go into the slot without a certificate, so '
+        'without security',
+        NotificationLevel.WARNING,
+    ),
+)
+
+# Write tags left out of the predictions_batch opc_output_config, by resolve_write_tags
+# issue kind; values are schedule_name/<tag_id or connector_id>
+WRITE_TAG_ISSUES = (
+    (
+        'tags',
+        'ORCHESTRATOR_WRITE_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION',
+        'Write tags {} not found among the active, complete Write and Prediction Confidence '
+        'tags of connector_tags',
+        NotificationLevel.ERROR,
+    ),
+    (
+        'connectors',
+        'ORCHESTRATOR_WRITE_TAG_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION',
+        'Connectors {} of write tags not found, not active or incomplete in connectors',
+        NotificationLevel.ERROR,
+    ),
+    (
+        'invalid',
+        'ORCHESTRATOR_INVALID_WRITE_TAG_DURING_SCHEDULE_CONFIGURATION',
+        'Write tags {} have no integer tag_id or a data_type the OPC Gateway does not accept',
+        NotificationLevel.ERROR,
+    ),
+)
+
+
+def _merge_issues(
+    issues: dict[str, set[str]], pipeline_issues: dict[str, set[str]], schedule_name: str
+) -> None:
+    """Add one pipeline's issues to the run's, each value prefixed with schedule_name/."""
+    for kind, values in pipeline_issues.items():
+        issues.setdefault(kind, set()).update(f'{schedule_name}/{value}' for value in values)
+
 
 class ScheduleType(TypedDict):
     """
@@ -101,6 +178,20 @@ Each entry specifies:
 """
 
 
+def schedule_config_hash(schedule: dict[str, Any]) -> str:
+    """
+    Fingerprint of a schedule as process_schedules builds it, without updated_at.
+
+    Stored in orchestrated_schedules when the schedule is deployed: any change to what
+    reaches Temporal (a pipeline edited without bumping updated_at, a tag renamed, moved,
+    retyped, deactivated or deleted, a connector changed, or a new Orchestrator version
+    building it differently) changes the hash and redeploys the schedule.
+    """
+    content = {k: v for k, v in schedule.items() if k not in CONFIG_HASH_EXCLUDED_FIELDS}
+    serialized = json.dumps(content, sort_keys=True, separators=(',', ':'), default=str)
+    return hashlib.sha256(serialized.encode()).hexdigest()
+
+
 class Formatters(SientiaMonitoring):
     """
     Schedule and slot configuration formatting and notification filtering activity.
@@ -173,6 +264,16 @@ class Formatters(SientiaMonitoring):
             - input_data (dict[str, Any]): The input data containing
               the schedules to process.
                 - pipelines (list[dict[str, Any]]): The schedules to process.
+                - connectors (list[dict[str, Any]], optional): OPC UA connectors of the
+                  tag database, for the write tags.
+                - connector_tags (list[dict[str, Any]], optional): The connector_tags the
+                  pipelines reference, of every category and status.
+
+        Scouter read tags that carry only a tag_id are named from the tag database (the
+        same names as the slot). A predictions_batch gets its opc_output_config from its
+        write_tags ([{tag_id}]) through the tag database; what cannot be resolved is left
+        out and notified once per run. A change to a tag reaches the schedule through
+        create_schedule_config, which compares the resolved schedule (config_hash).
 
         Returns:
             - dict[str, Any]: The schedule configuration dictionary keyed by namespace
@@ -183,6 +284,11 @@ class Formatters(SientiaMonitoring):
         self.info('Processing schedules...', metadata=metadata)
 
         pipelines = input_data['pipelines']
+        connector_tags = input_data.get('connector_tags', [])
+        read_tags_by_id = index_connector_tags(connector_tags)
+        write_tags_by_id = index_connector_tags(connector_tags, WRITE_CATEGORIES)
+        connectors = index_connectors(input_data.get('connectors', []))
+        write_issues: dict[str, set[str]] = {}
 
         schedule_config: dict[str, dict[str, Any]] = {
             self.scouter_namespace: {},
@@ -199,14 +305,31 @@ class Formatters(SientiaMonitoring):
             schedule_type = schedule_types[workflow_type]
             namespace = schedule_type['namespace']
             function = schedule_type['function']
+            schedule_name = pipeline['schedule_name']
 
-            schedule_config[namespace][pipeline['schedule_name']] = {
+            if workflow_type == 'scouter' and 'read_tags' in pipeline:
+                # Same names as the slot's names[] (process_slots): SIENTIAPDE-1974 SC1
+                pipeline = {
+                    **pipeline,
+                    'read_tags': resolve_tag_names(pipeline['read_tags'], read_tags_by_id),
+                }
+
+            if workflow_type in WRITE_WORKFLOW_TYPES:
+                opc_output_config, issues = resolve_write_tags(
+                    pipeline.get('write_tags') or [], write_tags_by_id, connectors
+                )
+                _merge_issues(write_issues, issues, schedule_name)
+                pipeline = {**pipeline, 'opc_output_config': opc_output_config}
+
+            schedule_config[namespace][schedule_name] = {
                 **function(pipeline),
-                'updated_at': pipeline.get(
-                    'updated_at', now().strftime(DATETIME_FORMAT_MS_WITH_TZ)
-                ),
+                # A pipeline without updated_at is redeployed on every run, as before
+                'updated_at': pipeline.get('updated_at')
+                or now().strftime(DATETIME_FORMAT_MS_WITH_TZ),
             }
 
+        self._notify_issues(write_issues, WRITE_TAG_ISSUES, metadata)
+
         self.info('Processed schedules', metadata=metadata)
         self.debug(json.dumps(schedule_config, indent=4, sort_keys=True), metadata=metadata)
 
@@ -216,14 +339,25 @@ class Formatters(SientiaMonitoring):
     def process_slots(self, input_data: dict[str, Any]) -> dict[str, Any]:
         """
         Extracts all read tags from input pipelines, divides them into slots and
-        returns a slot config dictionary. If no ingestor is available, only one slot
-        is created.
+        returns a slot config dictionary in the slot v2 format (contract C1). If no
+        ingestor is available, only one slot is created.
+
+        The division is by address: all the names of an address stay in the same slot
+        (SIENTIAPDE-1974, O5). Read tags that cannot be resolved are left out and
+        notified once per run: a read tag without tag_id or a usable frequency, a tag_id
+        missing from connector_tags, or a connector missing or inactive. A connector whose
+        client_certificate_id has no active client certificate goes into the slot without
+        a certificate, with a warning (D5).
 
         Args:
             - input_data (dict[str, Any]): The input data containing
               the schedules to process.
                 - pipelines (list[dict[str, Any]]): The schedules to process.
-                - opc_servers (list[str]): The OPC servers to create ingestor config.
+                - connectors (list[dict[str, Any]], optional): Active OPC UA connectors of
+                  the tag database, the only source of the OPC servers, each with the
+                  certificates of its client_certificate_id (connectors_query).
+                - connector_tags (list[dict[str, Any]], optional): The connector_tags the
+                  pipelines reference; only the active Read tags are used.
                 - active_ingestors (list[str]): The active ingestors to divide into slots.
 
         Returns:
@@ -235,55 +369,52 @@ class Formatters(SientiaMonitoring):
         self.info('Processing slots...', metadata=metadata)
 
         pipelines = input_data['pipelines']
-        opc_servers_list = input_data['opc_servers']
         active_ingestors = input_data['active_ingestors']
+        connectors = index_connectors(input_data.get('connectors', []))
+        tags_by_id = index_connector_tags(input_data.get('connector_tags', []))
 
-        opc_servers = {}
-        for server in opc_servers_list:
-            opc_servers[server['id']] = {
-                **server,
-            }
+        entries, issues = gather_read_tags(pipelines, connectors, tags_by_id)
+        self._notify_issues(issues, SLOT_ISSUES, metadata)
 
-        tags = list(gather_read_tags(pipelines).values())
-
-        number_of_tags = len(tags)
         number_of_slots = len(active_ingestors) if active_ingestors else 1
-        tags_per_slot = ceil(number_of_tags / number_of_slots)
+        entries_per_slot = ceil(len(entries) / number_of_slots)
 
         slot_config: dict[str, Any] = {}
-        last_index = 0
-
-        for i in range(1, number_of_slots):
-            slot_tags = tags[last_index : last_index + tags_per_slot]
-            slot_config[f'{i}'], notifications = build_tag_config(slot_tags, opc_servers)
-
-            if notifications:
-                self.send_notification(
-                    metadata=metadata,
-                    notification_id='ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION',
-                    message=f'Servers {", ".join(notifications)} not found in opc_servers',
-                    block='orchestrator',
-                    level=NotificationLevel.ERROR,
-                )
-
-            last_index += tags_per_slot
-
-        slot_tags = tags[last_index:]
-        slot_config[f'{number_of_slots}'], notifications = build_tag_config(slot_tags, opc_servers)
-        if notifications:
-            self.send_notification(
-                metadata=metadata,
-                notification_id='ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION',
-                message=f'Servers {", ".join(notifications)} not found in opc_servers',
-                block='orchestrator',
-                level=NotificationLevel.ERROR,
-            )
+        for i in range(number_of_slots):
+            slot_entries = entries[i * entries_per_slot : (i + 1) * entries_per_slot]
+            slot_config[f'{i + 1}'] = build_tag_config(slot_entries)
 
         self.info('Processed slots', metadata=metadata)
         self.debug(json.dumps(slot_config, indent=4, sort_keys=True), metadata=metadata)
 
         return slot_config
 
+    def _notify_issues(
+        self,
+        issues: dict[str, set[str]],
+        kinds: tuple[tuple[str, str, str, str], ...],
+        metadata: dict[str, Any],
+    ) -> None:
+        """
+        Notify what was left out of the slots or schedules, one notification per kind.
+
+        Args:
+            issues (dict[str, set[str]]): What was left out, by kind
+            kinds (tuple[tuple[str, str, str, str], ...]): (kind, notification_id, message
+                with one {} for the joined values, level), e.g. SLOT_ISSUES or
+                WRITE_TAG_ISSUES
+            metadata (dict[str, Any]): Metadata for the notifications
+        """
+        for kind, notification_id, message, level in kinds:
+            if issues.get(kind):
+                self.send_notification(
+                    metadata=metadata,
+                    notification_id=notification_id,
+                    message=message.format(', '.join(sorted(issues[kind]))),
+                    block='orchestrator',
+                    level=level,
+                )
+
     @activity.defn(name='format_schedule_config')
     def format_schedule_config(self, input_data: dict[str, Any]) -> dict[str, Any]:
         """
@@ -291,6 +422,8 @@ class Formatters(SientiaMonitoring):
 
         Transforms a list of schedule configurations into a nested dictionary structure
         organized by namespace and schedule name for efficient lookup and comparison.
+        Each schedule keeps when it was last deployed (updated_at) and the config_hash
+        it was deployed with (None for schedules deployed before config_hash existed).
 
         Args:
             input_data (dict[str, Any]): The input data containing:
@@ -298,7 +431,7 @@ class Formatters(SientiaMonitoring):
                 - metadata (dict): Metadata for logging purposes
 
         Returns:
-            dict[str, Any]: The formatted schedule config organized by namespace and schedule name
+            dict[str, Any]: {namespace: {schedule_name: {'updated_at', 'config_hash'}}}
         """
         metadata = input_data['metadata']
 
@@ -306,16 +439,18 @@ class Formatters(SientiaMonitoring):
 
         schedule_config = input_data['schedule_config']
 
-        config: dict[str, dict[str, str]] = {}
+        config: dict[str, dict[str, dict[str, Any]]] = {}
         for schedule in schedule_config:
             namespace = schedule['namespace']
             schedule_name = schedule['schedule_name']
-            updated_at = schedule['updated_at']
 
             if namespace not in config:
                 config[namespace] = {}
 
-            config[namespace][schedule_name] = updated_at
+            config[namespace][schedule_name] = {
+                'updated_at': schedule['updated_at'],
+                'config_hash': schedule.get('config_hash'),
+            }
 
         self.info('Formatted schedule config', metadata=metadata)
         self.debug(json.dumps(config, indent=4, sort_keys=True), metadata=metadata)
@@ -334,32 +469,42 @@ class Formatters(SientiaMonitoring):
         """
         Compare new and current schedules to determine which should be updated or created.
 
-        Compares schedule timestamps to identify schedules that need updating (newer timestamp)
-        or creating (schedule doesn't exist). Results are accumulated in the provided dictionaries.
+        A deployed schedule is updated when its updated_at is newer than its last deploy
+        (bumping updated_at forces a redeploy) or when its config_hash differs from the one
+        it was deployed with (any change to the resolved schedule, including tag database
+        changes and deleted tags). A schedule not deployed yet is created. Each schedule
+        sent to update or create carries its config_hash, stored once it is deployed.
+        Results are accumulated in the provided dictionaries.
 
         Args:
             schedules (dict[str, Any]): New schedules to compare
-            current_schedules (dict[str, Any]): Existing schedules to compare against
+            current_schedules (dict[str, Any]): Existing schedules to compare against, as
+                format_schedule_config gives them
             to_update (dict[str, Any]): Output accumulator for schedules that need updating
             to_create (dict[str, Any]): Output accumulator for schedules that need creating
             namespace (str): Namespace for the schedules being compared
             metadata (dict[str, Any]): Metadata for logging purposes
         """
         for schedule_name, schedule in schedules.items():
+            config_hash = schedule_config_hash(schedule)
+            deployed = {**schedule, 'config_hash': config_hash}
+
             if schedule_name in current_schedules:
                 update_timestamp = schedule.get('updated_at', now())
 
-                old_timestamp = current_schedules[schedule_name]
+                current = current_schedules[schedule_name]
+                old_timestamp = current['updated_at']
 
                 self.debug(
-                    f'Comparing schedule {schedule_name}:{update_timestamp} vs {old_timestamp}',
+                    f'Comparing schedule {schedule_name}:{update_timestamp} vs {old_timestamp}'
+                    f' (config_hash {config_hash} vs {current.get("config_hash")})',
                     metadata=metadata,
                 )
 
-                if update_timestamp > old_timestamp:
-                    to_update[namespace][schedule_name] = schedule
+                if update_timestamp > old_timestamp or config_hash != current.get('config_hash'):
+                    to_update[namespace][schedule_name] = deployed
             else:
-                to_create[namespace][schedule_name] = schedule
+                to_create[namespace][schedule_name] = deployed
 
     @activity.defn(name='create_schedule_config')
     def create_schedule_config(self, input_data: dict[str, Any]) -> dict[str, Any]:
@@ -368,7 +513,12 @@ class Formatters(SientiaMonitoring):
 
         Compares the current schedule configuration in Temporal with the new schedule
         configuration to determine which schedules need to be created, updated, or deleted.
-        Uses timestamp comparison to identify schedules that have changed.
+        Uses the updated_at timestamps and the config_hash (compare_config_timestamps) to
+        identify schedules that have changed.
+        Also sends a SIMPLE_METRICS_FREQUENCY_ALERT_BELOW_FLOOR WARNING for laborious schedules
+        whose frequency_alert is below 1m (the value is kept) or unparseable (it is ignored),
+        and a DRIFT_CHUNK_PERIOD_INVALID WARNING for drift schedules whose configured
+        chunk_period is not supported by laborious (replaced by 'min').
 
         Args:
             input_data (dict[str, Any]): The input data containing:
@@ -409,6 +559,39 @@ class Formatters(SientiaMonitoring):
                 schedules, current_schedules, to_update, to_create, namespace, metadata
             )
 
+        # Fires whenever the schedule is created or updated (each `updated_at` or
+        # config_hash change, including escalation flips).
+        changed = {**to_create[self.laborious_namespace], **to_update[self.laborious_namespace]}
+        for schedule_name, schedule in changed.items():
+            warning = frequency_alert_warning(schedule.get('frequency_alert'))
+            if warning:
+                self.send_notification(
+                    metadata={
+                        **metadata,
+                        'schedule_name': schedule_name,
+                        'model_id': schedule.get('model_id'),
+                        'model_name': schedule.get('model_name'),
+                    },
+                    notification_id='SIMPLE_METRICS_FREQUENCY_ALERT_BELOW_FLOOR',
+                    message=warning,
+                    block='create_schedule_config',
+                    level=NotificationLevel.WARNING,
+                )
+            warning = chunk_period_warning(schedule.get('chunk_period_configured'))
+            if warning:
+                self.send_notification(
+                    metadata={
+                        **metadata,
+                        'schedule_name': schedule_name,
+                        'model_id': schedule.get('model_id'),
+                        'model_name': schedule.get('model_name'),
+                    },
+                    notification_id='DRIFT_CHUNK_PERIOD_INVALID',
+                    message=warning,
+                    block='create_schedule_config',
+                    level=NotificationLevel.WARNING,
+                )
+
         for namespace, schedules in current_schedule_config.items():
             for schedule_name in schedules:
                 if schedule_name not in schedule_config[namespace]:
@@ -430,15 +613,21 @@ class Formatters(SientiaMonitoring):
         to determine which slots need to be inserted or deleted. Slots are identified
         by numeric IDs, and excess slots are marked for deletion.
 
+        Only slots whose content changed are inserted, so an unchanged slot is not
+        rewritten (nor reported) on every run. gather_read_tags builds the slot in a
+        stable order, so the same pipelines always give the same slot (SIENTIAPDE-1974, O6).
+        A slot in the v1 format differs from its v2 rewrite and is replaced once.
+
         Args:
             input_data (dict[str, Any]): The input data containing:
-                - current_slot_config (dict[str, Any]): The current slot config in Redis
+                - current_slot_config (dict[str, Any]): The current slot config in Redis,
+                  keyed by Redis key (slot:opc_tags:{id}) or by slot id
                 - slot_config (dict[str, Any]): The slot config to process
                 - metadata (dict): Metadata for logging purposes
 
         Returns:
             dict[str, Any]: A dictionary containing:
-                - to_insert (dict): Slots that need to be inserted/updated
+                - to_insert (dict): Slots that are new or changed
                 - to_delete (list[str]): Slot IDs that need to be deleted
         """
 
@@ -456,7 +645,16 @@ class Formatters(SientiaMonitoring):
         if number_of_current_slots > number_of_slots:
             to_delete = [str(i) for i in range(number_of_slots + 1, number_of_current_slots + 1)]
 
-        output = {'to_delete': to_delete, 'to_insert': slot_config}
+        current_by_id = {
+            str(key).rsplit(':', 1)[-1]: config for key, config in current_slot_config.items()
+        }
+        to_insert = {
+            slot: config
+            for slot, config in slot_config.items()
+            if slot not in current_by_id or current_by_id[slot] != config
+        }
+
+        output = {'to_delete': to_delete, 'to_insert': to_insert}
 
         self.info('Created slot config', metadata=metadata)
         self.debug(json.dumps(output, indent=4, sort_keys=True), metadata=metadata)
@@ -824,6 +1022,9 @@ class Formatters(SientiaMonitoring):
                     - group_name (str): Name of the receiver group
                     - contents (list): Content types to include (must contain "reports")
                     - ignore (list, optional): Notification IDs to exclude
+                    - levels (list, optional): Notification levels to include
+                    - ignore_models_id (list, optional): Model IDs to exclude
+                    - model_ids (list, optional): Model IDs to allow (SIENTIAPDE-2149)
 
         Returns:
             dict[str, Any]: Filtered receiver groups with their notifications, keyed by group_name.
@@ -847,6 +1048,9 @@ class Formatters(SientiaMonitoring):
             already_added_keys = []
 
             ignore_list = receiver_group.get('ignore', [])
+            levels_list = receiver_group.get('levels')
+            ignore_models_id_list = receiver_group.get('ignore_models_id') or []
+            model_ids_list = receiver_group.get('model_ids')
 
             for notification in notification_package:
                 alert_type = 'reports'
@@ -859,6 +1063,12 @@ class Formatters(SientiaMonitoring):
                     alert_type in receiver_group['contents']
                     and notification_id not in ignore_list
                     and key not in already_added_keys
+                    and (not levels_list or notification.get('level') in levels_list)
+                    and (
+                        not ignore_models_id_list
+                        or notification.get('model_id') not in ignore_models_id_list
+                    )
+                    and (not model_ids_list or notification.get('model_id') in model_ids_list)
                 ):
                     receiver_groups[group_name]['notifications'].append(notification)
                     already_added_keys.append(key)

orchestrator/activities/mongo_db.py

@@ -31,6 +31,8 @@ class MongoDB(SientiaMonitoring):
         ttl_index_seconds (int): TTL index duration in seconds
         logger (Logger): Application logger instance
         notification_handler (NotificationHandler): Notification management handler
+        notification_queue_batch_size (int): Hard cap on documents fetched per
+            `load_latest_data` run (SIENTIAPDE-2013). `0` means no limit.
     """
 
     def __init__(
@@ -41,6 +43,7 @@ class MongoDB(SientiaMonitoring):
         logger: Logger,
         notification_handler: NotificationHandler,
         metrics_controller: MetricsController,
+        notification_queue_batch_size: int = 0,
     ):
         SientiaMonitoring.__init__(
             self,
@@ -61,6 +64,7 @@ class MongoDB(SientiaMonitoring):
         )
 
         self.ttl_index_seconds = ttl_index_seconds
+        self.notification_queue_batch_size = notification_queue_batch_size
 
         # Initialize MongoDB client here (omitted for brevity)
         logger.info('MongoDB connection initialized')
@@ -189,7 +193,8 @@ class MongoDB(SientiaMonitoring):
 
             for document in aggregated_documents:
                 for timestamp_field in timestamp_fields:
-                    if timestamp_field in document:
+                    # Only BSON dates: a field already stored as text is left as it is
+                    if isinstance(document.get(timestamp_field), datetime):
                         document[timestamp_field] = (
                             document[timestamp_field]
                             .replace(tzinfo=UTC)
@@ -219,12 +224,14 @@ class MongoDB(SientiaMonitoring):
         """
         Update `updated_at` timestamps for successfully updated pipelines.
 
-        Updates the `updated_at` field in the `orchestrated_schedules` collection
-        for all pipelines that were successfully updated in Temporal.
+        Updates the `updated_at` and `config_hash` fields in the `orchestrated_schedules`
+        collection for all pipelines that were successfully updated in Temporal, so the
+        next run compares against what was deployed.
 
         Args:
             input_data (dict[str, Any]): Input data containing:
-                - updated_pipelines (list[dict]): Pipelines with success flags to consider
+                - updated_pipelines (list[dict]): Pipelines with success flags (and
+                  config_hash) to consider
                 - metadata (dict): Metadata for logging purposes
         """
         updated_pipelines = input_data.get('updated_pipelines', [])
@@ -233,29 +240,53 @@ class MongoDB(SientiaMonitoring):
 
         self.info('Updating pipelines timestamps...', metadata=metadata)
 
-        argument = [
-            {'schedule_name': pipeline['schedule_name'], 'namespace': pipeline['namespace']}
-            for pipeline in updated_pipelines
-            if pipeline['success']
-        ]
-        data_filter = {'$or': argument} if argument else {}
+        succeeded = [pipeline for pipeline in updated_pipelines if pipeline['success']]
+        if not succeeded:
+            # Every update failed: stamp nothing so they are retried next tick (same guard as
+            # create_pipelines_timestamps).
+            self.info('No successful pipeline updates to timestamp', metadata=metadata)
+            return
 
-        try:
-            self.mongo_db_repository.update_many(
-                'orchestrated_schedules', data_filter, {'$set': {'updated_at': date_now}}, metadata
-            )
-        except Exception as e:
-            trace = traceback.format_exc()
+        # One update per schedule (each has its own config_hash). A failure does not stop
+        # the others: they are all tried, and the failures are notified together.
+        failures: dict[str, str] = {}
+        traces: list[str] = []
+        for pipeline in succeeded:
+            schedule = f'{pipeline["namespace"]}/{pipeline["schedule_name"]}'
+            try:
+                self.mongo_db_repository.update_one(
+                    'orchestrated_schedules',
+                    {
+                        'schedule_name': pipeline['schedule_name'],
+                        'namespace': pipeline['namespace'],
+                    },
+                    {
+                        '$set': {
+                            'updated_at': date_now,
+                            'config_hash': pipeline.get('config_hash'),
+                        }
+                    },
+                    metadata,
+                )
+            except Exception as e:
+                failures[schedule] = str(e)
+                traces.append(f'{schedule}\n{traceback.format_exc()}')
+
+        if failures:
+            trace = '\n'.join(traces)
+            details = '; '.join(f'{schedule}: {error}' for schedule, error in failures.items())
             self.send_notification(
                 metadata=metadata,
                 notification_id='MONGODB_UPDATE_PIPELINES_ERROR',
-                message=f'Failed to update pipelines timestamps: {e}',
+                message=f'Failed to update pipelines timestamps: {details}',
                 block='update_pipelines_timestamps',
                 level=NotificationLevel.ERROR,
                 attachment_content=trace,
             )
             self.error(trace, metadata=metadata)
-            raise e
+            raise RuntimeError(
+                f'Failed to update {len(failures)} of {len(succeeded)} pipelines timestamps'
+            )
 
         self.info(
             f'Updated {len(updated_pipelines)} pipelines timestamps',
@@ -268,11 +299,13 @@ class MongoDB(SientiaMonitoring):
         Insert `updated_at` timestamps for newly created pipelines.
 
         Inserts new documents into the `orchestrated_schedules` collection
-        for all pipelines that were successfully created in Temporal.
+        for all pipelines that were successfully created in Temporal, with the
+        `config_hash` they were created with.
 
         Args:
             input_data (dict[str, Any]): Input data containing:
-                - created_pipelines (list[dict]): Pipelines with success flags to consider
+                - created_pipelines (list[dict]): Pipelines with success flags (and
+                  config_hash) to consider
                 - metadata (dict): Metadata for logging purposes
         """
         created_pipelines = input_data.get('created_pipelines', [])
@@ -287,6 +320,7 @@ class MongoDB(SientiaMonitoring):
                 'schedule_name': pipeline['schedule_name'],
                 'namespace': pipeline['namespace'],
                 'updated_at': date_now,
+                'config_hash': pipeline.get('config_hash'),
             }
             for pipeline in created_pipelines
             if pipeline['success']
@@ -339,7 +373,13 @@ class MongoDB(SientiaMonitoring):
             for pipeline in deleted_pipelines
             if pipeline['success']
         ]
-        data_filter = {'$or': argument} if argument else {}
+        if not argument:
+            # Every delete failed: remove nothing (same guard as create_pipelines_timestamps).
+            # An empty filter would wipe orchestrated_schedules and make the next tick delete
+            # and recreate every orchestrated schedule.
+            self.info('No successful pipeline deletions to untrack', metadata=metadata)
+            return
+        data_filter = {'$or': argument}
 
         try:
             self.mongo_db_repository.delete_many('orchestrated_schedules', data_filter, metadata)
@@ -447,6 +487,10 @@ class MongoDB(SientiaMonitoring):
         filtering by timestamp to enable incremental data processing. It
         handles connection management and provides comprehensive error reporting.
 
+        The query is capped at ``notification_queue_batch_size`` documents and sorted by
+        ``timestamp`` ascending, so a capped run returns the oldest pending documents first --
+        the caller advances its cursor only to what was actually returned, so nothing is skipped.
+
         Args:
             input_data (dict[str, Any]): Activity input parameters.
                 Required fields:
@@ -488,7 +532,13 @@ class MongoDB(SientiaMonitoring):
 
             self.debug(f'Data filter: {data_filter}', metadata=metadata)
 
-            data = self.mongo_db_repository.find(collection_name, data_filter, metadata)
+            data = self.mongo_db_repository.find(
+                collection_name,
+                data_filter,
+                metadata,
+                limit=self.notification_queue_batch_size,
+                sort=[('timestamp', 1)],
+            )
 
             self.debug(f'Collected: {data}', metadata=metadata)
 

orchestrator/activities/slot_manager.py

@@ -396,6 +396,9 @@ class SlotManager(SientiaMonitoring):
                     - group_name (str): Name of the receiver group
                     - contents (list): Alert types to include (core_alerts, persistent_alerts)
                     - ignore (list, optional): Notification IDs to exclude
+                    - levels (list, optional): Notification levels to include
+                    - ignore_models_id (list, optional): Model IDs to exclude
+                    - model_ids (list, optional): Model IDs to allow (SIENTIAPDE-2149)
                 - notification_ttl (int): Seconds before considering notification persistent
 
         Returns:
@@ -415,48 +418,125 @@ class SlotManager(SientiaMonitoring):
 
         for receiver_group in sending_configs:
             group_name = receiver_group['group_name']
-            receiver_groups[group_name] = {**receiver_group, 'notifications': []}
-            receiver_groups[group_name]['notifications'] = []
-
-            already_added_keys = []
-
-            ignore_list = receiver_group.get('ignore', [])
-
-            for notification in notification_package:
-                alert_type = 'do_nothing'
-                notification_id = notification['notification_id']
-                # Check if notification was recently sent
-                key = f'{notification["trigger"]}:{notification_id}'
-
-                last_sent = self.redis_repository.get(key)
-
-                if last_sent is None:
-                    alert_type = 'core_alerts'
-
-                else:
-                    last_sent = datetime.strptime(last_sent, DATETIME_FORMAT_MS_WITH_TZ)
-
-                    # Check if "notification_ttl" seconds has passed since last sent
-                    if (now() - last_sent) > timedelta(seconds=notification_ttl):
-                        alert_type = 'persistent_alerts'
-
-                # Check if this group must be notified
-                if (
-                    alert_type in receiver_group['contents']
-                    and notification_id not in ignore_list
-                    and key not in already_added_keys
-                ):
-                    receiver_groups[group_name]['notifications'].append(notification)
-                    already_added_keys.append(key)
+            notifications = self._filter_group_notifications(
+                receiver_group, notification_package, notification_ttl
+            )
+            receiver_groups[group_name] = {**receiver_group, 'notifications': notifications}
 
         # Remove groups with no notifications
-        receiver_groups = {
+        return {
             group_name: group
             for group_name, group in receiver_groups.items()
             if group['notifications']
         }
 
-        return receiver_groups
+    def _filter_group_notifications(
+        self,
+        receiver_group: dict[str, Any],
+        notification_package: list[dict[str, Any]],
+        notification_ttl: int,
+    ) -> list[dict[str, Any]]:
+        """
+        Select the notifications this receiver group should get.
+
+        Args:
+            receiver_group (dict[str, Any]): The receiver group configuration.
+            notification_package (list[dict[str, Any]]): Candidate notifications.
+            notification_ttl (int): Seconds before considering a notification persistent.
+
+        Returns:
+            list[dict[str, Any]]: Notifications matching this group's filters.
+        """
+        already_added_keys: list[str] = []
+        ignore_list = receiver_group.get('ignore', [])
+        levels_list = receiver_group.get('levels')
+        ignore_models_id_list = receiver_group.get('ignore_models_id') or []
+        model_ids_list = receiver_group.get('model_ids')
+
+        notifications = []
+        for notification in notification_package:
+            notification_id = notification['notification_id']
+            key = f'{notification["trigger"]}:{notification_id}'
+            alert_type = self._determine_alert_type(key, notification_ttl)
+
+            if (
+                alert_type in receiver_group['contents']
+                and self._check_keys_and_notif_id(
+                    notification_id, ignore_list, key, already_added_keys
+                )
+                and self._check_levels_and_model_filters(
+                    notification, levels_list, ignore_models_id_list, model_ids_list
+                )
+            ):
+                notifications.append(notification)
+                already_added_keys.append(key)
+
+        return notifications
+
+    def _determine_alert_type(self, key: str, notification_ttl: int) -> str:
+        """
+        Classify a notification as new, persistent, or not-yet-due.
+
+        Args:
+            key (str): The Redis cache key for this notification.
+            notification_ttl (int): Seconds before considering a notification persistent.
+
+        Returns:
+            str: 'core_alerts', 'persistent_alerts', or 'do_nothing'.
+        """
+        last_sent = self.redis_repository.get(key)
+        if last_sent is None:
+            return 'core_alerts'
+
+        last_sent = datetime.strptime(last_sent, DATETIME_FORMAT_MS_WITH_TZ)
+        if (now() - last_sent) > timedelta(seconds=notification_ttl):
+            return 'persistent_alerts'
+
+        return 'do_nothing'
+
+    @staticmethod
+    def _check_levels_and_model_filters(
+        notification, levels_list, ignore_models_id_list, model_ids_list
+    ):
+        """
+        Check the notification's level and model_id against a group's filters.
+
+        Args:
+            notification (dict): The notification to check.
+            levels_list (list, optional): List of levels to include. If None, all levels are included.
+            ignore_models_id_list (list, optional): List of model_ids to exclude. If None, no models are excluded.
+            model_ids_list (list, optional): List of model_ids to allow (SIENTIAPDE-2149).
+                If None/empty, all models are allowed. Independent of ignore_models_id_list --
+                both apply together when both are configured.
+
+        Returns:
+            bool: True if the notification passes the checks, False otherwise.
+        """
+        level_check = not levels_list or notification.get('level') in levels_list
+        ignore_models_id_check = (
+            not ignore_models_id_list or notification.get('model_id') not in ignore_models_id_list
+        )
+        model_ids_check = not model_ids_list or notification.get('model_id') in model_ids_list
+        return level_check and ignore_models_id_check and model_ids_check
+
+    @staticmethod
+    def _check_keys_and_notif_id(
+        notification_id, ignore_list, key, already_added_keys: list[str]
+    ) -> bool:
+        """
+        Check if the key has already been added to the list of already added keys.
+
+        Args:
+            notification_id (str): The ID of the notification.
+            ignore_list (list): List of notification IDs to ignore.
+            key (str): The key to check.
+            already_added_keys (list[str]): List of keys that have already been added.
+
+        Returns:
+            bool: True if the key has not been added yet, False otherwise.
+
+        """
+        return notification_id not in ignore_list and key not in already_added_keys
 
     @activity.defn(name='store_notification_cache')
     def store_notification_cache(self, input_data: dict[str, Any]) -> None:

orchestrator/activities/temporal_manager.py

@@ -177,7 +177,9 @@ class TemporalManager(SientiaMonitoring):
                 - schedules (dict[str, Any]): The schedules to create.
 
         Returns:
-            - list[dict[str, Any]]: Report entries for each attempted schedule creation.
+            - list[dict[str, Any]]: Report entries for each attempted schedule creation,
+              with the schedule's config_hash (create_schedule_config) for
+              create_pipelines_timestamps.
         """
 
         schedules_to_create = input_data['schedules']
@@ -198,6 +200,8 @@ class TemporalManager(SientiaMonitoring):
                 )
 
             for schedule_name, schedule in schedules.items():
+                # Stored in orchestrated_schedules, not sent to the workflow
+                config_hash = schedule.pop('config_hash', None)
                 search_attributes = TypedSearchAttributes(
                     [
                         SearchAttributePair(key=self.model_id_id_key, value=schedule['model_id']),
@@ -220,6 +224,9 @@ class TemporalManager(SientiaMonitoring):
                     task_queue_name = self._build_task_queue_name(workflow_type, schedule)
                     schedule['task_queue'] = task_queue_name
 
+                    # No `policy`: schedules rely on Temporal's default SKIP overlap. The
+                    # laborious `DriftAnalysis` cache assumes at most one running drift
+                    # workflow per model; revisit its locking if a policy is ever added here.
                     await client.create_schedule(
                         schedule_name,
                         Schedule(
@@ -255,6 +262,7 @@ class TemporalManager(SientiaMonitoring):
                             'schedule_name': schedule_name,
                             'success': True,
                             'message': 'Schedule created successfully',
+                            'config_hash': config_hash,
                         }
                     )
 
@@ -269,6 +277,7 @@ class TemporalManager(SientiaMonitoring):
                             'schedule_name': schedule_name,
                             'success': False,
                             'message': str(e),
+                            'config_hash': config_hash,
                         }
                     )
 
@@ -357,7 +366,9 @@ class TemporalManager(SientiaMonitoring):
                 - schedules (dict[str, Any]): The schedules to update.
 
         Returns:
-            - list[dict[str, Any]]: Report entries for each attempted schedule update.
+            - list[dict[str, Any]]: Report entries for each attempted schedule update,
+              with the schedule's config_hash (create_schedule_config) for
+              update_pipelines_timestamps.
         """
 
         schedules_to_update = input_data['schedules']
@@ -377,6 +388,8 @@ class TemporalManager(SientiaMonitoring):
                 )
 
             for schedule_name, schedule in schedules.items():
+                # Stored in orchestrated_schedules, not sent to the workflow
+                config_hash = schedule.pop('config_hash', None)
                 try:
                     await self._update_single_schedule(client, schedule_name, schedule, metadata)
 
@@ -386,6 +399,7 @@ class TemporalManager(SientiaMonitoring):
                             'schedule_name': schedule_name,
                             'success': True,
                             'message': 'Schedule updated successfully',
+                            'config_hash': config_hash,
                         }
                     )
 
@@ -400,6 +414,7 @@ class TemporalManager(SientiaMonitoring):
                             'schedule_name': schedule_name,
                             'success': False,
                             'message': str(e),
+                            'config_hash': config_hash,
                         }
                     )
 

orchestrator/utils/connectors_config.py

@@ -21,7 +21,8 @@ def build_mongodb_config():
     Build MongoDB configuration from environment variables.
 
     Returns:
-        dict: MongoDB configuration with connection string, database name, and TTL index seconds.
+        dict: MongoDB configuration with connection string, database name, TTL index seconds,
+            and the notification queue's per-run batch size cap.
     """
     username = getenv('MONGODB_USERNAME', 'root')
     password = getenv('MONGODB_PASSWORD', 'wKZDbMNU1c')
@@ -32,6 +33,7 @@ def build_mongodb_config():
         'connection_string': connection_string,
         'database_name': getenv('MONGODB_DATABASE_NAME', 'sientia'),
         'ttl_index_seconds': int(getenv('MONGODB_TTL_INDEX_HOURS', '1')) * 3600,
+        'notification_queue_batch_size': int(getenv('NOTIFICATION_QUEUE_BATCH_SIZE', '0')),
     }
 
 

orchestrator/utils/opc_slots.py

@@ -0,0 +1,301 @@
+"""OPC slot v2 (contract C1): read tags grouped by server and address for the gateway."""
+
+from typing import Any, cast
+
+from orchestrator.utils.tag_database import (
+    client_certificate_id,
+    is_complete,
+    resolve_certificate,
+    resolve_tag_names,
+    tag_id_of,
+)
+
+
+def _connector_server(connector: dict[str, Any], issues: dict[str, set[str]]) -> dict[str, Any]:
+    """
+    Slot server fields of a tag database connector (connectors collection).
+
+    A connector whose client_certificate_id has no active client certificate still goes
+    into the slot, without a certificate, and is recorded in issues['certificates'] (D5).
+    """
+    certificate = resolve_certificate(connector)
+    if certificate is None and client_certificate_id(connector) is not None:
+        issues['certificates'].add(str(connector['connector_id']))
+    return {
+        'server_id': str(connector['connector_id']),
+        'name': connector['name'],
+        'url': connector['url'],
+        'subscription_period_ms': connector.get('subscription_period_ms'),
+        'security': {
+            'mode': connector['security']['mode'],
+            'policy': connector['security']['policy'],
+        },
+        'certificate': certificate,
+    }
+
+
+def _frequency(tag: dict[str, Any]) -> int | None:
+    """The read tag's frequency as int, or None when it is missing or not an integer."""
+    try:
+        return int(tag['frequency'])
+    except (KeyError, TypeError, ValueError):
+        return None
+
+
+def _locate_read_tag(
+    tag_id: int,
+    connectors: dict[str, dict[str, Any]],
+    tags_by_id: dict[int, dict[str, Any]],
+    issues: dict[str, set[str]],
+) -> tuple[dict[str, Any], str] | None:
+    """
+    Server and address of a read tag, or None when they cannot be resolved.
+
+    The address and connector come from connector_tags and the server from connectors
+    (RF3). What cannot be resolved is recorded in issues ('tags' or 'connectors'), and a
+    server without its registered certificate in issues['certificates'].
+    """
+    connector_tag = tags_by_id.get(tag_id)
+    if connector_tag is None:
+        issues['tags'].add(str(tag_id))
+        return None
+    connector_id = str(connector_tag['connector_id'])
+    connector = connectors.get(connector_id)
+    if connector is None:
+        issues['connectors'].add(connector_id)
+        return None
+    return _connector_server(connector, issues), connector_tag['address']
+
+
+def _cycle_multiple(active_read: Any) -> int | None:
+    """The cycle_multiple of an enabled active_read as int (default 1), or None when invalid."""
+    try:
+        return int(active_read.get('cycle_multiple', 1))
+    except (AttributeError, TypeError, ValueError):
+        return None
+
+
+def _invalid_read_tag(tag: dict[str, Any]) -> bool:
+    """
+    A read tag the slot cannot use: no integer frequency, or an enabled active_read
+    without an integer cycle_multiple.
+    """
+    if _frequency(tag) is None:
+        return True
+    active_read = tag.get('active_read')
+    return (
+        isinstance(active_read, dict)
+        and bool(active_read.get('enabled'))
+        and _cycle_multiple(active_read) is None
+    )
+
+
+def _read_tag_ref(tag: dict[str, Any]) -> Any:
+    """How an invalid read tag is reported: its tag_id, or its tag_name without one."""
+    return tag.get('tag_id') if tag.get('tag_id') not in (None, '') else tag.get('tag_name')
+
+
+def _is_slot_pipeline(pipeline: dict[str, Any]) -> bool:
+    """A scouter pipeline whose read tags can be named in the slot (schedule, model id and name)."""
+    model = pipeline.get('model')
+    return (
+        is_complete(pipeline, ('schedule_name', 'model_id'))
+        and isinstance(model, dict)
+        and is_complete(model, ('name',))
+    )
+
+
+def _merge_active_read(current: dict[str, Any], requested: dict[str, Any] | None) -> dict[str, Any]:
+    """Active read of an address: on if any pipeline asks, with the lowest cycle_multiple (D3)."""
+    if not isinstance(requested, dict) or not requested.get('enabled'):
+        return current
+    cycle_multiple = cast(int, _cycle_multiple(requested))
+    if current.get('enabled'):
+        cycle_multiple = min(cycle_multiple, current['cycle_multiple'])
+    return {'enabled': True, 'cycle_multiple': cycle_multiple}
+
+
+def _assign_server_keys(servers: dict[str, dict[str, Any]]) -> None:
+    """
+    Give each server a unique slot key: its name, or name:server_id (extended until
+    free) when another server already took it. Assigned in server_id order, so the keys
+    are the same on every run and in every slot.
+    """
+    taken: set[str] = set()
+    for server_id in sorted(servers):
+        server = servers[server_id]
+        key = server['name']
+        while key in taken:
+            key = f'{key}:{server_id}'
+        server['key'] = key
+        taken.add(key)
+
+
+def _add_read_tag(
+    entries: dict[tuple[str, str], dict[str, Any]],
+    servers: dict[str, dict[str, Any]],
+    pipeline: dict[str, Any],
+    tag: dict[str, Any],
+    server: dict[str, Any],
+    address: str,
+) -> None:
+    """Merge one resolved read tag into its (server, address) entry and its server."""
+    server_id = server['server_id']
+    servers[server_id] = server
+
+    # int, so the slot does not depend on how each pipeline stored the number
+    frequency = cast(int, _frequency(tag))
+    entry = entries.setdefault(
+        (server_id, address),
+        {
+            'address': address,
+            'frequency': frequency,
+            'active_read': {'enabled': False},
+            'names': [],
+        },
+    )
+    entry['frequency'] = min(entry['frequency'], frequency)
+    entry['active_read'] = _merge_active_read(entry['active_read'], tag.get('active_read'))
+
+    name = {
+        'collection': f'raw_{pipeline["schedule_name"]}',
+        'tag_name': tag['tag_name'],
+        'model_id': pipeline['model_id'],
+        'model_name': pipeline['model']['name'],
+    }
+    if name not in entry['names']:
+        entry['names'].append(name)
+
+
+def _add_pipeline_read_tags(
+    entries: dict[tuple[str, str], dict[str, Any]],
+    servers: dict[str, dict[str, Any]],
+    issues: dict[str, set[str]],
+    pipeline: dict[str, Any],
+    connectors: dict[str, dict[str, Any]],
+    tags_by_id: dict[int, dict[str, Any]],
+) -> None:
+    """Add the read tags of one slot pipeline to entries, reporting the ones left out in issues."""
+    for tag in resolve_tag_names(pipeline.get('read_tags') or [], tags_by_id):
+        tag_id = tag_id_of(tag)
+        if tag_id is None or _invalid_read_tag(tag):
+            issues['invalid'].add(f'{pipeline["schedule_name"]}/{_read_tag_ref(tag)}')
+            continue
+
+        located = _locate_read_tag(tag_id, connectors, tags_by_id, issues)
+        if located is None:
+            continue
+        _add_read_tag(entries, servers, pipeline, tag, *located)
+
+
+def gather_read_tags(
+    pipelines: list[dict[str, Any]],
+    connectors: dict[str, dict[str, Any]],
+    tags_by_id: dict[int, dict[str, Any]],
+) -> tuple[list[dict[str, Any]], dict[str, set[str]]]:
+    """
+    Group the read tags of all scouter pipelines by (server, address) for the slot v2.
+
+    An address read by N pipelines becomes ONE entry with N items in names[]: the
+    gateway reads it once and writes it to N collections (SIENTIAPDE-1974, RF2). Each
+    read tag is resolved through the tag database by its tag_id (RF3): connectors is the
+    only source of the OPC servers.
+
+    Args:
+        pipelines (list[dict[str, Any]]): Pipeline configurations. Only workflow_type
+            'scouter' is read; each needs schedule_name, model_id, model.name and
+            read_tags (with tag_id and frequency, and optionally active_read)
+        connectors (dict[str, dict[str, Any]]): Active OPC UA connectors indexed by
+            index_connectors
+        tags_by_id (dict[int, dict[str, Any]]): Active Read connector_tags indexed by
+            index_connector_tags
+
+    Returns:
+        tuple[list[dict[str, Any]], dict[str, set[str]]]:
+            - One entry per (server, address), sorted by server_id then address, with:
+                - server (dict): server_id, name, url, subscription_period_ms (None =
+                  derive from the frequencies), security ({mode, policy}), certificate
+                  (resolve_certificate; None = connect without security) and key (the
+                  server's unique slot key)
+                - address (str): The OPC node address
+                - frequency (int): The lowest frequency among the pipelines (RF6)
+                - active_read (dict): {'enabled': False} unless a pipeline enables it (D3)
+                - names (list[dict]): collection ('raw_{schedule_name}'), tag_name,
+                  model_id and model_name, one per pipeline name, sorted
+            - What was left out: 'tags' (tag_id), 'connectors' (connector_id) and
+              'invalid' (schedule_name/read tag without an int tag_id, a usable frequency or
+              cycle_multiple, or the schedule_name of a pipeline without schedule_name,
+              model_id or model.name); and 'certificates': connector_ids that went into the
+              slot without the certificate their client_certificate_id points to
+    """
+    issues: dict[str, set[str]] = {
+        'tags': set(),
+        'connectors': set(),
+        'invalid': set(),
+        'certificates': set(),
+    }
+    servers: dict[str, dict[str, Any]] = {}
+    entries: dict[tuple[str, str], dict[str, Any]] = {}
+
+    for pipeline in pipelines:
+        if pipeline.get('workflow_type') != 'scouter':
+            continue
+        if not _is_slot_pipeline(pipeline):
+            issues['invalid'].add(str(pipeline.get('schedule_name')))
+            continue
+        _add_pipeline_read_tags(entries, servers, issues, pipeline, connectors, tags_by_id)
+
+    _assign_server_keys(servers)
+
+    gathered = []
+    for (server_id, _address), entry in sorted(entries.items()):
+        # A stable order keeps the slot identical between runs, so the gateway does
+        # not re-subscribe an unchanged slot (O6)
+        entry['names'].sort(key=lambda n: (n['collection'], n['tag_name'], str(n['model_id'])))
+        gathered.append({'server': servers[server_id], **entry})
+
+    return gathered, issues
+
+
+def build_tag_config(entries: list[dict[str, Any]]) -> dict[str, Any]:
+    """
+    Build one slot in the slot v2 format (contract C1) from address entries. Each server
+    carries its security and the MinIO objects of its client certificate, which the
+    gateway downloads (SIENTIAPDE-1974); certificate None means no security (D5).
+
+    Args:
+        entries (list[dict[str, Any]]): Address entries from gather_read_tags for this slot
+
+    Returns:
+        dict[str, Any]: The slot keyed by server key, each server with server_id, name,
+            url, subscription_period_ms, security, certificate and tags. Each tag (keyed
+            by address) has frequency, active_read and names. A connector without a
+            subscription_period_ms gets half its lowest tag frequency in this slot.
+    """
+    slot_config: dict[str, Any] = {}
+
+    for entry in entries:
+        server = entry['server']
+        if server['key'] not in slot_config:
+            slot_config[server['key']] = {
+                'server_id': server['server_id'],
+                'name': server['name'],
+                'url': server['url'],
+                'subscription_period_ms': server['subscription_period_ms'],
+                'security': server['security'],
+                'certificate': server['certificate'],
+                'tags': {},
+            }
+
+        slot_config[server['key']]['tags'][entry['address']] = {
+            'frequency': entry['frequency'],
+            'active_read': entry['active_read'],
+            'names': entry['names'],
+        }
+
+    for server_config in slot_config.values():
+        if server_config['subscription_period_ms'] is None:
+            frequencies = [tag['frequency'] for tag in server_config['tags'].values()]
+            server_config['subscription_period_ms'] = min(frequencies) / 2
+
+    return slot_config

orchestrator/utils/pipeline_configs.py

@@ -1,3 +1,6 @@
+"""Pipeline configuration builders: one Temporal schedule input per pipeline type."""
+
+from math import ceil
 from typing import Any
 
 from orchestrator.utils.converters import parse_frequency
@@ -43,6 +46,27 @@ def common_config(config: dict[str, Any]):
     }
 
 
+# Kept in sync by hand with the `chunk_period` validation in
+# sientia-dataops-laborious/laborious/activities/model_metrics.py (no shared contract).
+DRIFT_CHUNK_PERIODS = ('min', 's')
+
+
+def _drift_interval_minutes(config: dict[str, Any], frequency: Any) -> int:
+    """
+    Drift window in minutes: `interval_minutes` as configured, otherwise the schedule frequency
+    rounded up to whole minutes (min 1). Consecutive default windows do not overlap, so each
+    drift chunk is stored once.
+    """
+    if 'interval_minutes' in config:
+        return config['interval_minutes']
+    try:
+        seconds = parse_frequency(str(frequency))
+    except ValueError:
+        # create_schedules fails this schedule on its own; don't kill the tick for every model.
+        return 60
+    return max(1, ceil(seconds / 60))
+
+
 def drift(config: dict[str, Any]):
     """
     Build drift configuration from pipeline config.
@@ -52,25 +76,51 @@ def drift(config: dict[str, Any]):
 
     Args:
         config (dict[str, Any]): Pipeline configuration containing:
-            - interval_minutes (int, optional): Time window for data comparison in minutes (default: 60)
+            - interval_minutes (int, optional): Time window for data comparison in minutes,
+                used as-is. Default: `frequency` rounded up to whole minutes (min 1), so
+                consecutive windows do not overlap (1 with the default '1m');
+                `interval_minutes: 60` keeps the legacy window.
+            - chunk_period (str, optional): Drift bucket size, 'min' or 's' (default: 'min').
+                Any other value falls back to 'min' and create_schedule_config warns.
             - drift_metrics (list[str], optional): Statistical metrics to compute (default:
                 ['kolmogorov_smirnov', 'jensen_shannon', 'wasserstein'])
             - Additional fields from common_config
 
     Returns:
         dict[str, Any]: Drift detection configuration with source/target tables,
-            time interval, and metrics specifications. Results stored in 'drift_metrics' table
+            time interval, chunk period and metrics specifications. Results stored in
+            'drift_metrics' table
     """
-    return {
-        **common_config(config),
+    base = common_config(config)
+    configured_chunk_period = config.get('chunk_period')
+    chunk_period_valid = configured_chunk_period in DRIFT_CHUNK_PERIODS
+    output = {
+        **base,
         'schema': 'sientia_data',
         'source_table_name': 'laborious_data',
         'target_table_name': 'drift_metrics',
-        'interval': config.get('interval_minutes', 60),
+        'interval': _drift_interval_minutes(config, base['frequency']),
+        'chunk_period': configured_chunk_period if chunk_period_valid else 'min',
         'drift_metrics': config.get(
             'drift_metrics', ['kolmogorov_smirnov', 'jensen_shannon', 'wasserstein']
         ),
     }
+    if configured_chunk_period is not None and not chunk_period_valid:
+        # Forwarded so create_schedule_config can warn without re-reading the pipeline.
+        output['chunk_period_configured'] = configured_chunk_period
+    return output
+
+
+def chunk_period_warning(chunk_period: Any) -> str | None:
+    """
+    Warning message when a configured drift `chunk_period` is not supported by laborious,
+    otherwise None. Warn only: `drift()` already replaced it with 'min'.
+    """
+    if chunk_period is None or chunk_period in DRIFT_CHUNK_PERIODS:
+        return None
+    return (
+        f"chunk_period={chunk_period!r} is not one of {DRIFT_CHUNK_PERIODS}; falling back to 'min'"
+    )
 
 
 def simple_metrics(config: dict[str, Any]):
@@ -85,6 +135,9 @@ def simple_metrics(config: dict[str, Any]):
             - interval_minutes (int, optional): Computation interval in minutes (default: 60)
             - metrics (list[str], optional): List of metrics to compute
                 (default: ['rmse', 'mse', 'mae', 'r2'])
+            - frequency_alert (str, optional): Frequency used while `adaptive_state.escalated`
+                is true; same format as `frequency`. Absent = adaptive frequency off.
+            - adaptive_state (dict, optional): `{escalated, healthy_streak}`, written by laborious.
             - Additional fields from common_config
 
     Returns:
@@ -92,17 +145,56 @@ def simple_metrics(config: dict[str, Any]):
             actual data), target table, time interval, and metrics list. Results stored in
             'simple_metrics' table
     """
+    base = common_config(config)
+    frequency_alert = config.get('frequency_alert')
+    # SIENTIAPDE-1995: laborious flips `adaptive_state.escalated` on a threshold breach (and back
+    # after the window is renewed without one) and bumps `updated_at`, which makes this tick
+    # rewrite the schedule.
+    state = config.get('adaptive_state')
+    # Hand-written docs: a malformed state must not raise and kill the tick for every model.
+    if frequency_alert and isinstance(state, dict) and state.get('escalated') is True:
+        try:
+            # Temporal parses the schedule frequency on create/update; an unparseable value
+            # would fail it, so keep `frequency` (create_schedule_config still warns).
+            parse_frequency(str(frequency_alert))
+            base['frequency'] = frequency_alert
+        except ValueError:
+            pass
     return {
-        **common_config(config),
+        **base,
         'schema': 'sientia_data',
         'predictions_table_name': 'predictions',
         'data_table_name': 'laborious_data',
         'target_table_name': 'simple_metrics',
         'interval_minutes': config.get('interval_minutes', 60),
         'metrics': config.get('metrics', ['rmse', 'mse', 'mae', 'r2']),
+        # Forwarded so create_schedule_config can warn on a value below the floor.
+        'frequency_alert': frequency_alert,
     }
 
 
+MIN_FREQUENCY_ALERT_SECONDS = 60
+
+
+def frequency_alert_warning(frequency_alert: str | None) -> str | None:
+    """
+    Warning message when `frequency_alert` is below the 1-minute floor or does not parse,
+    otherwise None. Warn only (SIENTIAPDE-1995 decision 7): a value below the floor is kept;
+    an unparseable one is ignored by `simple_metrics()`, which keeps `frequency`.
+    """
+    if not frequency_alert:
+        return None
+    try:
+        # str(): a non-string value (e.g. a number typed into the doc) must warn, not raise
+        # AttributeError and fail the whole create_schedule_config activity.
+        seconds = parse_frequency(str(frequency_alert))
+    except ValueError:
+        return f'frequency_alert={frequency_alert!r} is not a valid frequency'
+    if seconds < MIN_FREQUENCY_ALERT_SECONDS:
+        return f'frequency_alert={frequency_alert} is below the 1m floor; kept as configured'
+    return None
+
+
 def minimal_retrain(config: dict[str, Any]):
     """
     Build minimal retrain configuration from pipeline config.
@@ -174,8 +266,10 @@ def scouter(config: dict[str, Any]):
     Args:
         config (dict[str, Any]): Pipeline configuration containing:
             - schedule_name (str): Name of the schedule (used for topic generation)
-            - read_tags (list[dict]): List of tag configurations with:
-                - tag_name (str): Name of the tag to read
+            - read_tags (list[dict]): List of tag configurations, already passed through
+              resolve_tag_names (process_schedules does it), with:
+                - tag_name (str): Name of the tag to read. A tag without one (a tag_id the
+                  tag database cannot resolve) is left out; process_slots notifies it
                 - aggr_func (str, optional): Aggregation function for data collection (default: 'lts')
                     Common values: 'lts' (last), 'avg' (average), 'min', 'max', 'sum'
                 - data_range (list[int], optional): Valid data range [min, max] (default: [-100, 100])
@@ -188,6 +282,8 @@ def scouter(config: dict[str, Any]):
 
     tags = {}
     for tag in config['read_tags']:
+        if not tag.get('tag_name'):
+            continue
         tags[tag['tag_name']] = {
             'aggr_func': tag.get('aggr_func', 'lts'),
             'data_range': tag.get('data_range', [-100, 100]),
@@ -325,11 +421,10 @@ def predictions_batch(config: dict[str, Any]):
     Args:
         config (dict[str, Any]): Pipeline configuration containing:
             - query (str): SQL query to retrieve input data for predictions
-            - write_tags (list[dict]): List of OPC tag configurations for write-back with:
-                - server_id (str): ID of the target OPC server
-                - type (str): Tag type - 'prediction' (model output) or 'confidence' (prediction confidence)
-                - addr (str): OPC tag address/path
-                - data_type (str, optional): OPC data type (default: 'float')
+            - opc_output_config (dict, optional): The OPC write-back configuration, built by
+              process_schedules from the pipeline's write_tags ([{tag_id}]) through the tag
+              database (resolve_write_tags): {connector_id: {prediction_tags |
+              confidence_tags: {address: {data_type}}}} (default: {})
             - datetime_columns (list[str], optional): Column names to parse as datetime (default: [])
             - path_priority (list[str], optional): Filter policy execution order (default: ["STOP", "CONTINUE", "REPEAT"])
             - input_filters (list[dict], optional): Input data validation filters
@@ -345,23 +440,6 @@ def predictions_batch(config: dict[str, Any]):
         dict[str, Any]: Complete predictions batch configuration with OPC output mappings,
             PI Web API output configuration, multi-stage filters, SQL query, and retention policies
     """
-    tags: dict[str, Any] = {}
-    for tag in config.get('write_tags', []):
-        if tag['server_id'] not in tags:
-            tags[tag['server_id']] = {}
-
-        tag_type = tag['type']
-
-        if tag_type == 'prediction' or tag_type == 'confidence':
-            tag_type_str = f'{tag_type}_tags'
-
-            if tag_type_str not in tags[tag['server_id']]:
-                tags[tag['server_id']][tag_type_str] = {}
-
-            tags[tag['server_id']][tag_type_str][tag['addr']] = {
-                'data_type': tag.get('data_type', 'float'),
-            }
-
     path_priority = process_path_priority(
         config.get('path_priority', ['STOP', 'CONTINUE', 'REPEAT'])
     )
@@ -375,7 +453,7 @@ def predictions_batch(config: dict[str, Any]):
         'save_transform': config.get('save_transform', True),
         'transform_table_name': 'transformed_data',
         'retention_time': config.get('model_retention_minutes', 60) * 60,
-        'opc_output_config': tags,
+        'opc_output_config': config.get('opc_output_config', {}),
         'pi_web_api_output_config': config.get('pi_web_api_output_config', {}),
         'input_filters': overlap_filter_config(
             {'EMPTY_DATA': {'policy': 'STOP', 'config': {}}}, config.get('input_filters', [])
@@ -394,112 +472,3 @@ def predictions_batch(config: dict[str, Any]):
         'path_priority': path_priority,
         'predictions_storage_policy': config.get('predictions_storage_policy', 'lts:1'),
     }
-
-
-def gather_read_tags(pipelines: list[dict[str, Any]]) -> dict[str, Any]:
-    """
-    Gather all read tags from scouter pipeline configurations.
-
-    Collects all read tags from scouter-type pipelines and organizes them by
-    server_id and tag_address, tracking which Kafka topics each tag is associated with.
-    This function is used during slot configuration to aggregate tags across multiple
-    scouter pipelines for efficient OPC server slot allocation.
-
-    Args:
-        pipelines (list[dict[str, Any]]): List of pipeline configurations to process.
-            Only pipelines with workflow_type 'scouter' are processed. Each scouter
-            pipeline should contain a 'read_tags' list with tag configurations.
-
-    Returns:
-        dict[str, Any]: Dictionary of read tags keyed by "server_id:tag_address",
-            where each entry contains:
-            - All original tag configuration fields
-            - topics (list[str]): List of Kafka topic names associated with this tag
-                (format: 'raw_{schedule_name}')
-    """
-
-    tags = {}
-
-    # Get all read tags from pipelines
-    for pipeline in pipelines:
-        if pipeline['workflow_type'] != 'scouter':
-            continue
-
-        for tag in pipeline.get('read_tags', []):
-            tag_string = f'{tag["server_id"]}:{tag["tag_address"]}'
-            if tag_string not in tags:
-                tags[tag_string] = {**tag, 'topics': []}
-
-            tags[tag_string]['topics'].append(f'raw_{pipeline["schedule_name"]}')
-
-    return tags
-
-
-def build_tag_config(
-    tags: list[dict[str, Any]], opc_servers: dict[str, Any]
-) -> tuple[dict[str, Any], list]:
-    """
-    Build tag configuration for a specific slot and OPC server.
-
-    Organizes tags by OPC server name and calculates the minimum subscription period
-    based on tag frequencies. Validates that all server IDs exist in the OPC
-    servers configuration. The subscription period is set to half of the minimum
-    tag frequency to ensure efficient data collection.
-
-    Args:
-        tags (list[dict[str, Any]]): List of tag configurations containing:
-            - server_id (str): ID of the OPC server
-            - tag_address (str): Address/path of the OPC tag
-            - frequency (int): Tag read frequency in milliseconds
-            - Additional tag-specific configuration fields
-        opc_servers (dict[str, Any]): Dictionary of OPC server configurations keyed by server_id.
-            Each server configuration should contain:
-            - server_name (str): Human-readable server name
-            - url (str): OPC server URL
-            - uri (str): OPC server URI
-            - cert_path (str, optional): Certificate file path
-            - private_key_path (str, optional): Private key file path
-            - server_cert_path (str, optional): Server certificate file path
-
-    Returns:
-        tuple[dict[str, Any], list]: A tuple containing:
-            - Slot configuration dictionary organized by server_name, where each server
-              contains connection details, tags dictionary, and subscription_period_ms
-            - List of server IDs (str) that were not found in opc_servers configuration
-    """
-
-    slot_config = {}
-    notifications = []
-
-    for tag in tags:
-        server_id = tag['server_id']
-
-        if server_id not in opc_servers:
-            notifications.append(server_id)
-            continue
-
-        server_name = opc_servers[server_id]['server_name']
-        if server_name not in slot_config:
-            slot_config[server_name] = {
-                'server_id': server_id,
-                'name': server_name,
-                'url': opc_servers[server_id]['url'],
-                'server_uri': opc_servers[server_id]['uri'],
-                'cert_path': opc_servers[server_id].get('cert_path', None),
-                'private_key_path': opc_servers[server_id].get('private_key_path', None),
-                'server_cert_path': opc_servers[server_id].get('server_cert_path', None),
-                'tags': {},
-            }
-
-        slot_config[server_name]['tags'][tag['tag_address']] = {
-            **tag,
-        }
-
-    for server_name in slot_config:
-        frequencies = [int(x['frequency']) for x in slot_config[server_name]['tags'].values()]
-
-        min_frequency = min(frequencies) if frequencies else 1000
-
-        slot_config[server_name]['subscription_period_ms'] = min_frequency / 2
-
-    return slot_config, notifications

orchestrator/utils/tag_database.py

@@ -0,0 +1,310 @@
+"""
+Tag database (connectors + connector_tags): which tags the pipelines reference, the
+indexes of what was loaded, and the resolution of the OPC UA read and write tags.
+"""
+
+from typing import Any
+
+# Fields a tag database document needs to take part in a slot or a write config.
+# Documents missing any of them are left out of the index, so their tags are notified
+# as not found instead of failing the whole run.
+CONNECTOR_TAG_FIELDS = ('tag_id', 'connector_id', 'address', 'tag_name')
+
+
+CONNECTOR_FIELDS = ('connector_id', 'name', 'url')
+
+
+# security fields of an OPC UA connector: the source of its OPC security (D4)
+SECURITY_FIELDS = ('mode', 'policy')
+
+
+# Fields of a client certificate the slot carries, and of each object it names in MinIO
+CERTIFICATE_FIELDS = ('certificate_id', 'application_uri')
+CERTIFICATE_OBJECTS = ('cert', 'private_key')
+STORED_OBJECT_FIELDS = ('bucket', 'object_name')
+
+
+# connector_tags categories each use accepts
+READ_CATEGORIES = ('Read',)
+WRITE_CATEGORIES = ('Write', 'Prediction Confidence')
+
+
+# opc_output_config block each write category goes to
+WRITE_TAG_GROUPS = {'Write': 'prediction_tags', 'Prediction Confidence': 'confidence_tags'}
+
+
+# data_type values the OPC Gateway write engine accepts (contract C2)
+OPC_DATA_TYPES = ('float', 'double', 'real', 'int', 'integer', 'bool', 'boolean', 'str', 'string')
+DEFAULT_DATA_TYPE = 'float'
+
+
+# Workflow types whose write_tags reference the tag database
+WRITE_WORKFLOW_TYPES = ('predictions_batch', 'xgboost_predictions_batch')
+
+
+def is_complete(document: dict[str, Any], fields: tuple[str, ...]) -> bool:
+    return all(document.get(field) not in (None, '') for field in fields)
+
+
+def tag_id_of(tag: Any) -> int | None:
+    """
+    The tag's tag_id, or None when it is not an integer.
+
+    The connector_tags validator stores tag_id as int, so a pipeline references a tag
+    by the same int: text ('7'), bool, null or a missing tag_id is no reference.
+    """
+    tag_id = tag.get('tag_id') if isinstance(tag, dict) else None
+    return tag_id if isinstance(tag_id, int) and not isinstance(tag_id, bool) else None
+
+
+def _is_active(document: dict[str, Any]) -> bool:
+    """A document without status (an override that does not project it) counts as active."""
+    return document.get('status') in (None, 'Active')
+
+
+def pipeline_tag_refs(pipeline: dict[str, Any]) -> list[Any]:
+    """
+    The tag references of a pipeline: the read_tags of a scouter and the write_tags of a
+    predictions_batch. Entries are returned as they are (they may lack a tag_id).
+    """
+    workflow_type = pipeline.get('workflow_type')
+    if workflow_type == 'scouter':
+        return list(pipeline.get('read_tags') or [])
+    if workflow_type in WRITE_WORKFLOW_TYPES:
+        return list(pipeline.get('write_tags') or [])
+    return []
+
+
+def referenced_tag_ids(pipelines: list[dict[str, Any]]) -> list[int]:
+    """
+    The tag_ids the pipelines reference (scouter read_tags and predictions_batch
+    write_tags), for the connector_tags query.
+
+    Sorted, so the query is the same on every run. Runs in the workflow, so it never
+    raises on malformed references.
+
+    Args:
+        pipelines (list[dict[str, Any]]): Pipeline configurations
+
+    Returns:
+        list[int]: The referenced tag_ids (empty when no tag carries a tag_id)
+    """
+    ids = {tag_id_of(tag) for pipeline in pipelines for tag in pipeline_tag_refs(pipeline)}
+    return sorted(tag_id for tag_id in ids if tag_id is not None)
+
+
+def filter_by_tag_ids(query: dict[str, Any], tag_ids: list[int]) -> dict[str, Any]:
+    """
+    Restrict a connector_tags aggregation to the referenced tag_ids.
+
+    Args:
+        query (dict[str, Any]): The connector_tags query ({'collection', 'aggregation'})
+        tag_ids (list[int]): The ids from referenced_tag_ids
+
+    Returns:
+        dict[str, Any]: A new query whose aggregation starts with a tag_id $match. The
+            aggregation list is new too: aggregate_documents_in_mongodb appends a stage
+            to the list it gets.
+    """
+    return {
+        **query,
+        'aggregation': [{'$match': {'tag_id': {'$in': tag_ids}}}, *query['aggregation']],
+    }
+
+
+def index_tag_documents(connector_tags: list[dict[str, Any]]) -> dict[int, dict[str, Any]]:
+    """
+    Every loaded connector_tags document keyed by tag_id, whatever its status, category
+    or completeness.
+    """
+    indexed = {}
+    for tag in connector_tags:
+        tag_id = tag_id_of(tag)
+        if tag_id is not None:
+            indexed[tag_id] = tag
+    return indexed
+
+
+def index_connector_tags(
+    connector_tags: list[dict[str, Any]], categories: tuple[str, ...] = READ_CATEGORIES
+) -> dict[int, dict[str, Any]]:
+    """
+    Index the tag database's usable connector_tags by tag_id.
+
+    Documents that are not active, have another category, or lack an int tag_id,
+    connector_id, address or tag_name are left out.
+
+    Args:
+        connector_tags (list[dict[str, Any]]): connector_tags documents (SIENTIAPDE-1569)
+        categories (tuple[str, ...]): Accepted categories (READ_CATEGORIES for the slot
+            and the scouter names, WRITE_CATEGORIES for the write tags). A document
+            without category is accepted.
+
+    Returns:
+        dict[int, dict[str, Any]]: connector_tags documents keyed by tag_id
+    """
+    return {
+        tag_id: tag
+        for tag_id, tag in index_tag_documents(connector_tags).items()
+        if is_complete(tag, CONNECTOR_TAG_FIELDS)
+        and _is_active(tag)
+        and tag.get('category') in (None, *categories)
+    }
+
+
+def index_connectors(connectors: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
+    """
+    Index the tag database's connectors by connector_id (as text).
+
+    Documents that are not active or lack connector_id, name, url, security.mode or
+    security.policy are left out.
+
+    Args:
+        connectors (list[dict[str, Any]]): connectors documents (SIENTIAPDE-1569)
+
+    Returns:
+        dict[str, dict[str, Any]]: connectors documents keyed by str(connector_id)
+    """
+    return {
+        str(connector['connector_id']): connector
+        for connector in connectors
+        if is_complete(connector, CONNECTOR_FIELDS)
+        and _has_security(connector)
+        and _is_active(connector)
+    }
+
+
+def _has_security(connector: dict[str, Any]) -> bool:
+    security = connector.get('security')
+    return isinstance(security, dict) and is_complete(security, SECURITY_FIELDS)
+
+
+def client_certificate_id(connector: dict[str, Any]) -> Any:
+    """The connector's security.client_certificate_id (None when it has none)."""
+    security = connector.get('security')
+    return security.get('client_certificate_id') if isinstance(security, dict) else None
+
+
+def _is_usable_certificate(certificate: Any) -> bool:
+    return (
+        isinstance(certificate, dict)
+        and is_complete(certificate, CERTIFICATE_FIELDS)
+        and all(
+            isinstance(certificate.get(name), dict)
+            and is_complete(certificate[name], STORED_OBJECT_FIELDS)
+            for name in CERTIFICATE_OBJECTS
+        )
+    )
+
+
+def resolve_certificate(connector: dict[str, Any]) -> dict[str, Any] | None:
+    """
+    The client certificate a connector goes into the slot with, or None (D5).
+
+    connectors_query's $lookup puts in `certificates` the opc_certificates document the
+    connector's security.client_certificate_id points to, only when it is active and of
+    kind client. None means no certificate: the gateway connects without security. A
+    connector with a client_certificate_id and no certificate is most likely a
+    registration error, reported by the caller (see client_certificate_id).
+
+    Args:
+        connector (dict[str, Any]): A connectors document from connectors_query
+
+    Returns:
+        dict[str, Any] | None: certificate_id, application_uri, and cert and private_key
+            as {bucket, object_name}; None without a complete certificate
+    """
+    for certificate in connector.get('certificates') or []:
+        if _is_usable_certificate(certificate):
+            return {
+                **{field: certificate[field] for field in CERTIFICATE_FIELDS},
+                **{
+                    name: {field: certificate[name][field] for field in STORED_OBJECT_FIELDS}
+                    for name in CERTIFICATE_OBJECTS
+                },
+            }
+    return None
+
+
+def resolve_tag_names(
+    read_tags: list[dict[str, Any]], tags_by_id: dict[int, dict[str, Any]]
+) -> list[dict[str, Any]]:
+    """
+    Fill in the tag_name of each read tag of a scouter pipeline.
+
+    The name is the pipeline's own tag_name when it has one, else the connector_tags
+    name of its tag_id. The scouter's model_tags (scouter()) and the slot's names[]
+    (gather_read_tags()) both take the name from here, so the name the gateway writes
+    is always the one the scouter looks for (SIENTIAPDE-1974, SC1).
+
+    Args:
+        read_tags (list[dict[str, Any]]): The pipeline's read_tags
+        tags_by_id (dict[int, dict[str, Any]]): connector_tags indexed by index_connector_tags
+
+    Returns:
+        list[dict[str, Any]]: The read tags, each with tag_name set when it could be
+            resolved. A tag_id missing from tags_by_id leaves the tag without a name.
+    """
+    resolved = []
+    for tag in read_tags:
+        name = tag.get('tag_name')
+        tag_id = tag_id_of(tag)
+        if not name and tag_id is not None:
+            name = tags_by_id.get(tag_id, {}).get('tag_name')
+        resolved.append({**tag, 'tag_name': name} if name else tag)
+    return resolved
+
+
+def resolve_write_tags(
+    write_tags: list[Any],
+    tags_by_id: dict[int, dict[str, Any]],
+    connectors: dict[str, dict[str, Any]],
+) -> tuple[dict[str, Any], dict[str, set[str]]]:
+    """
+    Build the opc_output_config of a predictions_batch from its write_tags.
+
+    Each write tag is only a tag_id: the tag database gives the rest. The category says
+    where the value goes ('Write' → prediction_tags, 'Prediction Confidence' →
+    confidence_tags), the connector_id is the server the OPC Gateway writes to (it
+    resolves it in connectors), and the address and data_type come from connector_tags
+    (data_type defaults to 'float').
+
+    Args:
+        write_tags (list[Any]): The pipeline's write_tags ([{tag_id}])
+        tags_by_id (dict[int, dict[str, Any]]): Write connector_tags indexed by
+            index_connector_tags(..., WRITE_CATEGORIES)
+        connectors (dict[str, dict[str, Any]]): Active OPC UA connectors indexed by
+            index_connectors
+
+    Returns:
+        tuple: the opc_output_config ({connector_id: {prediction_tags | confidence_tags:
+            {address: {data_type}}}}) and what was left out, by kind: 'tags' (tag_id not
+            an active, complete Write / Prediction Confidence tag), 'connectors'
+            (connector_id) and 'invalid' (an entry without an int tag_id, or a tag whose
+            data_type the gateway does not accept)
+    """
+    issues: dict[str, set[str]] = {'tags': set(), 'connectors': set(), 'invalid': set()}
+    output_config: dict[str, Any] = {}
+    for entry in write_tags:
+        tag_id = tag_id_of(entry)
+        if tag_id is None:
+            issues['invalid'].add(str(entry))
+            continue
+        connector_tag = tags_by_id.get(tag_id)
+        # The category is what tells a prediction from a confidence: it cannot be missing
+        if connector_tag is None or connector_tag.get('category') not in WRITE_TAG_GROUPS:
+            issues['tags'].add(str(tag_id))
+            continue
+        connector_id = str(connector_tag['connector_id'])
+        if connector_id not in connectors:
+            issues['connectors'].add(connector_id)
+            continue
+        data_type = connector_tag.get('data_type') or DEFAULT_DATA_TYPE
+        if data_type not in OPC_DATA_TYPES:
+            issues['invalid'].add(str(tag_id))
+            continue
+        group = WRITE_TAG_GROUPS[connector_tag['category']]
+        output_config.setdefault(connector_id, {}).setdefault(group, {})[
+            connector_tag['address']
+        ] = {'data_type': data_type}
+    return output_config, issues

orchestrator/worker/worker.py

@@ -5,6 +5,7 @@ with workflow.unsafe.imports_passed_through():
     import asyncio
     import os
     import sys
+    import traceback
 
     from prometheus_client import start_http_server
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
@@ -192,13 +193,12 @@ async def main():
     try:
         await asyncio.gather(*handlers)
     except BaseException as e:
-        logger.error(f'An unhandled exception occurred: {e}', exc_info=True)
+        # sientia_do's Logger.error takes only a message: the traceback goes in it
+        logger.error(f'An unhandled exception occurred: {e}\n{traceback.format_exc()}')
         exit_code = 1
     finally:
-        if notification_handler:
-            notification_handler.shutdown()
-        if activities:
-            activities.shutdown()
+        notification_handler.shutdown()
+        activities.shutdown()
         metrics.APP_UP.labels(pod_id=POD_ID).set(0)
         sys.exit(exit_code)
 

orchestrator/workflows/orchestrator.py

@@ -7,6 +7,90 @@ with workflow.unsafe.imports_passed_through():
     from sientia_do.temporal.policies import retry_policy
 
     from orchestrator.activities.activities import Activities
+    from orchestrator.utils.tag_database import (
+        filter_by_tag_ids,
+        referenced_tag_ids,
+    )
+
+# MongoDB aggregation stages
+MATCH = '$match'
+PROJECT = '$project'
+LOOKUP = '$lookup'
+
+# Tag database (SIENTIAPDE-1569) read by the slot v2 and the schedules (SIENTIAPDE-1974).
+# The workflow input may override these aggregations (connectors_query /
+# connector_tags_query); the workflow puts a tag_id $match in front of
+# connector_tags_query. Every category and status of the referenced tags is loaded: the
+# slot uses the active Read tags, predictions_batch the active Write / Prediction
+# Confidence ones (status and category are filtered in the code). Only the fields used
+# are projected: other attributes may hold BSON types a Temporal payload cannot carry
+# (so the certificate dates are left out).
+# Each connector brings, in `certificates`, the opc_certificates document its
+# security.client_certificate_id points to, only when it is active and of kind client
+# (D5); resolve_certificate turns it into the slot's certificate.
+DEFAULT_CONNECTORS_QUERY = {
+    'collection': 'connectors',
+    'aggregation': [
+        {MATCH: {'type': 'opc_ua', 'status': 'Active'}},
+        {
+            LOOKUP: {
+                'from': 'opc_certificates',
+                'let': {'certificate_id': '$security.client_certificate_id'},
+                'pipeline': [
+                    {
+                        MATCH: {
+                            '$expr': {'$eq': ['$certificate_id', '$$certificate_id']},
+                            'status': 'active',
+                            'kind': 'client',
+                        }
+                    },
+                    {
+                        PROJECT: {
+                            '_id': 0,
+                            'certificate_id': 1,
+                            'application_uri': 1,
+                            'cert.bucket': 1,
+                            'cert.object_name': 1,
+                            'private_key.bucket': 1,
+                            'private_key.object_name': 1,
+                        }
+                    },
+                ],
+                'as': 'certificates',
+            }
+        },
+        {
+            PROJECT: {
+                '_id': 0,
+                'connector_id': 1,
+                'name': 1,
+                'url': 1,
+                'subscription_period_ms': 1,
+                'security.mode': 1,
+                'security.policy': 1,
+                'security.client_certificate_id': 1,
+                'certificates': 1,
+            }
+        },
+    ],
+}
+DEFAULT_CONNECTOR_TAGS_QUERY = {
+    'collection': 'connector_tags',
+    'aggregation': [
+        {
+            PROJECT: {
+                '_id': 0,
+                'tag_id': 1,
+                'connector_id': 1,
+                'address': 1,
+                'tag_name': 1,
+                'category': 1,
+                'data_type': 1,
+                'status': 1,
+            }
+        },
+    ],
+}
 
 
 @workflow.defn(name='orchestrator')
@@ -20,14 +104,79 @@ class Orchestrator:
     and Redis infrastructure.
     """
 
+    async def _load_tag_database(
+        self,
+        input_data: dict[str, Any],
+        metadata: dict[str, Any],
+        pipelines: list[dict[str, Any]],
+    ) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+        """
+        Load the tag database (SIENTIAPDE-1974): the only source of the OPC servers and
+        tags of the read and write tags.
+
+        Only the referenced tags are loaded, with the fields used, so the payload stays
+        bounded by the pipelines. Nothing is loaded when no scouter read tag nor
+        predictions_batch write tag has a tag_id.
+
+        Args:
+            input_data (dict[str, Any]): Workflow input (optional connectors_query and
+                connector_tags_query)
+            metadata (dict[str, Any]): Activity metadata ({'metadata': {...}})
+            pipelines (list[dict[str, Any]]): The loaded pipeline configurations
+
+        Returns:
+            tuple[list[dict[str, Any]], list[dict[str, Any]]]: connectors and connector_tags
+        """
+        tag_ids = referenced_tag_ids(pipelines)
+        if not tag_ids:
+            return [], []
+
+        return await self._load_connectors(
+            metadata,
+            input_data.get('connectors_query', DEFAULT_CONNECTORS_QUERY),
+            input_data.get('connector_tags_query', DEFAULT_CONNECTOR_TAGS_QUERY),
+            tag_ids,
+        )
+
+    async def _load_connectors(
+        self,
+        metadata: dict[str, Any],
+        connectors_query: dict[str, Any],
+        connector_tags_query: dict[str, Any],
+        tag_ids: list[Any],
+    ) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+        """Run a connectors query and a connector_tags query restricted to tag_ids, in parallel."""
+        connectors_handler = workflow.start_local_activity_method(
+            Activities.aggregate_documents_in_mongodb,
+            {
+                **metadata,
+                # A fresh aggregation list: the activity appends a stage to the one it gets
+                'query': {**connectors_query, 'aggregation': list(connectors_query['aggregation'])},
+            },
+            retry_policy=retry_policy,
+            start_to_close_timeout=timedelta(seconds=60),
+        )
+
+        connector_tags_handler = workflow.start_local_activity_method(
+            Activities.aggregate_documents_in_mongodb,
+            {
+                **metadata,
+                'query': filter_by_tag_ids(connector_tags_query, tag_ids),
+            },
+            retry_policy=retry_policy,
+            start_to_close_timeout=timedelta(seconds=60),
+        )
+
+        return await connectors_handler, await connector_tags_handler
+
     @workflow.run
     async def run(self, input_data: dict[str, Any]):
         """
         Execute the orchestration workflow for pipeline and slot management.
 
-        This workflow retrieves pipeline configurations and OPC server data,
-        processes schedules and slot configurations, and deploys them to
-        the appropriate services. It handles creation, updates, and deletion
+        This workflow retrieves pipeline configurations and the tag database
+        (connectors and connector_tags), processes schedules and slot configurations,
+        and deploys them to the appropriate services. It handles creation, updates, and deletion
         of schedules and slots based on current system state.
 
         Args:
@@ -35,7 +184,11 @@ class Orchestrator:
                 Required fields:
                 - schedule_name (str): Name of the orchestration schedule
                 - pipelines_query (dict[str, Any]): MongoDB query for pipeline configurations
-                - opc_servers_query (dict[str, Any]): MongoDB query for OPC server data
+                Optional fields:
+                - connectors_query (dict[str, Any]): MongoDB aggregation for the tag
+                  database connectors (default: DEFAULT_CONNECTORS_QUERY)
+                - connector_tags_query (dict[str, Any]): MongoDB aggregation for the tag
+                  database tags (default: DEFAULT_CONNECTOR_TAGS_QUERY)
 
         Returns:
             None: Workflow completes without return value
@@ -66,13 +219,6 @@ class Orchestrator:
             start_to_close_timeout=timedelta(seconds=60),
         )
 
-        opc_servers_handler = workflow.start_local_activity_method(
-            Activities.find_documents_in_mongodb,
-            {**metadata, 'query': input_data['opc_servers_query']},
-            retry_policy=retry_policy,
-            start_to_close_timeout=timedelta(seconds=60),
-        )
-
         orchestrated_schedules_handler = workflow.start_local_activity_method(
             Activities.find_documents_in_mongodb,
             {
@@ -103,9 +249,13 @@ class Orchestrator:
         )
 
         pipeline_config = await pipeline_config_handler
+
+        connectors, connector_tags = await self._load_tag_database(
+            input_data, metadata, pipeline_config
+        )
+
         orchestrated_schedules = await orchestrated_schedules_handler
         current_slot_config = await current_slot_config_handler
-        opc_servers = await opc_servers_handler
         active_ingestors = await active_ingestors_handler
 
         formatted_orchestrated_schedules_handler = workflow.start_local_activity_method(
@@ -117,7 +267,12 @@ class Orchestrator:
 
         schedules_config_handler = workflow.start_local_activity_method(
             Activities.process_schedules,
-            {**metadata, 'pipelines': pipeline_config},
+            {
+                **metadata,
+                'pipelines': pipeline_config,
+                'connectors': connectors,
+                'connector_tags': connector_tags,
+            },
             retry_policy=retry_policy,
             start_to_close_timeout=timedelta(seconds=60),
         )
@@ -126,7 +281,8 @@ class Orchestrator:
             Activities.process_slots,
             {
                 **metadata,
-                'opc_servers': opc_servers,
+                'connectors': connectors,
+                'connector_tags': connector_tags,
                 'active_ingestors': active_ingestors,
                 'pipelines': pipeline_config,
             },

orchestrator/workflows/subworkflows/load_notification_package.py

@@ -137,18 +137,21 @@ class LoadNotificationPackage:
           is only safe to drop from the fetch if EVERY active group ignores it.
           A group with no `ignore_models_id` is treated as "ignores nothing" (empty
           set), which naturally zeroes out the intersection via set math.
+        - model_ids (SIENTIAPDE-2149): union of every group's `model_ids` allow-list,
+          mirroring `levels`' own union-or-nothing logic. If ANY active group has no
+          restriction (missing/empty `model_ids`), that group wants every model, so
+          the query can't restrict to an allow-list at all. `model_ids` and
+          `ignore_models_id` are independent axes and can both narrow the same
+          `model_id` field at once (Mongo supports multiple operators on one field).
         """
         if not sending_configs:
             return default_filter
 
-        sending_configs_levels = [group.get('levels') for group in sending_configs]
-
-        if all(sending_configs_levels):
-            levels_union: set[str] = set()
-            for levels in sending_configs_levels:
-                if levels:
-                    levels_union.update(levels)
-            query_filter = {**default_filter, 'level': {'$in': sorted(levels_union)}}
+        levels_union = self._union_when_all_restrict(
+            [group.get('levels') for group in sending_configs]
+        )
+        if levels_union is not None:
+            query_filter = {**default_filter, 'level': {'$in': levels_union}}
         else:
             query_filter = dict(default_filter)
 
@@ -160,7 +163,40 @@ class LoadNotificationPackage:
         ]
         models_intersection = set.intersection(*sending_configs_ignore_models_id)
 
+        model_ids_union = self._union_when_all_restrict(
+            [group.get('model_ids') for group in sending_configs]
+        )
+
+        model_id_filter: dict[str, Any] = {}
+        if model_ids_union is not None:
+            model_id_filter['$in'] = model_ids_union
         if models_intersection:
-            query_filter['model_id'] = {'$nin': sorted(models_intersection)}
+            model_id_filter['$nin'] = sorted(models_intersection)
+        if model_id_filter:
+            query_filter['model_id'] = model_id_filter
 
         return query_filter
+
+    @staticmethod
+    def _union_when_all_restrict(per_group_values: list[list[str] | None]) -> list[str] | None:
+        """
+        Union a list of per-group optional restrictions, but only when every group has one.
+
+        Shared by `levels` and `model_ids` (SIENTIAPDE-2149) in `_compute_query_time_filter`,
+        which both follow the same rule: if ANY active group is unrestricted on that axis
+        (missing/empty value), that group wants everything, so the union can't restrict the
+        query either — the caller falls back to no filter on that axis in that case (`None`).
+
+        Args:
+            per_group_values: Each active group's raw value for the axis being unioned
+                (e.g. `group.get('levels')` for every group).
+
+        Returns:
+            Sorted union of every group's values, or None if any group was unrestricted.
+        """
+        if not all(per_group_values):
+            return None
+        union: set[str] = set()
+        for values in per_group_values:
+            union.update(values or [])
+        return sorted(union)

pipelines_sample.json

@@ -16,25 +16,22 @@
   ],
   "read_tags": [
     {
-      "server_id": "1",
+      "tag_id": 1,
       "tag_name": "Counter",
-      "tag_address": "ns=2;s=GatewayProSYS.ProSYS.Simulation.Counter",
       "aggr_func": "lts",
       "data_range": [1, 30],
       "frequency": 30000
     },
     {
-      "server_id": "1",
+      "tag_id": 2,
       "tag_name": "Square",
-      "tag_address": "ns=2;s=GatewayProSYS.ProSYS.Simulation.Square",
       "aggr_func": "lts",
       "data_range": [-2, 2],
       "frequency": 30000
     },
     {
-      "server_id": "1",
+      "tag_id": 3,
       "tag_name": "Rollout",
-      "tag_address": "ns=2;s=GatewayProSYS.ProSYS.Simulation.Rollout",
       "aggr_func": "lts",
       "data_range": [-2, 2],
       "frequency": 30000

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "orchestrator"
-version = "0.6.4"
+version = "0.8.2"
 description = "Sientia DataOps Orchestrator - ML Model Orchestration System"
 readme = "README.md"
 requires-python = ">=3.11"

requirements-local.txt

@@ -4,5 +4,5 @@ sqlalchemy
 redis
 pymongo
 jinja2
-git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.12.2
+git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.12.5
 prometheus-client

requirements.txt

@@ -4,5 +4,5 @@ sqlalchemy
 redis
 pymongo
 jinja2
-sientia_do>=1.12.2
+sientia_do>=1.12.5
 prometheus-client

tests/orchestrator/activities/test_activities.py

@@ -27,6 +27,7 @@ def test___init__(
         'connection_string': 'mongodb://localhost:27017',
         'database_name': 'test_db',
         'ttl_index_seconds': 3600,
+        'notification_queue_batch_size': 5000,
     }
 
     redis_config = {'host': 'localhost', 'port': 6379, 'username': 'admin', 'password': 'password'}
@@ -89,6 +90,7 @@ def test___init__(
         connection_string='mongodb://localhost:27017',
         database_name='test_db',
         ttl_index_seconds=3600,
+        notification_queue_batch_size=5000,
         logger=logger,
         notification_handler=notification_handler,
         metrics_controller=mock_metrics_controller.return_value,

tests/orchestrator/activities/test_email.py

@@ -85,7 +85,9 @@ def test_close(sientia_monitoring_mock, email):
     email.close()
 
     email.server.quit.assert_called_once()
-    sientia_monitoring_mock.shutdown.assert_called_once()
+    # Count only this instance: an Email left by another test may be garbage-collected
+    # (Email.__del__ -> close) while the patch is active
+    assert sientia_monitoring_mock.shutdown.call_args_list.count(call(email)) == 1
 
 
 def test_build_email_html(email):
@@ -324,3 +326,37 @@ def test_send_email(mimemultipart, mimetext, email):
     )
 
     assert email.try_send_email.call_count == 2
+
+
+@patch('orchestrator.activities.email.EmailBuilder')
+@patch('orchestrator.activities.email.smtplib')
+def test___init___without_smtp_server(smtplib, email_builder):
+    email = Email(
+        sender_email='test@test.com',
+        sender_password='test',
+        smtp_server=None,
+        smtp_port=587,
+        logger=MagicMock(),
+        notification_handler=MagicMock(),
+        metrics_controller=AsyncMock(),
+    )
+
+    smtplib.SMTP.assert_not_called()
+    assert not hasattr(email, 'server')
+
+
+@patch('orchestrator.activities.email.smtplib.SMTP')
+def test_try_send_email_reconnect_without_password(smtp, email):
+    email.sender_password = None
+    email.server.sendmail = MagicMock(side_effect=SMTPServerDisconnected('test'))
+    email.server.quit = MagicMock()
+
+    msg = MagicMock()
+
+    email.try_send_email(msg, 'test')
+
+    smtp.return_value.starttls.assert_not_called()
+    smtp.return_value.login.assert_not_called()
+    smtp.return_value.sendmail.assert_called_once_with(
+        'test@test.com', 'test', msg.as_string.return_value
+    )

tests/orchestrator/activities/test_mongo_db.py

@@ -1,14 +1,14 @@
 from datetime import UTC, datetime
-from unittest.mock import ANY, AsyncMock, MagicMock, patch
+from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
 
-from pytest import fixture
+import pytest
 from sientia_do.notifications.models import NotificationLevel
 from sientia_do.temporal.constants import DATETIME_FORMAT_MS_WITH_TZ
 
 from orchestrator.activities.mongo_db import MongoDB
 
 
-@fixture
+@pytest.fixture
 @patch('orchestrator.activities.mongo_db.MongoDBRepository')
 def mongo_db(mongo_mock):
     mongo = MongoDB(
@@ -31,13 +31,14 @@ def test___init__(mongo_mock):
     notification_handler = MagicMock()
     metrics_controller = AsyncMock()
 
-    MongoDB(
+    mongo = MongoDB(
         connection_string='mongodb://localhost:27017',
         database_name='test_db',
         ttl_index_seconds=3600,
         logger=logger,
         notification_handler=notification_handler,
         metrics_controller=metrics_controller,
+        notification_queue_batch_size=5000,
     )
 
     mongo_mock.assert_called_once_with(
@@ -47,6 +48,22 @@ def test___init__(mongo_mock):
         notification_handler=notification_handler,
         metrics_controller=metrics_controller,
     )
+    assert mongo.notification_queue_batch_size == 5000
+
+
+@patch('orchestrator.activities.mongo_db.MongoDBRepository')
+def test___init___notification_queue_batch_size_defaults_to_zero(mongo_mock):
+    """0 means no limit, matching the underlying find()'s own convention."""
+    mongo = MongoDB(
+        connection_string='mongodb://localhost:27017',
+        database_name='test_db',
+        ttl_index_seconds=3600,
+        logger=MagicMock(),
+        notification_handler=MagicMock(),
+        metrics_controller=AsyncMock(),
+    )
+
+    assert mongo.notification_queue_batch_size == 0
 
 
 @patch('orchestrator.activities.mongo_db.SientiaMonitoring')
@@ -175,6 +192,30 @@ def test_aggregate_documents_in_mongodb_success(mongo_db):
     )
 
 
+def test_aggregate_documents_in_mongodb_formats_only_datetimes(mongo_db):
+    """A timestamp field stored as text, or missing, is left as it is"""
+    mongo_db.mongo_db_repository.aggregate = MagicMock(
+        return_value=[
+            {'tag_id': 1, 'updated_at': datetime(2026, 10, 1, 12, 0, 0)},
+            {'tag_id': 2, 'updated_at': '2026-10-02 00:00:00.000+0000'},
+            {'tag_id': 3},
+        ]
+    )
+
+    result = mongo_db.aggregate_documents_in_mongodb(
+        {
+            'query': {'collection': 'connector_tags', 'aggregation': [{'$match': {}}]},
+            'timestamp_fields': ['updated_at'],
+        }
+    )
+
+    assert result == [
+        {'tag_id': 1, 'updated_at': '2026-10-01 12:00:00.000000+0000'},
+        {'tag_id': 2, 'updated_at': '2026-10-02 00:00:00.000+0000'},
+        {'tag_id': 3},
+    ]
+
+
 def test_aggregate_documents_in_mongodb_failure(mongo_db):
     input_data = {
         'collection': 'test_collection',
@@ -229,21 +270,59 @@ def test_aggregate_documents_in_mongodb_missing_aggregation(mongo_db):
 def test_update_pipelines_timestamps_success(now_mock, mongo_db):
     input_data = {
         'updated_pipelines': [
-            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
+            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True, 'config_hash': 'h1'},
+            # Updated by a run started before config_hash
             {'schedule_name': 'test2', 'namespace': 'test2', 'success': True},
         ]
     }
-    mongo_db.mongo_db_repository.update_many = MagicMock(return_value=MagicMock())
+    mongo_db.mongo_db_repository.update_one = MagicMock(return_value=1)
     mongo_db.update_pipelines_timestamps(input_data)
-    mongo_db.mongo_db_repository.update_many.assert_called_once_with(
+    assert mongo_db.mongo_db_repository.update_one.call_args_list == [
+        call(
+            'orchestrated_schedules',
+            {'schedule_name': 'test1', 'namespace': 'test1'},
+            {'$set': {'updated_at': now_mock.return_value, 'config_hash': 'h1'}},
+            {},
+        ),
+        call(
+            'orchestrated_schedules',
+            {'schedule_name': 'test2', 'namespace': 'test2'},
+            {'$set': {'updated_at': now_mock.return_value, 'config_hash': None}},
+            {},
+        ),
+    ]
+
+
+@patch('orchestrator.activities.mongo_db.now')
+@pytest.mark.parametrize(
+    'pipelines', [[], [{'schedule_name': 'test1', 'namespace': 'test1', 'success': False}]]
+)
+def test_update_pipelines_timestamps_all_failed_stamps_nothing(now_mock, mongo_db, pipelines):
+    input_data = {'updated_pipelines': pipelines}
+    mongo_db.mongo_db_repository.update_one = MagicMock()
+
+    mongo_db.update_pipelines_timestamps(input_data)
+
+    # A failed update is not stamped, so it is retried on the next tick
+    mongo_db.mongo_db_repository.update_one.assert_not_called()
+
+
+@patch('orchestrator.activities.mongo_db.now')
+def test_update_pipelines_timestamps_mixed_stamps_only_successes(now_mock, mongo_db):
+    input_data = {
+        'updated_pipelines': [
+            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True, 'config_hash': 'h1'},
+            {'schedule_name': 'test2', 'namespace': 'test2', 'success': False, 'config_hash': 'h2'},
+        ]
+    }
+    mongo_db.mongo_db_repository.update_one = MagicMock()
+
+    mongo_db.update_pipelines_timestamps(input_data)
+
+    mongo_db.mongo_db_repository.update_one.assert_called_once_with(
         'orchestrated_schedules',
-        {
-            '$or': [
-                {'schedule_name': 'test1', 'namespace': 'test1'},
-                {'schedule_name': 'test2', 'namespace': 'test2'},
-            ]
-        },
-        {'$set': {'updated_at': now_mock.return_value}},
+        {'schedule_name': 'test1', 'namespace': 'test1'},
+        {'$set': {'updated_at': now_mock.return_value, 'config_hash': 'h1'}},
         {},
     )
 
@@ -258,15 +337,17 @@ def test_update_pipelines_timestamps_failure(now_mock, mongo_db):
         **metadata,
     }
 
-    mongo_db.mongo_db_repository.update_many.side_effect = Exception('Error')
+    # The first update fails: the second is still tried, and only the failure is notified
+    mongo_db.mongo_db_repository.update_one.side_effect = [Exception('Error'), 1]
     try:
         mongo_db.update_pipelines_timestamps(input_data)
-    except Exception as e:
-        assert str(e) == 'Error'
+    except RuntimeError as e:
+        assert str(e) == 'Failed to update 1 of 2 pipelines timestamps'
+        assert mongo_db.mongo_db_repository.update_one.call_count == 2
         mongo_db.send_notification.assert_called_once_with(
             metadata=metadata['metadata'],
             notification_id='MONGODB_UPDATE_PIPELINES_ERROR',
-            message='Failed to update pipelines timestamps: Error',
+            message='Failed to update pipelines timestamps: test1/test1: Error',
             block='update_pipelines_timestamps',
             level=NotificationLevel.ERROR,
             attachment_content=ANY,
@@ -276,11 +357,37 @@ def test_update_pipelines_timestamps_failure(now_mock, mongo_db):
         raise AssertionError('Expected an exception to be raised')
 
 
+@patch('orchestrator.activities.mongo_db.now')
+def test_update_pipelines_timestamps_failures_notified_together(now_mock, mongo_db):
+    input_data = {
+        'updated_pipelines': [
+            {'schedule_name': 'test1', 'namespace': 'ns', 'success': True},
+            {'schedule_name': 'test2', 'namespace': 'ns', 'success': True},
+            {'schedule_name': 'test3', 'namespace': 'ns', 'success': True},
+        ],
+        **metadata,
+    }
+    mongo_db.mongo_db_repository.update_one.side_effect = [
+        Exception('down'),
+        1,
+        Exception('timeout'),
+    ]
+
+    with pytest.raises(RuntimeError, match='Failed to update 2 of 3 pipelines timestamps'):
+        mongo_db.update_pipelines_timestamps(input_data)
+
+    assert mongo_db.mongo_db_repository.update_one.call_count == 3
+    mongo_db.send_notification.assert_called_once()
+    assert mongo_db.send_notification.call_args.kwargs['message'] == (
+        'Failed to update pipelines timestamps: ns/test1: down; ns/test3: timeout'
+    )
+
+
 @patch('orchestrator.activities.mongo_db.now')
 def test_create_pipelines_timestamps_success(now_mock, mongo_db):
     input_data = {
         'created_pipelines': [
-            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
+            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True, 'config_hash': 'h1'},
             {'schedule_name': 'test2', 'namespace': 'test2', 'success': True},
         ]
     }
@@ -289,8 +396,18 @@ def test_create_pipelines_timestamps_success(now_mock, mongo_db):
     mongo_db.mongo_db_repository.insert_many.assert_called_once_with(
         'orchestrated_schedules',
         [
-            {'schedule_name': 'test1', 'namespace': 'test1', 'updated_at': now_mock.return_value},
-            {'schedule_name': 'test2', 'namespace': 'test2', 'updated_at': now_mock.return_value},
+            {
+                'schedule_name': 'test1',
+                'namespace': 'test1',
+                'updated_at': now_mock.return_value,
+                'config_hash': 'h1',
+            },
+            {
+                'schedule_name': 'test2',
+                'namespace': 'test2',
+                'updated_at': now_mock.return_value,
+                'config_hash': None,
+            },
         ],
         {},
     )
@@ -345,6 +462,38 @@ def test_delete_pipelines_timestamps_success(now_mock, mongo_db):
     )
 
 
+@pytest.mark.parametrize(
+    'pipelines', [[], [{'schedule_name': 'test1', 'namespace': 'test1', 'success': False}]]
+)
+def test_delete_pipelines_timestamps_all_failed_deletes_nothing(mongo_db, pipelines):
+    input_data = {'deleted_pipelines': pipelines}
+    mongo_db.mongo_db_repository.delete_many = MagicMock()
+
+    mongo_db.delete_pipelines_timestamps(input_data)
+
+    # An empty filter would wipe orchestrated_schedules; the next tick's normalize_schedules
+    # would then delete and recreate every orchestrated Temporal schedule.
+    mongo_db.mongo_db_repository.delete_many.assert_not_called()
+
+
+def test_delete_pipelines_timestamps_mixed_deletes_only_successes(mongo_db):
+    input_data = {
+        'deleted_pipelines': [
+            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
+            {'schedule_name': 'test2', 'namespace': 'test2', 'success': False},
+        ]
+    }
+    mongo_db.mongo_db_repository.delete_many = MagicMock()
+
+    mongo_db.delete_pipelines_timestamps(input_data)
+
+    mongo_db.mongo_db_repository.delete_many.assert_called_once_with(
+        'orchestrated_schedules',
+        {'$or': [{'schedule_name': 'test1', 'namespace': 'test1'}]},
+        {},
+    )
+
+
 @patch('orchestrator.activities.mongo_db.now')
 def test_delete_pipelines_timestamps_failure(now_mock, mongo_db):
     input_data = {
@@ -480,6 +629,8 @@ def test_load_latest_data_none_last_data_timestamp(mongo_db):
         'test_collection',
         {'level': 'ERROR'},
         {'workflow_name': 'test_pipeline', 'schedule_name': 'test_schedule'},
+        limit=0,
+        sort=[('timestamp', 1)],
     )
 
     assert result == [
@@ -487,6 +638,29 @@ def test_load_latest_data_none_last_data_timestamp(mongo_db):
     ]
 
 
+def test_load_latest_data_passes_configured_batch_size_as_limit(mongo_db):
+    """SIENTIAPDE-2013: the configured notification_queue_batch_size caps the query."""
+    mongo_db.notification_queue_batch_size = 100
+    mongo_db.mongo_db_repository.find = MagicMock(return_value=[])
+
+    mongo_db.load_latest_data(
+        {
+            'metadata': {'workflow_name': 'test_pipeline', 'schedule_name': 'test_schedule'},
+            'collection_name': 'test_collection',
+            'last_data_timestamp': None,
+            'base_data_filter': {'level': 'ERROR'},
+        }
+    )
+
+    mongo_db.mongo_db_repository.find.assert_called_once_with(
+        'test_collection',
+        {'level': 'ERROR'},
+        {'workflow_name': 'test_pipeline', 'schedule_name': 'test_schedule'},
+        limit=100,
+        sort=[('timestamp', 1)],
+    )
+
+
 def test_load_latest_data_not_none_last_data_timestamp(mongo_db):
     """Test load_latest_data compares native datetime against native datetime in $gt"""
 
@@ -516,6 +690,8 @@ def test_load_latest_data_not_none_last_data_timestamp(mongo_db):
             'timestamp': {'$gt': datetime(2023, 1, 1, 12, 0, 0, tzinfo=UTC)},
         },
         {'workflow_name': 'test_pipeline', 'schedule_name': 'test_schedule'},
+        limit=0,
+        sort=[('timestamp', 1)],
     )
 
     assert result == [
@@ -529,7 +705,7 @@ def test_load_latest_data_run_boundary_no_skip_or_duplicate(mongo_db):
 
     boundary_timestamp = datetime(2023, 1, 1, 12, 0, 0, tzinfo=UTC)
 
-    def fake_find(collection_name, data_filter, metadata):
+    def fake_find(collection_name, data_filter, metadata, limit=0, sort=None):
         gt = data_filter.get('timestamp', {}).get('$gt')
         all_docs = [
             {
@@ -593,3 +769,48 @@ def test_load_latest_data_error(mongo_db):
             level=NotificationLevel.ERROR,
             attachment_content=ANY,
         )
+
+
+def test_find_documents_in_mongodb_leaves_documents_without_timestamp_field(mongo_db):
+    mongo_db.mongo_db_repository.find = MagicMock(return_value=[{'name': 'no-timestamp'}])
+
+    result = mongo_db.find_documents_in_mongodb(
+        {'query': {'collection': 'test_collection'}, 'timestamp_fields': ['timestamp']}
+    )
+
+    assert result == [{'name': 'no-timestamp'}]
+
+
+def test_aggregate_documents_in_mongodb_leaves_documents_without_timestamp_field(mongo_db):
+    mongo_db.mongo_db_repository.aggregate = MagicMock(return_value=[{'name': 'no-timestamp'}])
+
+    result = mongo_db.aggregate_documents_in_mongodb(
+        {
+            'query': {'collection': 'test_collection', 'aggregation': [{'$match': {}}]},
+            'timestamp_fields': ['timestamp'],
+        }
+    )
+
+    assert result == [{'name': 'no-timestamp'}]
+
+
+@patch('orchestrator.activities.mongo_db.now')
+def test_create_pipelines_timestamps_all_failed_inserts_nothing(_now_mock, mongo_db):
+    mongo_db.mongo_db_repository.insert_many = MagicMock()
+
+    mongo_db.create_pipelines_timestamps(
+        {'created_pipelines': [{'schedule_name': 'a', 'namespace': 'n', 'success': False}]}
+    )
+
+    mongo_db.mongo_db_repository.insert_many.assert_not_called()
+
+
+def test_create_collection_with_ttl_index_skips_pipelines_without_topic(mongo_db):
+    database = mongo_db.mongo_db_repository.database
+    database.list_collection_names = MagicMock(return_value=[])
+
+    mongo_db.create_collection_with_ttl_index(
+        {**metadata, 'pipelines': {'pi-scouter': {'model_tags': {}}, 'held': {'held': True}}}
+    )
+
+    database.create_collection.assert_not_called()

tests/orchestrator/activities/test_slot_manager.py

@@ -389,6 +389,293 @@ def test_filter_notification_alerts(slot_manager):
     }
 
 
+def test_filter_notification_alerts_levels_restriction(slot_manager):
+    # SIENTIAPDE-2012: a group with `levels` set only receives notifications
+    # matching one of those levels.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None, None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'level': 'ERROR',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'level': 'WARNING',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['core_alerts'], 'levels': ['ERROR']},
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['core_alerts'],
+            'levels': ['ERROR'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_1',
+                    'notification_id': 'test_notification_id_1',
+                    'level': 'ERROR',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_alerts_ignore_models_id_restriction(slot_manager):
+    # SIENTIAPDE-2012: a group with `ignore_models_id` set does not receive
+    # notifications from those models, but still receives everything else.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None, None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['core_alerts'],
+                'ignore_models_id': ['model_x'],
+            },
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['core_alerts'],
+            'ignore_models_id': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_2',
+                    'notification_id': 'test_notification_id_2',
+                    'model_id': 'model_y',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_alerts_model_ids_allow_list_restriction(slot_manager):
+    # SIENTIAPDE-2149: a group with `model_ids` set only receives notifications
+    # from those models, unlike `ignore_models_id` which is a deny-list.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None, None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['core_alerts'],
+                'model_ids': ['model_x'],
+            },
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['core_alerts'],
+            'model_ids': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_1',
+                    'notification_id': 'test_notification_id_1',
+                    'model_id': 'model_x',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_alerts_model_ids_and_ignore_models_id_together(slot_manager):
+    # SIENTIAPDE-2149: the allow-list and deny-list apply together (AND), not
+    # as alternatives -- a model must pass both checks.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None, None, None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+            {
+                'trigger': 'test_trigger_3',
+                'notification_id': 'test_notification_id_3',
+                'model_id': 'model_z',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['core_alerts'],
+                'model_ids': ['model_x', 'model_y'],
+                'ignore_models_id': ['model_x'],
+            },
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    # model_x: allowed but ignored -> excluded. model_y: allowed and not
+    # ignored -> included. model_z: not in the allow-list -> excluded.
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['core_alerts'],
+            'model_ids': ['model_x', 'model_y'],
+            'ignore_models_id': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_2',
+                    'notification_id': 'test_notification_id_2',
+                    'model_id': 'model_y',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_alerts_model_ids_explicit_none_does_not_raise(slot_manager):
+    # Same defensive concern as ignore_models_id: receiver_groups has no schema
+    # validation, so model_ids can be explicitly null instead of omitted.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['core_alerts'], 'model_ids': None},
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response['test_group_1']['notifications'] == input_data['notification_package']
+
+
+def test_filter_notification_alerts_no_levels_or_ignore_models_id_passes_everything(
+    slot_manager,
+):
+    # SIENTIAPDE-2012: a group configuring neither field is unaffected -- same
+    # behavior as before this subtask (only `contents`/`ignore` apply).
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None, None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'level': 'ERROR',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'level': 'INFO',
+                'model_id': 'model_y',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['core_alerts']},
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response['test_group_1']['notifications'] == input_data['notification_package']
+
+
+def test_filter_notification_alerts_ignore_models_id_explicit_none_does_not_raise(
+    slot_manager,
+):
+    # receiver_groups has no schema validation -- ignore_models_id can be
+    # explicitly null (key present, value None) instead of omitted. Must not
+    # crash (see the equivalent SIENTIAPDE-2011 fix in load_notification_package).
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['core_alerts'], 'ignore_models_id': None},
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response['test_group_1']['notifications'] == input_data['notification_package']
+
+
 def test_store_notification_cache(slot_manager):
     """Test store_notification_cache"""
     test_data = {

tests/orchestrator/activities/test_temporal_manager.py

@@ -3,6 +3,7 @@ from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
 
 from pytest import fixture, mark
 from sientia_do.notifications.models import NotificationLevel
+from temporalio.client import ScheduleOverlapPolicy
 
 from orchestrator.activities.temporal_manager import TemporalManager
 from orchestrator.utils.converters import parse_frequency
@@ -144,6 +145,7 @@ async def test_create_schedule(
                     'data': {'test': 'test'},
                     'execution_timeout_seconds': 100,
                     'task_timeout_seconds': 100,
+                    'config_hash': 'h1',
                 },
                 'test-schedule-invalid-frequency': {
                     'model_id': 2,
@@ -290,24 +292,29 @@ async def test_create_schedule(
         ]
     )
 
+    # config_hash goes to the report (orchestrated_schedules), not to the workflow
+    assert 'config_hash' not in input_data['schedules']['scouter']['test-schedule']
     assert report == [
         {
             'schedule_name': 'test-schedule',
             'namespace': 'scouter',
             'success': True,
             'message': 'Schedule created successfully',
+            'config_hash': 'h1',
         },
         {
             'schedule_name': 'test-schedule-invalid-frequency',
             'namespace': 'scouter',
             'success': False,
             'message': 'Invalid frequency',
+            'config_hash': None,
         },
         {
             'schedule_name': 'test-schedule-laborious',
             'namespace': 'laborious',
             'success': True,
             'message': 'Schedule created successfully',
+            'config_hash': None,
         },
     ]
 
@@ -354,6 +361,7 @@ async def test_update_schedules(
                     'workflow_type': 'scouter',
                     'frequency': '1m',
                     'data': {'test': 'test'},
+                    'config_hash': 'h1',
                 },
                 'test-schedule_no_handler': {
                     'workflow_type': 'scouter',
@@ -399,24 +407,28 @@ async def test_update_schedules(
     handler_scouter.update.assert_called_once()
     handler_laborious.update.assert_called_once()
 
+    assert 'config_hash' not in input_data['schedules']['scouter']['test-schedule']
     assert report == [
         {
             'schedule_name': 'test-schedule',
             'namespace': 'scouter',
             'success': True,
             'message': 'Schedule updated successfully',
+            'config_hash': 'h1',
         },
         {
             'schedule_name': 'test-schedule_no_handler',
             'namespace': 'scouter',
             'success': False,
             'message': 'Schedule test-schedule_no_handler not found',
+            'config_hash': None,
         },
         {
             'schedule_name': 'test-schedule-laborious',
             'namespace': 'laborious',
             'success': True,
             'message': 'Schedule updated successfully',
+            'config_hash': None,
         },
     ]
 
@@ -684,3 +696,61 @@ async def test_create_schedules_tenant_runtime_queue(
         task_timeout=timedelta(seconds=300),
         typed_search_attributes=_mock_typed_search_attributes.return_value,
     )
+
+
+@mark.asyncio
+@patch('orchestrator.activities.temporal_manager.Schedule')
+@patch('orchestrator.activities.temporal_manager.ScheduleActionStartWorkflow')
+@patch('orchestrator.activities.temporal_manager.ScheduleIntervalSpec')
+@patch('orchestrator.activities.temporal_manager.ScheduleSpec')
+@patch('orchestrator.activities.temporal_manager.TypedSearchAttributes')
+@patch('orchestrator.activities.temporal_manager.SearchAttributePair')
+async def test_create_schedules_drift_keeps_default_skip_overlap(
+    _mock_search_attribute_pair,
+    _mock_typed_search_attributes,
+    _mock_schedule_spec,
+    _mock_schedule_interval_spec,
+    _mock_schedule_action_start_workflow,
+    mock_schedule,
+    temporal_manager,
+):
+    # The laborious DriftAnalysis cache assumes no two drift runs of a model overlap.
+    input_data = {
+        'schedules': {
+            'laborious': {
+                'm1-drift-1m': {
+                    'model_id': 'm1',
+                    'model_name': 'm1',
+                    'workflow_type': 'drift',
+                    'frequency': '1m',
+                }
+            }
+        }
+    }
+    temporal_manager.temporal_clients['laborious'].create_schedule = AsyncMock()
+
+    await temporal_manager.create_schedules(input_data)
+
+    mock_schedule.assert_called_once()
+    policy = mock_schedule.call_args.kwargs.get('policy')
+    assert policy is None or policy.overlap == ScheduleOverlapPolicy.SKIP
+
+
+@mark.asyncio
+async def test_normalize_schedules_keeps_schedules_known_in_mongo(temporal_manager):
+    """An orchestrated schedule still tracked in orchestrated_schedules is not deleted"""
+    input_data = {
+        'orchestrated_schedules': {
+            'scouter': {'test-schedule-id': '2021-01-01'},
+            'laborious': {'test-schedule-id': '2021-01-01'},
+        },
+        **metadata,
+    }
+    for client in temporal_manager.temporal_clients.values():
+        client.list_schedules = AsyncMock(return_value=async_iter())
+        client.get_schedule_handle = MagicMock()
+
+    await temporal_manager.normalize_schedules(input_data)
+
+    for client in temporal_manager.temporal_clients.values():
+        client.get_schedule_handle.assert_not_called()

tests/orchestrator/utils/tag_database_fixtures.py

@@ -0,0 +1,82 @@
+"""Pipeline and tag database documents shared by the utils tests."""
+
+
+def scouter_pipeline_doc(schedule_name, model_id, read_tags, workflow_type='scouter'):
+    return {
+        'workflow_type': workflow_type,
+        'schedule_name': schedule_name,
+        'model_id': model_id,
+        'model': {'name': f'model {model_id}'},
+        'read_tags': read_tags,
+    }
+
+
+TAGS_BY_ID = {
+    7: {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1001', 'tag_name': 'Pressure'},
+    8: {'tag_id': 8, 'connector_id': 5, 'address': 'ns=3;i=1002', 'tag_name': 'Level'},
+}
+
+
+# Connector security and client certificates (SIENTIAPDE-1974)
+
+NO_SECURITY = {'mode': 'None', 'policy': 'None'}
+SECURE = {'mode': 'SignAndEncrypt', 'policy': 'Basic256Sha256'}
+
+
+# A client certificate as connectors_query's $lookup brings it (certificates[])
+def certificate_doc(certificate_id=1, name='sientia-platform', **extra):
+    return {
+        'certificate_id': certificate_id,
+        'application_uri': 'urn:sientia:opc-ingestor',
+        'cert': {'bucket': 'opc-certificates', 'object_name': f'client/{name}/cert.pem'},
+        'private_key': {'bucket': 'opc-certificates', 'object_name': f'client/{name}/key.pem'},
+        **extra,
+    }
+
+
+# The slot's certificate for certificate_doc(): the same fields
+CERTIFICATE = certificate_doc()
+
+
+def secure_connector_fields(certificate_id=1, certificates=None):
+    """security and certificates of a connector with a client certificate."""
+    return {
+        'security': {**SECURE, 'client_certificate_id': certificate_id},
+        'certificates': [certificate_doc(certificate_id)] if certificates is None else certificates,
+    }
+
+
+# Write tags (predictions_batch write_tags through the tag database)
+
+OPC_CONNECTORS = {
+    'plant': {
+        'connector_id': 'plant',
+        'name': 'plant',
+        'url': 'opc.tcp://plant:4840',
+        'security': NO_SECURITY,
+    },
+}
+
+
+def write_tag_doc(tag_id, address, category='Write', connector_id='plant', **extra):
+    return {
+        'tag_id': tag_id,
+        'connector_id': connector_id,
+        'address': address,
+        'tag_name': f'tag-{tag_id}',
+        'category': category,
+        'status': 'Active',
+        **extra,
+    }
+
+
+def batch_pipeline_doc(schedule_name, write_tags, **fields):
+    return {
+        'workflow_type': 'predictions_batch',
+        'schedule_name': schedule_name,
+        'model_id': 'model_id',
+        'model': {'name': 'model_name'},
+        'query': 'SELECT 1',
+        'write_tags': write_tags,
+        **fields,
+    }

tests/orchestrator/utils/test_connectors_config.py

@@ -64,10 +64,12 @@ def test_build_mongo_db_config_with_env_vars():
     environ['MONGODB_URL'] = 'localhost:27018'
     environ['MONGODB_DATABASE_NAME'] = 'test_db'
     environ['MONGODB_TTL_INDEX_HOURS'] = '2'
+    environ['NOTIFICATION_QUEUE_BATCH_SIZE'] = '2000'
     assert build_mongodb_config() == {
         'connection_string': 'mongodb://sientia1:sientia1@localhost:27018',
         'database_name': 'test_db',
         'ttl_index_seconds': 7200,
+        'notification_queue_batch_size': 2000,
     }
 
 
@@ -77,10 +79,12 @@ def test_build_mongo_db_config_with_defaults():
     environ.pop('MONGODB_DATABASE_NAME', None)
     environ.pop('MONGODB_URL', None)
     environ.pop('MONGODB_TTL_INDEX_HOURS', None)
+    environ.pop('NOTIFICATION_QUEUE_BATCH_SIZE', None)
     assert build_mongodb_config() == {
         'connection_string': 'mongodb://root:wKZDbMNU1c@localhost:27018',
         'database_name': 'sientia',
         'ttl_index_seconds': 3600,
+        'notification_queue_batch_size': 0,
     }
 
 

tests/orchestrator/utils/test_opc_slots.py

@@ -0,0 +1,450 @@
+import pytest
+
+from orchestrator.utils.opc_slots import (
+    build_tag_config,
+    gather_read_tags,
+)
+from tests.orchestrator.utils.tag_database_fixtures import (
+    CERTIFICATE,
+    NO_SECURITY,
+    SECURE,
+    TAGS_BY_ID,
+    certificate_doc,
+    scouter_pipeline_doc,
+    secure_connector_fields,
+)
+
+NO_ISSUES = {'tags': set(), 'connectors': set(), 'invalid': set(), 'certificates': set()}
+
+
+def _name(schedule_name, tag_name, model_id):
+    return {
+        'collection': f'raw_{schedule_name}',
+        'tag_name': tag_name,
+        'model_id': model_id,
+        'model_name': f'model {model_id}',
+    }
+
+
+def _read(tag_id, frequency=1000, **extra):
+    return {'tag_id': tag_id, 'frequency': frequency, **extra}
+
+
+def _connector(connector_id, name, **extra):
+    return {
+        'connector_id': connector_id,
+        'name': name,
+        'url': f'opc.tcp://{name}:4840',
+        'security': NO_SECURITY,
+        **extra,
+    }
+
+
+def _db_tag(tag_id, connector_id, address, tag_name):
+    return {
+        'tag_id': tag_id,
+        'connector_id': connector_id,
+        'address': address,
+        'tag_name': tag_name,
+    }
+
+
+connectors = {
+    '1': _connector(1, 'kepserver'),
+    '3': _connector(3, 'plant', subscription_period_ms=1500, **secure_connector_fields()),
+}
+
+
+tags_by_id = {
+    **TAGS_BY_ID,
+    10: _db_tag(10, 1, 'ns=2;s=A', 'A'),
+    11: _db_tag(11, 1, 'ns=2;s=B', 'B'),
+    12: _db_tag(12, 1, 'ns=2;s=A', 'A2'),
+    13: _db_tag(13, 3, 'ns=2;s=A', 'A on plant'),
+    14: _db_tag(14, 1, 'ns=2;s=C', 'C'),
+}
+
+
+def _gather(pipelines, connector_index=None, tag_index=None):
+    return gather_read_tags(
+        pipelines,
+        connectors if connector_index is None else connector_index,
+        tags_by_id if tag_index is None else tag_index,
+    )
+
+
+def test_gather_read_tags_multi_name():
+    """One address read by two pipelines with different names: one entry, two names"""
+    pipelines = [
+        scouter_pipeline_doc('sin', 'm1', [_read(10, 15000, tag_name='Counter')]),
+        scouter_pipeline_doc('cos', 'm2', [_read(10, 5000, tag_name='Contador')]),
+    ]
+
+    entries, issues = _gather(pipelines)
+
+    assert len(entries) == 1
+    assert entries[0]['address'] == 'ns=2;s=A'
+    assert entries[0]['frequency'] == 5000
+    assert entries[0]['names'] == [_name('cos', 'Contador', 'm2'), _name('sin', 'Counter', 'm1')]
+    assert issues == NO_ISSUES
+
+
+def test_gather_read_tags_same_name_in_two_pipelines():
+    """The same name read by two pipelines still goes to both collections"""
+    pipelines = [
+        scouter_pipeline_doc('sin', 'm1', [_read(10)]),
+        scouter_pipeline_doc('cos', 'm2', [_read(10)]),
+    ]
+
+    entries, _issues = _gather(pipelines)
+
+    assert entries[0]['names'] == [_name('cos', 'A', 'm2'), _name('sin', 'A', 'm1')]
+
+
+def test_gather_read_tags_duplicate_tag_in_one_pipeline():
+    entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(10), _read(10)])])
+
+    assert entries[0]['names'] == [_name('sin', 'A', 'm1')]
+
+
+def test_gather_read_tags_two_tag_ids_on_one_address():
+    """Two connector_tags on the same node: one entry, one name per tag"""
+    entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(10), _read(12)])])
+
+    assert len(entries) == 1
+    assert entries[0]['names'] == [_name('sin', 'A', 'm1'), _name('sin', 'A2', 'm1')]
+
+
+def test_gather_read_tags_same_address_on_two_servers():
+    entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(13), _read(10)])])
+
+    assert [(e['server']['server_id'], e['address']) for e in entries] == [
+        ('1', 'ns=2;s=A'),
+        ('3', 'ns=2;s=A'),
+    ]
+
+
+def test_gather_read_tags_with_tag_database():
+    """Address, name and server come from connector_tags + connectors"""
+    entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(7)])])
+
+    assert entries == [
+        {
+            'server': {
+                'server_id': '3',
+                'name': 'plant',
+                'url': 'opc.tcp://plant:4840',
+                'subscription_period_ms': 1500,
+                'security': SECURE,
+                'certificate': CERTIFICATE,
+                'key': 'plant',
+            },
+            'address': 'ns=3;i=1001',
+            'frequency': 1000,
+            'active_read': {'enabled': False},
+            'names': [_name('sin', 'Pressure', 'm1')],
+        }
+    ]
+
+
+def test_gather_read_tags_pipeline_name_wins_over_tag_database():
+    entries, _issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(7, tag_name='P1')])])
+
+    assert entries[0]['names'] == [_name('sin', 'P1', 'm1')]
+
+
+def test_gather_read_tags_unknown_tag_id():
+    entries, issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(404)])])
+
+    assert entries == []
+    assert issues == {**NO_ISSUES, 'tags': {'404'}}
+
+
+def test_gather_read_tags_inactive_connector():
+    """Connector 5 is not among the active connectors: its tag is left out"""
+    entries, issues = _gather([scouter_pipeline_doc('sin', 'm1', [_read(8)])])
+
+    assert entries == []
+    assert issues == {**NO_ISSUES, 'connectors': {'5'}}
+
+
+def test_gather_read_tags_without_tag_id():
+    """connectors is the only source of the servers: a read tag needs a tag_id"""
+    pipelines = [
+        scouter_pipeline_doc(
+            'sin',
+            'm1',
+            [
+                _read(10),
+                {'tag_name': 'b', 'frequency': 1000},
+                {'tag_id': '', 'tag_name': 'c', 'frequency': 1000},
+                {'tag_id': None, 'tag_name': 'd', 'frequency': 1000},
+                # tag_id is int in connector_tags: text is no reference
+                {'tag_id': '11', 'frequency': 1000},
+            ],
+        )
+    ]
+
+    entries, issues = _gather(pipelines)
+
+    assert [e['address'] for e in entries] == ['ns=2;s=A']
+    assert issues == {**NO_ISSUES, 'invalid': {'sin/b', 'sin/c', 'sin/d', 'sin/11'}}
+
+
+def test_gather_read_tags_server_name_clash():
+    """Two servers with the same name get distinct slot keys, stable by server_id"""
+    connector_index = {**connectors, '2': _connector(2, 'kepserver')}
+    tag_index = {**tags_by_id, 20: _db_tag(20, 2, 'ns=2;s=A', 'A')}
+    pipelines = [scouter_pipeline_doc('sin', 'm1', [_read(20), _read(11)])]
+
+    entries, _issues = _gather(pipelines, connector_index, tag_index)
+
+    assert [e['server']['key'] for e in entries] == ['kepserver', 'kepserver:2']
+    assert [e['server']['name'] for e in entries] == ['kepserver', 'kepserver']
+
+
+def test_gather_read_tags_active_read():
+    """Active read: on if any pipeline asks, with the lowest cycle_multiple (D3)"""
+    pipelines = [
+        scouter_pipeline_doc('p1', 'm1', [_read(10)]),
+        scouter_pipeline_doc(
+            'p2', 'm2', [_read(10, active_read={'enabled': True, 'cycle_multiple': 4})]
+        ),
+        scouter_pipeline_doc(
+            'p3', 'm3', [_read(10, active_read={'enabled': True, 'cycle_multiple': 2})]
+        ),
+        scouter_pipeline_doc(
+            'p4', 'm4', [_read(11, active_read={'enabled': False, 'cycle_multiple': 1})]
+        ),
+        scouter_pipeline_doc('p5', 'm5', [_read(14, active_read={'enabled': True})]),
+    ]
+
+    entries, _issues = _gather(pipelines)
+
+    assert [e['active_read'] for e in entries] == [
+        {'enabled': True, 'cycle_multiple': 2},
+        {'enabled': False},
+        {'enabled': True, 'cycle_multiple': 1},
+    ]
+
+
+def test_gather_read_tags_filters_non_scouter_workflows():
+    """Only scouter pipelines feed the slots"""
+    pipelines = [
+        scouter_pipeline_doc('sin', 'm1', [_read(10)]),
+        scouter_pipeline_doc('pred', 'm2', [_read(11)], workflow_type='predictions_batch'),
+        scouter_pipeline_doc('retrain', 'm3', [_read(14)], workflow_type='minimal_retrain'),
+        scouter_pipeline_doc('drift', 'm4', [_read(7)], workflow_type='drift'),
+    ]
+
+    entries, _issues = _gather(pipelines)
+
+    assert [e['address'] for e in entries] == ['ns=2;s=A']
+
+
+def test_gather_read_tags_without_read_tags():
+    pipeline = scouter_pipeline_doc('sin', 'm1', [])
+    del pipeline['read_tags']
+
+    assert _gather([pipeline]) == ([], NO_ISSUES)
+
+
+def test_build_tag_config():
+    """Slot v2 (contract C1): security, certificate and subscription period per server"""
+    pipelines = [
+        scouter_pipeline_doc('sin', 'm1', [_read(10, 1000), _read(11, '300'), _read(7, 100)])
+    ]
+    entries, _issues = _gather(pipelines)
+
+    result = build_tag_config(entries)
+
+    assert result == {
+        'kepserver': {
+            'server_id': '1',
+            'name': 'kepserver',
+            'url': 'opc.tcp://kepserver:4840',
+            # No client certificate: no security (D5)
+            'security': NO_SECURITY,
+            'certificate': None,
+            # No subscription_period_ms on the connector: half its lowest tag frequency
+            'subscription_period_ms': 150,
+            'tags': {
+                'ns=2;s=A': {
+                    'frequency': 1000,
+                    'active_read': {'enabled': False},
+                    'names': [_name('sin', 'A', 'm1')],
+                },
+                'ns=2;s=B': {
+                    'frequency': 300,
+                    'active_read': {'enabled': False},
+                    'names': [_name('sin', 'B', 'm1')],
+                },
+            },
+        },
+        'plant': {
+            'server_id': '3',
+            'name': 'plant',
+            'url': 'opc.tcp://plant:4840',
+            'subscription_period_ms': 1500,
+            'security': SECURE,
+            'certificate': CERTIFICATE,
+            'tags': {
+                'ns=3;i=1001': {
+                    'frequency': 100,
+                    'active_read': {'enabled': False},
+                    'names': [_name('sin', 'Pressure', 'm1')],
+                },
+            },
+        },
+    }
+
+
+def test_build_tag_config_empty():
+    assert build_tag_config([]) == {}
+
+
+def test_gather_read_tags_invalid_read_tags():
+    """Unusable read tags are reported and skipped; the others still make the slot"""
+    pipelines = [
+        scouter_pipeline_doc('sin', 'm1', [_read(10), {'tag_id': 11}, _read(7, frequency='fast')])
+    ]
+
+    entries, issues = _gather(pipelines)
+
+    assert [e['address'] for e in entries] == ['ns=2;s=A']
+    assert issues['invalid'] == {'sin/11', 'sin/7'}
+
+
+def test_gather_read_tags_frequency_type_does_not_depend_on_order():
+    """15000 and '15000' give the same int frequency in any pipeline order (O6)"""
+    as_int = scouter_pipeline_doc('sin', 'm1', [_read(10, 15000)])
+    as_text = scouter_pipeline_doc('cos', 'm2', [_read(10, '15000')])
+
+    first, _ = _gather([as_int, as_text])
+    second, _ = _gather([as_text, as_int])
+
+    assert first[0]['frequency'] == second[0]['frequency'] == 15000
+    assert isinstance(first[0]['frequency'], int)
+    assert isinstance(second[0]['frequency'], int)
+
+
+def test_gather_read_tags_server_key_never_collides():
+    """A server literally named 'x:3' does not share its key with server 3"""
+    connector_index = {
+        '1': _connector(1, 'x:3'),
+        '2': _connector(2, 'x'),
+        '3': _connector(3, 'x'),
+    }
+    tag_index = {
+        1: _db_tag(1, 1, 'ns=2;s=A', 'a'),
+        2: _db_tag(2, 2, 'ns=2;s=B', 'b'),
+        3: _db_tag(3, 3, 'ns=2;s=C', 'c'),
+    }
+    pipelines = [scouter_pipeline_doc('sin', 'm1', [_read(1), _read(2), _read(3)])]
+
+    entries, _issues = _gather(pipelines, connector_index, tag_index)
+    slot = build_tag_config(entries)
+
+    keys = [e['server']['key'] for e in entries]
+    assert len(set(keys)) == 3
+    assert {server['server_id'] for server in slot.values()} == {'1', '2', '3'}
+    assert all(len(server['tags']) == 1 for server in slot.values())
+
+
+def test_gather_read_tags_invalid_cycle_multiple_does_not_fail_the_slot():
+    """A bad active_read is reported as invalid instead of failing every pipeline's slot"""
+    pipelines = [
+        scouter_pipeline_doc(
+            'sin',
+            'm1',
+            [
+                _read(10),
+                _read(11, active_read={'enabled': True, 'cycle_multiple': 'x'}),
+                _read(7, active_read={'enabled': True, 'cycle_multiple': None}),
+                # Not a dict: treated as no active read
+                _read(14, active_read=True),
+            ],
+        )
+    ]
+
+    entries, issues = _gather(pipelines)
+
+    assert [e['address'] for e in entries] == ['ns=2;s=A', 'ns=2;s=C']
+    assert entries[1]['active_read'] == {'enabled': False}
+    assert issues['invalid'] == {'sin/11', 'sin/7'}
+
+
+def test_gather_read_tags_skips_incomplete_pipelines():
+    """A pipeline without schedule_name, model_id or model.name is reported, not raised"""
+    no_model = scouter_pipeline_doc('no-model', 'm1', [_read(10)])
+    del no_model['model']
+    no_name = scouter_pipeline_doc('no-name', 'm2', [_read(10)])
+    no_name['model'] = {}
+    no_model_id = scouter_pipeline_doc('no-id', None, [_read(10)])
+    no_schedule = scouter_pipeline_doc(None, 'm4', [_read(10)])
+    ok = scouter_pipeline_doc('ok', 'm5', [_read(11)])
+
+    entries, issues = _gather([no_model, no_name, no_model_id, no_schedule, ok])
+
+    assert [e['address'] for e in entries] == ['ns=2;s=B']
+    assert issues['invalid'] == {'no-model', 'no-name', 'no-id', 'None'}
+
+
+def _plant_with(**fields):
+    return {'1': connectors['1'], '3': {**connectors['3'], **fields}}
+
+
+@pytest.mark.parametrize(
+    'fields',
+    [
+        # $lookup found nothing: missing id, inactive or not a client certificate
+        pytest.param(secure_connector_fields(certificates=[]), id='not-found'),
+        pytest.param(
+            secure_connector_fields(certificates=[certificate_doc(application_uri='')]),
+            id='no-application-uri',
+        ),
+        pytest.param(
+            secure_connector_fields(
+                certificates=[{**certificate_doc(), 'private_key': {'bucket': 'b'}}]
+            ),
+            id='key-without-object',
+        ),
+    ],
+)
+def test_gather_read_tags_registered_certificate_missing(fields):
+    """An id without a usable certificate: in the slot without a certificate, with a warning (D5)"""
+    entries, issues = _gather(
+        [scouter_pipeline_doc('sin', 'm1', [_read(7)])], _plant_with(**fields)
+    )
+
+    assert entries[0]['server']['certificate'] is None
+    assert entries[0]['server']['security'] == SECURE
+    assert issues == {**NO_ISSUES, 'certificates': {'3'}}
+
+
+@pytest.mark.parametrize('certificate_id', [None, 'absent'])
+def test_gather_read_tags_without_certificate_id(certificate_id):
+    """No client_certificate_id: no certificate and nothing to report (D5)"""
+    security = {**NO_SECURITY, 'client_certificate_id': certificate_id}
+    if certificate_id == 'absent':
+        security = NO_SECURITY
+    entries, issues = _gather(
+        [scouter_pipeline_doc('sin', 'm1', [_read(7)])],
+        _plant_with(security=security, certificates=[]),
+    )
+
+    assert entries[0]['server']['certificate'] is None
+    assert issues == NO_ISSUES
+
+
+def test_gather_read_tags_reports_a_certificate_once_per_connector():
+    entries, issues = _gather(
+        [
+            scouter_pipeline_doc('sin', 'm1', [_read(7)]),
+            scouter_pipeline_doc('cos', 'm2', [_read(7)]),
+        ],
+        _plant_with(**secure_connector_fields(certificates=[])),
+    )
+
+    assert len(entries) == 1
+    assert issues['certificates'] == {'3'}

tests/orchestrator/utils/test_pipeline_configs.py

@@ -1,11 +1,13 @@
 from unittest.mock import call, patch
 
-from orchestrator.utils.orchestrator_functions import (
+from pytest import mark
+
+from orchestrator.utils.pipeline_configs import (
     base_scouter,
-    build_tag_config,
+    chunk_period_warning,
     common_config,
     drift,
-    gather_read_tags,
+    frequency_alert_warning,
     minimal_retrain,
     overlap_filter_config,
     pi_web_api_scouter,
@@ -76,6 +78,7 @@ def test_drift():
         'source_table_name': 'laborious_data',
         'target_table_name': 'drift_metrics',
         'interval': 120,
+        'chunk_period': 'min',
         'drift_metrics': ['kolmogorov_smirnov', 'jensen_shannon'],
         'execution_timeout_seconds': 300,
         'task_timeout_seconds': 300,
@@ -84,6 +87,79 @@ def test_drift():
     assert result == expected
 
 
+def _drift_config(**fields):
+    return {
+        'workflow_type': 'drift',
+        'schedule_name': 'm1-drift',
+        'model_id': 'm1',
+        'model': {'name': 'm1'},
+        **fields,
+    }
+
+
+def test_drift_default_window_is_frequency():
+    result = drift(_drift_config())
+
+    assert result['frequency'] == '1m'
+    assert result['interval'] == 1
+
+
+@mark.parametrize(
+    ('frequency', 'interval'),
+    [('1h', 60), ('30s', 1), ('90s', 2)],
+)
+def test_drift_window_derived_from_frequency(frequency, interval):
+    assert drift(_drift_config(frequency=frequency))['interval'] == interval
+
+
+def test_drift_explicit_interval_minutes_wins():
+    result = drift(_drift_config(frequency='1m', interval_minutes=60))
+
+    assert result['interval'] == 60
+
+
+def test_drift_invalid_frequency_falls_back_to_legacy_window():
+    result = drift(_drift_config(frequency='abc'))
+
+    assert result['frequency'] == 'abc'
+    assert result['interval'] == 60
+
+
+def test_drift_chunk_period_defaults_to_min():
+    result = drift(_drift_config())
+
+    assert result['chunk_period'] == 'min'
+    assert 'chunk_period_configured' not in result
+
+
+def test_drift_chunk_period_seconds_kept():
+    result = drift(_drift_config(chunk_period='s'))
+
+    assert result['chunk_period'] == 's'
+    assert 'chunk_period_configured' not in result
+
+
+@mark.parametrize('chunk_period', ['hour', 5])
+def test_drift_invalid_chunk_period_falls_back_to_min(chunk_period):
+    result = drift(_drift_config(chunk_period=chunk_period))
+
+    assert result['chunk_period'] == 'min'
+    assert result['chunk_period_configured'] == chunk_period
+
+
+@mark.parametrize('chunk_period', [None, 'min', 's'])
+def test_chunk_period_warning_none_when_absent_or_valid(chunk_period):
+    assert chunk_period_warning(chunk_period) is None
+
+
+@mark.parametrize('chunk_period', ['hour', 5])
+def test_chunk_period_warning_message_when_invalid(chunk_period):
+    warning = chunk_period_warning(chunk_period)
+
+    assert warning is not None
+    assert repr(chunk_period) in warning
+
+
 def test_simple_metrics():
     config = {
         'workflow_type': 'simple_metrics',
@@ -110,6 +186,7 @@ def test_simple_metrics():
         'target_table_name': 'simple_metrics',
         'interval_minutes': 120,
         'metrics': ['rmse', 'mse'],
+        'frequency_alert': None,
         'execution_timeout_seconds': 300,
         'task_timeout_seconds': 300,
         'runtime': 'legacy',
@@ -117,6 +194,61 @@ def test_simple_metrics():
     assert result == expected
 
 
+def _simple_metrics_config(**fields):
+    return {
+        'workflow_type': 'simple_metrics',
+        'schedule_name': 'm1-simple_metrics-60m',
+        'model_id': 'm1',
+        'model': {'name': 'm1'},
+        'frequency': '1h',
+        **fields,
+    }
+
+
+def test_simple_metrics_escalated_uses_frequency_alert():
+    config = _simple_metrics_config(
+        frequency_alert='5m', adaptive_state={'escalated': True, 'healthy_streak': 0}
+    )
+    result = simple_metrics(config)
+    assert result['frequency'] == '5m'
+    assert result['frequency_alert'] == '5m'
+
+
+def test_simple_metrics_not_escalated_keeps_frequency():
+    config = _simple_metrics_config(
+        frequency_alert='5m', adaptive_state={'escalated': False, 'healthy_streak': 2}
+    )
+    assert simple_metrics(config)['frequency'] == '1h'
+
+
+def test_simple_metrics_escalated_without_frequency_alert_keeps_frequency():
+    # Removing frequency_alert turns adaptive mode off even if a stale state says escalated.
+    config = _simple_metrics_config(adaptive_state={'escalated': True, 'healthy_streak': 0})
+    assert simple_metrics(config)['frequency'] == '1h'
+
+
+def test_simple_metrics_frequency_alert_without_state_keeps_frequency():
+    assert simple_metrics(_simple_metrics_config(frequency_alert='5m'))['frequency'] == '1h'
+
+
+def test_simple_metrics_escalated_unparseable_frequency_alert_keeps_frequency():
+    # An unparseable frequency would make the Temporal schedule create/update fail.
+    config = _simple_metrics_config(frequency_alert='5min', adaptive_state={'escalated': True})
+    assert simple_metrics(config)['frequency'] == '1h'
+
+
+def test_simple_metrics_escalated_below_floor_frequency_alert_is_applied():
+    # Decision 7: below the floor only warns, the value is kept.
+    config = _simple_metrics_config(frequency_alert='30s', adaptive_state={'escalated': True})
+    assert simple_metrics(config)['frequency'] == '30s'
+
+
+@mark.parametrize('adaptive_state', ['yes', {'escalated': 'false'}])
+def test_simple_metrics_malformed_adaptive_state_keeps_frequency(adaptive_state):
+    config = _simple_metrics_config(frequency_alert='5m', adaptive_state=adaptive_state)
+    assert simple_metrics(config)['frequency'] == '1h'
+
+
 def test_minimal_retrain():
     config = {
         'workflow_type': 'minimal_retrain',
@@ -211,11 +343,11 @@ def test_process_path_priority():
 
 
 @patch(
-    'orchestrator.utils.orchestrator_functions.overlap_filter_config',
+    'orchestrator.utils.pipeline_configs.overlap_filter_config',
     return_value={'test_filter_name': {'policy': 'test_policy', 'config': {}}},
 )
 @patch(
-    'orchestrator.utils.orchestrator_functions.process_path_priority',
+    'orchestrator.utils.pipeline_configs.process_path_priority',
     return_value=['STOP', 'CONTINUE', 'REPEAT'],
 )
 def test_predictions_batch(mock_process_path_priority, mock_overlap_filter_config):
@@ -225,10 +357,14 @@ def test_predictions_batch(mock_process_path_priority, mock_overlap_filter_confi
         'model_id': 'test_model_id',
         'model': {'name': 'test_model_name', 'model_config': {'test_config': 'test_config'}},
         'query': 'test_query',
-        'write_tags': [
-            {'server_id': 'test_server_id', 'type': 'prediction', 'addr': 'test_addr'},
-            {'server_id': 'test_server_id', 'type': 'confidence', 'addr': 'test_addr'},
-        ],
+        'write_tags': [{'tag_id': 77}, {'tag_id': 78}],
+        # Resolved from write_tags by process_schedules
+        'opc_output_config': {
+            'test_server_id': {
+                'prediction_tags': {'test_addr': {'data_type': 'float'}},
+                'confidence_tags': {'test_addr': {'data_type': 'float'}},
+            }
+        },
         'pi_web_api_output_config': {
             'endpoint': 'test_endpoint',
             'prediction_tags': {
@@ -311,249 +447,17 @@ def test_predictions_batch(mock_process_path_priority, mock_overlap_filter_confi
     assert result == expected
 
 
-def test_gather_read_tags():
-    pipelines = [
-        {
-            'workflow_type': 'scouter',
-            'schedule_name': 'test_schedule',
-            'read_tags': [
-                {
-                    'server_id': '1',
-                    'server_name': 'test_server_name',
-                    'tag_address': 'test_tag_address',
-                }
-            ],
-        },
-        {
-            'workflow_type': 'scouter',
-            'schedule_name': 'test_schedule2',
-            'read_tags': [
-                {
-                    'server_id': '2',
-                    'server_name': 'test_server_name2',
-                    'tag_address': 'test_tag_address2',
-                },
-                {
-                    'server_id': '2',
-                    'server_name': 'test_server_name2',
-                    'tag_address': 'test_tag_address3',
-                },
-            ],
-        },
-    ]
-
-    result = gather_read_tags(pipelines)
-
-    expected = {
-        '1:test_tag_address': {
-            'server_id': '1',
-            'server_name': 'test_server_name',
-            'tag_address': 'test_tag_address',
-            'topics': ['raw_test_schedule'],
-        },
-        '2:test_tag_address2': {
-            'server_id': '2',
-            'server_name': 'test_server_name2',
-            'tag_address': 'test_tag_address2',
-            'topics': ['raw_test_schedule2'],
-        },
-        '2:test_tag_address3': {
-            'server_id': '2',
-            'server_name': 'test_server_name2',
-            'tag_address': 'test_tag_address3',
-            'topics': ['raw_test_schedule2'],
-        },
+def test_scouter_skips_tags_without_name():
+    """A tag_id the tag database cannot resolve has no name and stays out of model_tags"""
+    config = {
+        'workflow_type': 'scouter',
+        'schedule_name': 'sin',
+        'model_id': 'm1',
+        'model': {'name': 'model m1'},
+        'read_tags': [{'tag_name': 'a'}, {'tag_id': 404}],
     }
 
-    assert result == expected
-
-
-def test_gather_read_tags_filters_non_scouter_workflows():
-    """Test that gather_read_tags only processes scouter workflow types and ignores others"""
-    pipelines = [
-        {
-            'workflow_type': 'scouter',
-            'schedule_name': 'test_scouter_schedule',
-            'read_tags': [
-                {
-                    'server_id': '1',
-                    'server_name': 'test_server_name',
-                    'tag_address': 'test_tag_address',
-                }
-            ],
-        },
-        {
-            'workflow_type': 'predictions_batch',
-            'schedule_name': 'test_predictions_schedule',
-            'read_tags': [
-                {
-                    'server_id': '2',
-                    'server_name': 'test_server_name2',
-                    'tag_address': 'test_tag_address_predictions',
-                }
-            ],
-        },
-        {
-            'workflow_type': 'minimal_retrain',
-            'schedule_name': 'test_retrain_schedule',
-            'read_tags': [
-                {
-                    'server_id': '3',
-                    'server_name': 'test_server_name3',
-                    'tag_address': 'test_tag_address_retrain',
-                }
-            ],
-        },
-        {
-            'workflow_type': 'pi_web_api_scouter',
-            'schedule_name': 'test_pi_web_api_schedule',
-            'read_tags': [
-                {
-                    'server_id': '4',
-                    'server_name': 'test_server_name4',
-                    'tag_address': 'test_tag_address_pi_web_api',
-                }
-            ],
-        },
-        {
-            'workflow_type': 'drift',
-            'schedule_name': 'test_drift_schedule',
-            'read_tags': [
-                {
-                    'server_id': '5',
-                    'server_name': 'test_server_name5',
-                    'tag_address': 'test_tag_address_drift',
-                }
-            ],
-        },
-        {
-            'workflow_type': 'scouter',
-            'schedule_name': 'test_scouter_schedule2',
-            'read_tags': [
-                {
-                    'server_id': '1',
-                    'server_name': 'test_server_name',
-                    'tag_address': 'test_tag_address2',
-                }
-            ],
-        },
-    ]
-
-    result = gather_read_tags(pipelines)
-
-    # Only scouter workflow types should be included
-    expected = {
-        '1:test_tag_address': {
-            'server_id': '1',
-            'server_name': 'test_server_name',
-            'tag_address': 'test_tag_address',
-            'topics': ['raw_test_scouter_schedule'],
-        },
-        '1:test_tag_address2': {
-            'server_id': '1',
-            'server_name': 'test_server_name',
-            'tag_address': 'test_tag_address2',
-            'topics': ['raw_test_scouter_schedule2'],
-        },
-    }
-
-    assert result == expected
-    # Ensure non-scouter pipelines are not included
-    assert '2:test_tag_address_predictions' not in result
-    assert '3:test_tag_address_retrain' not in result
-    assert '4:test_tag_address_pi_web_api' not in result
-    assert '5:test_tag_address_drift' not in result
-
-
-def test_build_tag_config():
-    tags = [
-        {
-            'server_id': '1',
-            'server_name': 'test_server_name',
-            'tag_address': 'test_tag_address',
-            'frequency': 1000,
-        },
-        {
-            'server_id': '2',
-            'server_name': 'test_server_name2',
-            'tag_address': 'test_tag_address2',
-            'frequency': 1000,
-        },
-        {
-            'server_id': '2',
-            'server_name': 'test_server_name2',
-            'tag_address': 'test_tag_address3',
-            'frequency': 300,
-        },
-        {
-            'server_id': '3',
-            'server_name': 'test_server_name3',
-            'tag_address': 'test_tag_address4',
-            'frequency': 200,
-        },
-    ]
-
-    opc_servers = {
-        '1': {
-            'server_name': 'test_server_name',
-            'url': 'test_url',
-            'uri': 'test_uri',
-        },
-        '2': {
-            'server_name': 'test_server_name2',
-            'url': 'test_url2',
-            'uri': 'test_uri2',
-        },
-    }
-
-    result = build_tag_config(tags, opc_servers)
-
-    expected = {
-        'test_server_name': {
-            'server_id': '1',
-            'name': 'test_server_name',
-            'url': 'test_url',
-            'server_uri': 'test_uri',
-            'cert_path': None,
-            'private_key_path': None,
-            'server_cert_path': None,
-            'subscription_period_ms': 500,
-            'tags': {
-                'test_tag_address': {
-                    'server_id': '1',
-                    'server_name': 'test_server_name',
-                    'tag_address': 'test_tag_address',
-                    'frequency': 1000,
-                },
-            },
-        },
-        'test_server_name2': {
-            'server_id': '2',
-            'name': 'test_server_name2',
-            'url': 'test_url2',
-            'server_uri': 'test_uri2',
-            'cert_path': None,
-            'private_key_path': None,
-            'server_cert_path': None,
-            'subscription_period_ms': 150,
-            'tags': {
-                'test_tag_address2': {
-                    'server_id': '2',
-                    'server_name': 'test_server_name2',
-                    'tag_address': 'test_tag_address2',
-                    'frequency': 1000,
-                },
-                'test_tag_address3': {
-                    'server_id': '2',
-                    'server_name': 'test_server_name2',
-                    'tag_address': 'test_tag_address3',
-                    'frequency': 300,
-                },
-            },
-        },
-    }
-
-    assert result == (expected, ['3'])
+    assert list(scouter(config)['model_tags']) == ['a']
 
 
 def test_base_scouter():
@@ -762,3 +666,77 @@ def test_pi_web_api_scouter_with_no_timeout():
         'runtime': 'legacy',
     }
     assert result == expected
+
+
+@mark.parametrize(
+    'frequency_alert, warns',
+    [
+        (None, False),
+        ('5m', False),
+        ('1m', False),  # the floor itself is allowed
+        ('60s', False),
+        ('30s', True),
+        ('5min', True),  # unparseable
+        (5, True),  # not a string: must warn, never raise
+    ],
+)
+def test_frequency_alert_warning(frequency_alert, warns):
+    assert (frequency_alert_warning(frequency_alert) is not None) is warns
+
+
+def test_pi_web_api_scouter_caps_api_timeout_at_frequency_only_when_larger():
+    config = {
+        'workflow_type': 'pi_web_api_scouter',
+        'schedule_name': 'pi_schedule',
+        'model_id': 'model_id',
+        'model': {'name': 'model_name'},
+        'frequency': '1m',
+        'read_tags': {'TI-101': {'webid': 'F1DP-X'}},
+        'pi_web_api_config': {
+            'endpoint': '/streamsets/recorded',
+            'max_count': 5,
+            'api_timeout': 10,
+        },
+    }
+
+    result = pi_web_api_scouter(config)
+
+    assert result['model_tags'] == {
+        'TI-101': {'webid': 'F1DP-X', 'aggr_func': 'lts', 'data_range': [-100, 100]}
+    }
+    assert result['pi_web_api_query'] == {
+        'endpoint': '/streamsets/recorded',
+        'period': '*-1d',
+        'max_count': 5,
+        'api_timeout': 10,
+    }
+
+
+def test_predictions_batch_copies_the_resolved_opc_output_config():
+    opc_output_config = {'plant': {'prediction_tags': {'ns=2;s=P1': {'data_type': 'int'}}}}
+    config = {
+        'workflow_type': 'predictions_batch',
+        'schedule_name': 'batch',
+        'model_id': 'model_id',
+        'model': {'name': 'model_name'},
+        'query': 'SELECT 1',
+        'write_tags': [{'tag_id': 77}],
+        'opc_output_config': opc_output_config,
+    }
+
+    result = predictions_batch(config)
+
+    assert result['opc_output_config'] == opc_output_config
+    assert 'write_tags' not in result
+
+
+def test_predictions_batch_without_write_tags_has_no_opc_output():
+    config = {
+        'workflow_type': 'predictions_batch',
+        'schedule_name': 'batch',
+        'model_id': 'model_id',
+        'model': {'name': 'model_name'},
+        'query': 'SELECT 1',
+    }
+
+    assert predictions_batch(config)['opc_output_config'] == {}

tests/orchestrator/utils/test_tag_database.py

@@ -0,0 +1,279 @@
+import pytest
+
+from orchestrator.utils.tag_database import (
+    WRITE_CATEGORIES,
+    client_certificate_id,
+    filter_by_tag_ids,
+    index_connector_tags,
+    index_connectors,
+    index_tag_documents,
+    referenced_tag_ids,
+    resolve_certificate,
+    resolve_tag_names,
+    resolve_write_tags,
+    tag_id_of,
+)
+from tests.orchestrator.utils.tag_database_fixtures import (
+    CERTIFICATE,
+    NO_SECURITY,
+    OPC_CONNECTORS,
+    TAGS_BY_ID,
+    batch_pipeline_doc,
+    certificate_doc,
+    scouter_pipeline_doc,
+    secure_connector_fields,
+    write_tag_doc,
+)
+
+
+def test_index_connector_tags():
+    tags = [
+        {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1', 'tag_name': 'a'},
+        # tag_id is int in connector_tags: a text id is no tag
+        {'tag_id': '8', 'connector_id': 3, 'address': 'ns=3;i=2', 'tag_name': 'b'},
+    ]
+
+    assert index_connector_tags(tags) == {7: tags[0]}
+
+
+def test_resolve_tag_names():
+    read_tags = [
+        {'tag_id': 7},
+        {'tag_id': 7, 'tag_name': 'own name'},
+        {'tag_id': '7'},
+        {'tag_id': 404},
+        {'tag_name': 'no tag_id'},
+    ]
+
+    assert resolve_tag_names(read_tags, TAGS_BY_ID) == [
+        {'tag_id': 7, 'tag_name': 'Pressure'},
+        {'tag_id': 7, 'tag_name': 'own name'},
+        {'tag_id': '7'},
+        {'tag_id': 404},
+        {'tag_name': 'no tag_id'},
+    ]
+
+
+def test_referenced_tag_ids():
+    """Int ids only, scouter pipelines only; text, empty or null tag_id ignored"""
+    pipelines = [
+        scouter_pipeline_doc(
+            'sin', 'm1', [{'tag_id': 7}, {'tag_id': '12'}, {'tag_id': ''}, {'tag_id': None}]
+        ),
+        scouter_pipeline_doc(
+            'cos', 'm2', [{'tag_id': 'abc'}, {'tag_name': 'a', 'frequency': 1000}]
+        ),
+        scouter_pipeline_doc('pred', 'm3', [{'tag_id': 99}], workflow_type='predictions_batch'),
+    ]
+
+    # Sorted, so the query is the same on every run
+    assert referenced_tag_ids(pipelines) == [7]
+
+
+def test_referenced_tag_ids_without_tag_ids():
+    pipeline = scouter_pipeline_doc('sin', 'm1', [{'tag_name': 'a', 'frequency': 1000}])
+    no_read_tags = scouter_pipeline_doc('cos', 'm2', [])
+    del no_read_tags['read_tags']
+
+    assert referenced_tag_ids([pipeline, no_read_tags]) == []
+
+
+def test_filter_by_tag_ids():
+    """A tag_id $match goes in front, on a new aggregation list"""
+    query = {'collection': 'connector_tags', 'aggregation': [{'$project': {'_id': 0}}]}
+
+    result = filter_by_tag_ids(query, [7, 8])
+
+    assert result == {
+        'collection': 'connector_tags',
+        'aggregation': [{'$match': {'tag_id': {'$in': [7, 8]}}}, {'$project': {'_id': 0}}],
+    }
+    assert result['aggregation'] is not query['aggregation']
+    assert query['aggregation'] == [{'$project': {'_id': 0}}]
+
+
+def test_index_skips_incomplete_documents():
+    """A malformed tag database document is left out instead of failing the run"""
+    tags = [
+        {'tag_id': 7, 'connector_id': 3, 'address': 'ns=3;i=1', 'tag_name': 'a'},
+        {'connector_id': 3, 'address': 'ns=3;i=2', 'tag_name': 'no id'},
+        {'tag_id': 9, 'connector_id': 3, 'address': '', 'tag_name': 'no address'},
+    ]
+    connector_list = [
+        {'connector_id': 3, 'name': 'plant', 'url': 'opc.tcp://p:4840', 'security': NO_SECURITY},
+        {'connector_id': 4, 'name': 'no url', 'security': NO_SECURITY},
+        {'name': 'no id', 'url': 'opc.tcp://x:4840', 'security': NO_SECURITY},
+        # security.mode and security.policy are the source of the OPC security (D4)
+        {'connector_id': 5, 'name': 'no security', 'url': 'opc.tcp://x:4840'},
+        {
+            'connector_id': 6,
+            'name': 'no mode',
+            'url': 'opc.tcp://x:4840',
+            'security': {'policy': 'None'},
+        },
+        {
+            'connector_id': 7,
+            'name': 'no policy',
+            'url': 'opc.tcp://x:4840',
+            'security': {'mode': 'None'},
+        },
+    ]
+
+    assert list(index_connector_tags(tags)) == [7]
+    assert list(index_connectors(connector_list)) == ['3']
+
+
+def test_tag_id_of_accepts_only_int():
+    """connector_tags stores tag_id as int: anything else is no reference"""
+    assert tag_id_of({'tag_id': 7}) == 7
+    assert tag_id_of({'tag_id': '7'}) is None
+    assert tag_id_of({'tag_id': 'abc'}) is None
+    assert tag_id_of({'tag_id': 7.0}) is None
+    assert tag_id_of({'tag_id': True}) is None
+    assert tag_id_of({'tag_id': ''}) is None
+    assert tag_id_of({'tag_id': None}) is None
+    assert tag_id_of({}) is None
+    assert tag_id_of('ns=2;s=raw') is None
+
+
+def test_referenced_tag_ids_includes_write_tags():
+    pipelines = [
+        scouter_pipeline_doc('sin', 'm1', [{'tag_id': 7}]),
+        batch_pipeline_doc('pred', [{'tag_id': 77}, {'tag_id': 78}, {'server_id': '1'}]),
+        batch_pipeline_doc('xgb', [{'tag_id': 79}], workflow_type='xgboost_predictions_batch'),
+        batch_pipeline_doc('retrain', [{'tag_id': 99}], workflow_type='minimal_retrain'),
+    ]
+
+    assert referenced_tag_ids(pipelines) == [7, 77, 78, 79]
+
+
+def test_index_connector_tags_filters_status_and_category():
+    tags = [
+        write_tag_doc(1, 'ns=2;s=R', category='Read'),
+        write_tag_doc(2, 'ns=2;s=W'),
+        write_tag_doc(3, 'ns=2;s=C', category='Prediction Confidence'),
+        write_tag_doc(4, 'ns=2;s=I', status='Inactive'),
+        # An override that projects neither status nor category
+        {'tag_id': 5, 'connector_id': 'plant', 'address': 'ns=2;s=N', 'tag_name': 'n'},
+    ]
+
+    assert list(index_connector_tags(tags)) == [1, 5]
+    assert list(index_connector_tags(tags, WRITE_CATEGORIES)) == [2, 3, 5]
+    # Every document, whatever its status or category
+    assert list(index_tag_documents(tags)) == [1, 2, 3, 4, 5]
+
+
+def test_index_connectors_leaves_out_inactive():
+    connectors = [
+        {**OPC_CONNECTORS['plant'], 'status': 'Active'},
+        {**OPC_CONNECTORS['plant'], 'connector_id': 'old', 'status': 'Inactive'},
+    ]
+
+    assert list(index_connectors(connectors)) == ['plant']
+
+
+def test_resolve_write_tags():
+    tags = index_connector_tags(
+        [
+            write_tag_doc(77, 'ns=2;s=P1'),
+            write_tag_doc(78, 'ns=2;s=C1', category='Prediction Confidence', data_type='double'),
+            write_tag_doc(79, 'ns=2;s=P2', connector_id='other', data_type='int'),
+        ],
+        WRITE_CATEGORIES,
+    )
+    connectors = {**OPC_CONNECTORS, 'other': {**OPC_CONNECTORS['plant'], 'connector_id': 'other'}}
+
+    output_config, issues = resolve_write_tags(
+        [{'tag_id': 77}, {'tag_id': 78}, {'tag_id': 79}], tags, connectors
+    )
+
+    assert output_config == {
+        'plant': {
+            'prediction_tags': {'ns=2;s=P1': {'data_type': 'float'}},
+            'confidence_tags': {'ns=2;s=C1': {'data_type': 'double'}},
+        },
+        'other': {'prediction_tags': {'ns=2;s=P2': {'data_type': 'int'}}},
+    }
+    assert not any(issues.values())
+
+
+def test_resolve_write_tags_issues():
+    tags = index_connector_tags(
+        [
+            write_tag_doc(1, 'ns=2;s=R', category='Read'),
+            write_tag_doc(2, 'ns=2;s=X', connector_id='gone'),
+            write_tag_doc(3, 'ns=2;s=B', data_type='Double'),
+            write_tag_doc(4, 'ns=2;s=I', status='Inactive'),
+            # No category: a write tag cannot tell prediction from confidence
+            {'tag_id': 5, 'connector_id': 'plant', 'address': 'ns=2;s=N', 'tag_name': 'n'},
+            write_tag_doc(6, 'ns=2;s=OK'),
+        ],
+        WRITE_CATEGORIES,
+    )
+    write_tags = [
+        {'tag_id': 1},
+        {'tag_id': 2},
+        {'tag_id': 3},
+        {'tag_id': 4},
+        {'tag_id': 5},
+        {'tag_id': 404},
+        # Every write tag is a tag_id
+        {'tag_name': 'L'},
+        {'tag_id': None},
+        # tag_id is int: a text id is invalid
+        {'tag_id': '6'},
+        'ns=2;s=raw',
+        {'tag_id': 6},
+    ]
+
+    output_config, issues = resolve_write_tags(write_tags, tags, OPC_CONNECTORS)
+
+    assert output_config == {'plant': {'prediction_tags': {'ns=2;s=OK': {'data_type': 'float'}}}}
+    assert issues['tags'] == {'1', '4', '5', '404'}
+    assert issues['connectors'] == {'gone'}
+    assert issues['invalid'] == {
+        '3',
+        "{'tag_name': 'L'}",
+        "{'tag_id': '6'}",
+        "{'tag_id': None}",
+        'ns=2;s=raw',
+    }
+
+
+def test_resolve_certificate():
+    """Only the fields the gateway needs: dates and other fields stay out of the slot"""
+    connector = {
+        **OPC_CONNECTORS['plant'],
+        **secure_connector_fields(
+            certificates=[certificate_doc(thumbprint_sha1='ab', not_after=None)]
+        ),
+    }
+
+    assert resolve_certificate(connector) == CERTIFICATE
+    assert client_certificate_id(connector) == 1
+
+
+@pytest.mark.parametrize(
+    'certificates',
+    [
+        pytest.param(None, id='no-lookup'),
+        pytest.param([], id='not-found'),
+        pytest.param([certificate_doc(application_uri=None)], id='no-application-uri'),
+        pytest.param([{**certificate_doc(), 'cert': None}], id='no-cert'),
+        pytest.param(
+            [{**certificate_doc(), 'private_key': {'object_name': 'k'}}], id='key-no-bucket'
+        ),
+        pytest.param(['not a document'], id='not-a-document'),
+    ],
+)
+def test_resolve_certificate_without_a_usable_certificate(certificates):
+    connector = {**OPC_CONNECTORS['plant'], **secure_connector_fields()}
+    connector['certificates'] = certificates
+
+    assert resolve_certificate(connector) is None
+
+
+def test_client_certificate_id_without_one():
+    assert client_certificate_id(OPC_CONNECTORS['plant']) is None
+    assert client_certificate_id({'connector_id': 'x'}) is None
+    assert client_certificate_id({'security': None}) is None

tests/orchestrator/worker/test_worker.py

@@ -3,7 +3,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
 
 from pytest import mark
 
-from orchestrator.worker.worker import main
+from orchestrator.worker.worker import main, start_prometheus_server
 from orchestrator.workflows.alerts import Alerts
 from orchestrator.workflows.orchestrator import Orchestrator
 from orchestrator.workflows.reports import Reports
@@ -47,3 +47,43 @@ async def test_main_starts_three_workers_for_expected_workflows(
     assert main_workflows == [Orchestrator, Alerts, Reports]
 
     assert worker_mock.run.call_count == 3
+
+
+@mark.asyncio
+@patch('orchestrator.worker.worker.Runtime')
+@patch('orchestrator.worker.worker.sys')
+@patch('orchestrator.worker.worker.start_http_server')
+@patch('orchestrator.worker.worker.NotificationHandler')
+@patch('orchestrator.worker.worker.Activities')
+@patch('orchestrator.worker.worker.prepare_worker')
+@patch('orchestrator.worker.worker.client.Client.connect', new_callable=AsyncMock)
+async def test_main_exits_with_error_when_a_worker_fails(
+    _connect_mock,
+    prepare_worker_mock,
+    activities_mock,
+    notification_handler_mock,
+    _start_http_server,
+    sys_mock,
+    _runtime_mock,
+):
+    """A worker failure shuts everything down and exits with code 1."""
+    activities_instance = activities_mock.return_value
+    activities_instance.connect_to_temporal = AsyncMock()
+
+    worker_mock = MagicMock()
+    worker_mock.run = AsyncMock(side_effect=RuntimeError('worker crashed'))
+    prepare_worker_mock.return_value = worker_mock
+
+    await main()
+
+    notification_handler_mock.return_value.shutdown.assert_called_once()
+    activities_instance.shutdown.assert_called_once()
+    sys_mock.exit.assert_called_once_with(1)
+
+
+@patch('orchestrator.worker.worker.os._exit')
+@patch('orchestrator.worker.worker.start_http_server', side_effect=OSError('port in use'))
+def test_start_prometheus_server_exits_on_failure(_start_http_server, exit_mock):
+    start_prometheus_server()
+
+    exit_mock.assert_called_once_with(1)

tests/orchestrator/workflows/subworkflows/test_load_notification_package.py

@@ -237,6 +237,65 @@ def test_compute_query_time_filter_model_ignored_by_only_one_group_not_excluded(
     assert 'model_id' not in result
 
 
+def test_compute_query_time_filter_model_ids_union_when_all_groups_restrict(
+    load_notification_package,
+):
+    # SIENTIAPDE-2149: every active group sets a model_ids allow-list -> the
+    # union applies at query time, mirroring how `levels` already behaves.
+    sending_configs = [
+        {'group_name': 'group_a', 'model_ids': ['model_a']},
+        {'group_name': 'group_b', 'model_ids': ['model_b']},
+    ]
+    default_filter = {}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert result == {'model_id': {'$in': ['model_a', 'model_b']}}
+
+
+def test_compute_query_time_filter_model_ids_no_restriction_when_any_group_missing(
+    load_notification_package,
+):
+    # One active group has no model_ids restriction -> it wants every model,
+    # so the query can't restrict to an allow-list at all.
+    sending_configs = [
+        {'group_name': 'group_a', 'model_ids': ['model_a']},
+        {'group_name': 'group_b'},
+    ]
+    default_filter = {}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert 'model_id' not in result
+
+
+def test_compute_query_time_filter_model_ids_union_and_ignore_models_id_intersection_together(
+    load_notification_package,
+):
+    # SIENTIAPDE-2149: the allow-list union ($in) and the ignore-list
+    # intersection ($nin) are independent axes and can both narrow the same
+    # model_id field at once.
+    sending_configs = [
+        {
+            'group_name': 'group_a',
+            'model_ids': ['model_a', 'model_b'],
+            'ignore_models_id': ['model_a'],
+        },
+        {
+            'group_name': 'group_b',
+            'model_ids': ['model_b', 'model_c'],
+            'ignore_models_id': ['model_a'],
+        },
+    ]
+    default_filter = {}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert result == {
+        'model_id': {'$in': ['model_a', 'model_b', 'model_c'], '$nin': ['model_a']},
+    }
+
+
 def test_compute_query_time_filter_union_levels_and_intersection_models(
     load_notification_package,
 ):

tests/orchestrator/workflows/test_orchestrator.py

@@ -3,7 +3,11 @@ from unittest.mock import ANY, AsyncMock, call, patch
 from pytest import fixture, mark
 
 from orchestrator.activities.activities import Activities
-from orchestrator.workflows.orchestrator import Orchestrator
+from orchestrator.workflows.orchestrator import (
+    DEFAULT_CONNECTOR_TAGS_QUERY,
+    DEFAULT_CONNECTORS_QUERY,
+    Orchestrator,
+)
 
 
 @fixture
@@ -22,11 +26,11 @@ metadata = {
 
 
 @mark.asyncio
+@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[7, '7'])
 @patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
-async def test_run(workflow_mock, orchestrator):
+async def test_run(workflow_mock, referenced_tag_ids_mock, orchestrator):
     input_data = {
         'pipelines_query': 'SELECT * FROM bucket',
-        'opc_servers_query': 'SELECT * FROM servers',
         'schedule_name': 'test-schedule-name',
     }
 
@@ -47,16 +51,13 @@ async def test_run(workflow_mock, orchestrator):
         ]
     )
 
-    workflow_mock.start_local_activity_method.assert_has_calls(
-        [
-            call(
-                Activities.find_documents_in_mongodb,
-                {**metadata, 'query': input_data['opc_servers_query']},
-                retry_policy=ANY,
-                start_to_close_timeout=ANY,
-            )
-        ]
-    )
+    # connectors is the only source of the OPC servers: opc-servers is never loaded
+    assert not [
+        c
+        for c in workflow_mock.start_local_activity_method.call_args_list
+        if c.args[0] is Activities.find_documents_in_mongodb
+        and c.args[1]['query'] != {'collection': 'orchestrated_schedules'}
+    ]
 
     workflow_mock.start_local_activity_method.assert_has_calls(
         [
@@ -109,11 +110,54 @@ async def test_run(workflow_mock, orchestrator):
         ]
     )
 
+    referenced_tag_ids_mock.assert_called_once_with(
+        workflow_mock.start_local_activity_method.return_value
+    )
+
+    workflow_mock.start_local_activity_method.assert_has_calls(
+        [
+            call(
+                Activities.aggregate_documents_in_mongodb,
+                {**metadata, 'query': DEFAULT_CONNECTORS_QUERY},
+                retry_policy=ANY,
+                start_to_close_timeout=ANY,
+            ),
+            call(
+                Activities.aggregate_documents_in_mongodb,
+                {
+                    **metadata,
+                    'query': {
+                        **DEFAULT_CONNECTOR_TAGS_QUERY,
+                        'aggregation': [
+                            {'$match': {'tag_id': {'$in': [7, '7']}}},
+                            *DEFAULT_CONNECTOR_TAGS_QUERY['aggregation'],
+                        ],
+                    },
+                },
+                retry_policy=ANY,
+                start_to_close_timeout=ANY,
+            ),
+        ]
+    )
+
+    # The activity appends a stage to the aggregation it gets: never the module default
+    tag_database_calls = _tag_database_calls(workflow_mock)
+    assert len(tag_database_calls) == 2
+    for c in tag_database_calls:
+        aggregation = c.args[1]['query']['aggregation']
+        assert aggregation is not DEFAULT_CONNECTORS_QUERY['aggregation']
+        assert aggregation is not DEFAULT_CONNECTOR_TAGS_QUERY['aggregation']
+
     workflow_mock.start_local_activity_method.assert_has_calls(
         [
             call(
                 Activities.process_schedules,
-                {**metadata, 'pipelines': workflow_mock.start_local_activity_method.return_value},
+                {
+                    **metadata,
+                    'pipelines': workflow_mock.start_local_activity_method.return_value,
+                    'connectors': workflow_mock.start_local_activity_method.return_value,
+                    'connector_tags': workflow_mock.start_local_activity_method.return_value,
+                },
                 retry_policy=ANY,
                 start_to_close_timeout=ANY,
             )
@@ -126,7 +170,8 @@ async def test_run(workflow_mock, orchestrator):
                 Activities.process_slots,
                 {
                     **metadata,
-                    'opc_servers': workflow_mock.start_local_activity_method.return_value,
+                    'connectors': workflow_mock.start_local_activity_method.return_value,
+                    'connector_tags': workflow_mock.start_local_activity_method.return_value,
                     'active_ingestors': workflow_mock.start_local_activity_method.return_value,
                     'pipelines': workflow_mock.start_local_activity_method.return_value,
                 },
@@ -346,3 +391,136 @@ async def test_run(workflow_mock, orchestrator):
             )
         ]
     )
+
+
+def _tag_database_calls(workflow_mock):
+    return [
+        c
+        for c in workflow_mock.start_local_activity_method.call_args_list
+        if c.args[0] is Activities.aggregate_documents_in_mongodb
+        and isinstance(c.args[1]['query'], dict)
+        and c.args[1]['query'].get('collection') in ('connectors', 'connector_tags')
+    ]
+
+
+def _process_slots_input(workflow_mock):
+    return next(
+        c.args[1]
+        for c in workflow_mock.start_local_activity_method.call_args_list
+        if c.args[0] is Activities.process_slots
+    )
+
+
+@mark.asyncio
+@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[7])
+@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
+async def test_run_with_tag_database_queries_from_input(workflow_mock, _ids, orchestrator):
+    """The input overrides the default tag database aggregations (SIENTIAPDE-1974, D9)"""
+    connectors_query = {'collection': 'connectors', 'aggregation': [{'$match': {'type': 'opc_ua'}}]}
+    connector_tags_query = {
+        'collection': 'connector_tags',
+        'aggregation': [{'$match': {'category': 'Read'}}],
+    }
+    input_data = {
+        'pipelines_query': 'SELECT * FROM bucket',
+        'connectors_query': connectors_query,
+        'connector_tags_query': connector_tags_query,
+        'schedule_name': 'test-schedule-name',
+    }
+
+    await orchestrator.run(input_data)
+
+    queries = [c.args[1]['query'] for c in _tag_database_calls(workflow_mock)]
+    assert queries == [
+        connectors_query,
+        {
+            'collection': 'connector_tags',
+            'aggregation': [
+                {'$match': {'tag_id': {'$in': [7]}}},
+                {'$match': {'category': 'Read'}},
+            ],
+        },
+    ]
+
+
+@mark.asyncio
+@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[])
+@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
+async def test_run_without_tag_ids_skips_tag_database(workflow_mock, _ids, orchestrator):
+    """No read tag with tag_id: the tag database is not queried at all (P4)"""
+    input_data = {
+        'pipelines_query': 'SELECT * FROM bucket',
+        'schedule_name': 'test-schedule-name',
+    }
+
+    await orchestrator.run(input_data)
+
+    assert _tag_database_calls(workflow_mock) == []
+    slots_input = _process_slots_input(workflow_mock)
+    assert slots_input['connectors'] == []
+    assert slots_input['connector_tags'] == []
+
+
+def _process_schedules_input(workflow_mock):
+    return next(
+        c.args[1]
+        for c in workflow_mock.start_local_activity_method.call_args_list
+        if c.args[0] is Activities.process_schedules
+    )
+
+
+def test_default_connector_tags_query_loads_every_category_and_status():
+    """
+    The status / category filters are applied in the code. updated_at is not loaded: a
+    tag change reaches the schedules through their config_hash.
+    """
+    stages = DEFAULT_CONNECTOR_TAGS_QUERY['aggregation']
+    assert all('$match' not in stage for stage in stages)
+    projected = stages[-1]['$project']
+    for field in ('category', 'data_type', 'status'):
+        assert projected[field] == 1
+    assert 'updated_at' not in projected
+
+
+@mark.asyncio
+@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[77])
+@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
+async def test_run_hands_the_tag_database_to_process_schedules(workflow_mock, _ids, orchestrator):
+    """The write tags of process_schedules need the connectors as well as the tags"""
+    input_data = {
+        'pipelines_query': 'SELECT * FROM bucket',
+        'schedule_name': 'test-schedule-name',
+    }
+
+    await orchestrator.run(input_data)
+
+    loaded = workflow_mock.start_local_activity_method.return_value
+    schedules_input = _process_schedules_input(workflow_mock)
+    assert schedules_input['connectors'] is loaded
+    assert schedules_input['connector_tags'] is loaded
+    assert 'pi_connectors' not in schedules_input
+
+
+@mark.asyncio
+@patch('orchestrator.workflows.orchestrator.referenced_tag_ids', return_value=[])
+@patch('orchestrator.workflows.orchestrator.workflow', new_callable=AsyncMock)
+async def test_run_without_changes_skips_reports_and_timestamps(workflow_mock, _ids, orchestrator):
+    """Nothing created, updated or deleted: no report and no pipeline timestamp activity"""
+    workflow_mock.start_activity_method = AsyncMock(return_value=[])
+    input_data = {
+        'pipelines_query': 'SELECT * FROM bucket',
+        'schedule_name': 'test-schedule-name',
+    }
+
+    await orchestrator.run(input_data)
+
+    started = [c.args[0] for c in workflow_mock.start_activity_method.call_args_list]
+    assert started == [
+        Activities.normalize_schedules,
+        Activities.create_collection_with_ttl_index,
+        Activities.delete_slots,
+        Activities.update_slots,
+        Activities.delete_schedules,
+        Activities.create_schedules,
+        Activities.update_schedules,
+    ]

tests/orchestrator/workflows/test_orchestrator_replay.py

@@ -0,0 +1,27 @@
+"""
+Replay a real Orchestrator history against the current workflow code.
+
+orchestrator_tag_database.json was recorded on the e2e stack (real Temporal, Mongo and
+Redis), with the connectors / connector_tags loads of read and write tags by tag_id.
+
+A NondeterminismError here means a change to the workflow breaks runs in flight.
+"""
+
+from pathlib import Path
+
+import pytest
+from temporalio.client import WorkflowHistory
+from temporalio.worker import Replayer
+
+from orchestrator.workflows.orchestrator import Orchestrator
+
+HISTORIES = Path(__file__).parent / 'histories'
+
+
+@pytest.mark.asyncio
+async def test_replay_recorded_history():
+    history = WorkflowHistory.from_json(
+        'orchestrator-history', (HISTORIES / 'orchestrator_tag_database.json').read_text()
+    )
+
+    await Replayer(workflows=[Orchestrator]).replay_workflow(history)