chore: bump version to 0.8.1

fix: bump sientia_do to 1.12.5 (psycopg2 driver pin)

.github/workflows/deploy.yml

@@ -20,7 +20,7 @@ jobs:
       branch_name: ${{ github.event.pull_request.head.ref }}
       project_name: 'orchestrator'
       chart_name: 'orchestrator-worker'
-      env_list_path: '.env'
+      env_list_path: '[".env"]'
       update_version_in: '["pyproject"]'
       create_tarball: ${{ github.event_name == 'pull_request' }}
       mirror: true

CLAUDE.md

@@ -45,7 +45,7 @@ CI quality gate (`.github/workflows/quality-gate.yml`) calls the shared reusable
 
 ### Layering
 - **`orchestrator/worker/worker.py`** — entrypoint (`python -m orchestrator.worker.worker`). Builds a single `Activities` instance, connects to Temporal, starts Prometheus (`HTTP_METRICS_PORT`) and Temporal SDK metrics (`HTTP_SDK_METRICS_PORT`) servers, then runs three `Worker`s concurrently via `asyncio.gather`, each on its own task queue: `orchestrator-queue`, `alerts-queue`, `reports-queue`. `prepare_worker.py` derives the queue name from the workflow class name (`camel_to_snake`) and reads per-queue concurrency/poller tuning from env vars prefixed with the workflow name (e.g. `ORCHESTRATOR_MAX_CONCURRENT_ACTIVITIES`).
-- **`orchestrator/workflows/`** — Temporal workflow definitions: `orchestrator.py` (pipeline/slot/schedule orchestration), `alerts.py` (real-time ERROR notifications), `reports.py` (scheduled all-level reports). `alerts.py` and `reports.py` both delegate to the same two subworkflows in `workflows/subworkflows/`: `load_notification_package.py` (load notifications + receiver configs from Mongo, incremental via Redis timestamp) and `process_notifications.py` (build HTML email, send, log to Postgres). The difference between alerts and reports is which filter activity they call (`filter_notification_alerts` vs `filter_notification_reports`) and their base data filter (`level: ERROR` vs all levels).
+- **`orchestrator/workflows/`** — Temporal workflow definitions: `orchestrator.py` (pipeline/slot/schedule orchestration), `alerts.py` (real-time ERROR notifications), `reports.py` (scheduled all-level reports). `alerts.py` and `reports.py` both delegate to the same two subworkflows in `workflows/subworkflows/`: `load_notification_package.py` (load notifications + receiver configs from Mongo, incremental via Redis timestamp) and `process_notifications.py` (build HTML email, send, log to Postgres). The difference between alerts and reports is which filter activity they call (`filter_notification_alerts` vs `filter_notification_reports`) and their base data filter (`level: ERROR` by default vs all levels) — `load_notification_package.py`'s `_compute_query_time_filter` can widen either beyond that default to the union of active receiver groups' own `levels` (SIENTIAPDE-2011); the ERROR-only default only holds when no active group narrows further. Receiver groups also support a `model_ids` allow-list (SIENTIAPDE-2149), independent of the pre-existing `ignore_models_id` deny-list — both apply together (AND) when a group configures both, and `model_ids` follows the same union-at-query-time / precise-per-group-slice split as `levels`. `load_latest_data`'s Mongo query is capped at `NOTIFICATION_QUEUE_BATCH_SIZE` documents per run (default `0`, meaning unlimited — unchanged from pre-SIENTIAPDE-2013 behavior unless a deployment opts in), sorted by `timestamp` ascending so a capped run always returns the oldest pending batch first (SIENTIAPDE-2013) — a hard cap with no drain loop, since the incremental Redis cursor picks up any remainder on the next run.
 - **`orchestrator/activities/activities.py`** — `Activities` is a single class combining every activity mixin via multiple inheritance: `TemporalManager, SlotManager, Formatters, MongoDB, Email, Postgres` (Postgres comes from `sientia_do.temporal.activities.postgres`, the external `sientia-dataops-library` dependency). One instance is constructed in `worker.py` and its bound methods are handed to each `Worker` as the activity list — activities are plain methods on this shared object, not separate classes. Each mixin's `__init__` takes its own config dict plus shared `logger`, `notification_handler`, `metrics_controller`. `Activities.shutdown()` closes every mixin's connections.
 - **`orchestrator/utils/`** — `connectors_config.py` builds each config dict from env vars (Temporal/Redis/MongoDB/Postgres/Email); `email_builder.py` renders Jinja2 templates (`utils/templates/email_template.html`, `general_template.html`); `orchestrator_functions.py` transforms pipeline documents into pipeline-type-specific configs (`scouter`, `pi_web_api_scouter`, `predictions_batch`, `minimal_retrain`, `drift`, `simple_metrics`); `converters.py` handles Temporal schedule frequency parsing.
 

README.md

@@ -612,6 +612,37 @@ Temporal input configuration sample:
 }
 ```
 
+#### Simple metrics: parallel windows & adaptive frequency
+
+One `pipelines` doc per window, all with the same `model_id`. Name them `<model>-simple_metrics-<interval_minutes>m`; each doc becomes its own Temporal schedule. Thresholds are shared through `model.model_config`. The adaptive state is per doc.
+
+```json
+{
+    "schedule_name": "model-x-simple_metrics-60m",
+    "workflow_type": "simple_metrics",
+    "model_id": "model-x",
+    "active": true,
+    "interval_minutes": 60,
+    "frequency": "1h",
+    "frequency_alert": "5m"
+}
+```
+
+| Field | Meaning | Default |
+|---|---|---|
+| `frequency_alert` | Frequency while escalated, same format as `frequency`. Absent = adaptive mode off. Below `1m` (or unparseable), the orchestrator sends a WARNING each time it creates or updates the schedule. A value below `1m` is kept; an unparseable value is ignored (the schedule keeps `frequency`) | — |
+| `adaptive_state` | `{escalated, healthy_streak}`, written **only** by laborious (with `updated_at` as a BSON datetime when `escalated` flips) | `{false, 0}` |
+| `model.model_config.simple_metrics_thresholds` | `{<metric>_max, <metric>_min}` shared by all windows | — |
+| `model.model_config.simple_metrics_min_samples` | Windows with fewer rows neither alert nor escalate, and count toward recovery | — |
+
+Escalation, recovery and the per-model alert are implemented in laborious (SIENTIAPDE-2189, SIENTIAPDE-2191); the orchestrator only picks `frequency_alert` while `adaptive_state.escalated` is true.
+
+Rules:
+- **Escalation:** laborious escalates on the first breach.
+- **Recovery:** after `min(12, ceil(interval_minutes / frequency_alert))` consecutive runs without one. Empty windows count as runs without a breach.
+- **Alerts** are per model (trigger `<model_id>-simple_metrics`), so several windows seeing the same breach produce one alert.
+- **Delay:** the schedule switches on the orchestrator tick after laborious flips the state, at most one tick later.
+
 #### Alerts Workflow
 
 ```json

e2e/conftest.py

@@ -330,6 +330,7 @@ async def test_activities(
             'connection_string': f'mongodb://localhost:{mongo_port}',
             'database_name': E2E_DATABASE,
             'ttl_index_seconds': 3600,
+            'notification_queue_batch_size': 0,
         },
         email_config={
             'sender_email': 'e2e@example.com',

e2e/helpers.py

@@ -7,6 +7,7 @@ import json
 import re
 import uuid
 from datetime import UTC, datetime, timedelta
+from email.message import EmailMessage, Message
 from pathlib import Path
 from typing import Any
 
@@ -29,7 +30,6 @@ MONGO_COLLECTIONS = (
 )
 
 DATETIME_FORMAT_MS_WITH_TZ = '%Y-%m-%d %H:%M:%S.%f%z'
-DATETIME_FORMAT_WITH_TZ = '%Y-%m-%d %H:%M:%S%z'
 
 _TIMESTAMP_MARKER_PATTERN = re.compile(r'^@now(?:([+-])(\d+)([smhd]))?$')
 _TIMESTAMP_UNIT_TO_KWARG = {'s': 'seconds', 'm': 'minutes', 'h': 'hours', 'd': 'days'}
@@ -222,17 +222,19 @@ def seed_last_timestamp(redis_client: Redis, mail_type: str, value: str) -> None
     Set notification_last_timestamp for a mail type, JSON-encoded.
 
     Values must be JSON-encoded so ``redis_repository.get`` (which calls
-    ``json.loads`` on the raw payload) can deserialize them. The value
-    must follow the exact format ``sientia_do.notifications.models.Notification``
-    writes into ``notification_queue.timestamp``: ``DATETIME_FORMAT_WITH_TZ``
-    (e.g. ``"2026-05-22 16:47:02+0000"``) — no microseconds and no colon in
-    the timezone offset.
+    ``json.loads`` on the raw payload) can deserialize them. The value must
+    follow the exact format ``put_last_data_timestamp``
+    (``orchestrator/activities/slot_manager.py``) writes into Redis today:
+    ``DATETIME_FORMAT_MS_WITH_TZ`` (e.g. ``"2026-05-22 16:47:02.000000+0000"``)
+    — microseconds included, no colon in the timezone offset. ``load_latest_data``
+    parses this same format back into a ``datetime`` for the ``$gt`` comparison,
+    so the two sides must agree exactly.
 
     Args:
         redis_client: Redis client connected to the test instance.
         mail_type: Mail type identifier (e.g. ``"Alerts"``, ``"Reports"``).
-        value: Timestamp string in ``DATETIME_FORMAT_WITH_TZ``
-            (e.g. ``"2024-06-01 10:30:00+0000"``).
+        value: Timestamp string in ``DATETIME_FORMAT_MS_WITH_TZ``
+            (e.g. ``"2024-06-01 10:30:00.000000+0000"``).
     """
     redis_client.set(f'notification_last_timestamp:{mail_type}', json.dumps(value))
 
@@ -295,34 +297,35 @@ def default_notification(
     *,
     notification_id: str,
     level: str = 'ERROR',
-    timestamp: str | None = None,
+    timestamp: datetime | None = None,
     model_name: str = 'model-a',
     model_id: str = '1',
 ) -> dict[str, Any]:
     """
     Build a minimal notification_queue document mirroring production layout.
 
-    The ``timestamp`` field is stored as a string in ``DATETIME_FORMAT_WITH_TZ``
-    because that is exactly what ``sientia_do.notifications.models.Notification``
-    writes into ``notification_queue`` in production (``now().strftime(
-    DATETIME_FORMAT_WITH_TZ)``). Tests intentionally use this same format so we
+    The ``timestamp`` field is stored as a native, timezone-aware ``datetime``
+    (BSON Date) because that is exactly what
+    ``sientia_do.notifications.models.Notification`` writes into
+    ``notification_queue`` in production since SIENTIAPDE-2007
+    (``self.timestamp = now()`` — a real ``datetime``, no more string
+    formatting). Tests intentionally use this same representation so we
     surface, rather than hide, real production behavior in downstream
-    activities.
+    activities (e.g. ``put_last_data_timestamp``'s ``DataFrame(...)['timestamp']
+    .max().strftime(...)``, which requires a real ``datetime``, not a string).
 
     Args:
         notification_id: Unique identifier for the notification.
         level: Notification level (e.g. ``"ERROR"``, ``"WARNING"``).
-        timestamp: Optional production-format timestamp string. ``None`` falls
-            back to a fixed sample value.
+        timestamp: Optional timezone-aware datetime. ``None`` falls back to a
+            fixed sample value.
         model_name: Model name attached to the notification.
         model_id: Model id attached to the notification.
 
     Return:
         dict[str, Any]: A notification document ready for insertion.
     """
-    ts = timestamp if timestamp is not None else datetime(
-        2024, 6, 1, 12, 0, 0, tzinfo=UTC
-    ).strftime(DATETIME_FORMAT_WITH_TZ)
+    ts = timestamp if timestamp is not None else datetime(2024, 6, 1, 12, 0, 0, tzinfo=UTC)
     return {
         'notification_id': notification_id,
         'level': level,
@@ -343,7 +346,8 @@ def default_receiver_group(
     members: list[str] | None = None,
     levels: list[str] | None = None,
     contents: list[str] | None = None,
-    ignore_models: list[str] | None = None,
+    ignore_models_id: list[str] | None = None,
+    model_ids: list[str] | None = None,
 ) -> dict[str, Any]:
     """Minimal active receiver_groups document."""
     return {
@@ -352,5 +356,43 @@ def default_receiver_group(
         'members': members or ['admin@example.com'],
         'levels': levels or ['ERROR', 'WARNING', 'INFO'],
         'contents': contents or ['core_alerts', 'persistent_alerts', 'reports'],
-        'ignore_models': ignore_models or [],
+        'ignore_models_id': ignore_models_id or [],
+        'model_ids': model_ids or [],
     }
+
+
+def extract_html_body(message: EmailMessage | Message) -> str:
+    """
+    Return the text/html portion of an email message as a decoded string.
+
+    Walks every part looking for the first text/html payload, decoding it
+    according to the part's transfer encoding and charset. Falls back to
+    the message's own ``get_content``/raw payload when no HTML part is
+    present so callers can still inspect plain-text reports. Shared between
+    the Alerts and Reports e2e suites, both of which send mail through the
+    same ``build_email_html``/``send_email`` activities.
+
+    Args:
+        message: Captured email message returned by the test SMTP server.
+
+    Return:
+        str: HTML body content, or an empty string when nothing decodable
+            is found.
+    """
+    if message.is_multipart():
+        for part in message.walk():
+            if part.get_content_type() != 'text/html':
+                continue
+            payload = part.get_payload(decode=True)
+            if payload is None:
+                continue
+            charset = part.get_content_charset() or 'utf-8'
+            return payload.decode(charset, errors='replace')
+    payload = message.get_payload(decode=True)
+    if payload is not None:
+        charset = message.get_content_charset() or 'utf-8'
+        return payload.decode(charset, errors='replace')
+    try:
+        return message.get_content()
+    except (AttributeError, KeyError):
+        return str(message.get_payload() or '')

e2e/scenario_inputs/orchestrator_simple_metrics_windows.json

@@ -0,0 +1,45 @@
+{
+  "workflow_input": {
+    "schedule_name": "orchestrator-e2e-sm-windows",
+    "pipelines_query": {
+      "collection": "pipelines",
+      "aggregation": [{"$match": {"active": true}}]
+    },
+    "opc_servers_query": {
+      "collection": "opc_servers",
+      "filters": {"active": true}
+    }
+  },
+  "pipelines": [
+    {
+      "schedule_name": "model-sm-simple_metrics-60m",
+      "workflow_type": "simple_metrics",
+      "model_id": "model-sm",
+      "model": {
+        "name": "Model model-sm",
+        "model_config": {"target": "y", "simple_metrics_thresholds": {"rmse_max": 1.0}}
+      },
+      "active": true,
+      "updated_at": "@now",
+      "frequency": "1h",
+      "frequency_alert": "5m",
+      "interval_minutes": 60
+    },
+    {
+      "schedule_name": "model-sm-simple_metrics-1440m",
+      "workflow_type": "simple_metrics",
+      "model_id": "model-sm",
+      "model": {
+        "name": "Model model-sm",
+        "model_config": {"target": "y", "simple_metrics_thresholds": {"rmse_max": 1.0}}
+      },
+      "active": true,
+      "updated_at": "@now",
+      "frequency": "1d",
+      "frequency_alert": "1h",
+      "interval_minutes": 1440
+    }
+  ],
+  "opc_servers": [],
+  "active_ingestors": ["heartbeat:ingestor:1"]
+}

e2e/scenarios.md

@@ -6,6 +6,11 @@ Functional reference for orchestrator E2E scenarios. Tests live under `e2e/`, us
 pytest e2e/ --override-ini testpaths=e2e -m e2e
 ```
 
+**Every section below maps 1:1 to an actual test function** (named in each section, or as a bullet
+under a `Source:` heading) — this file describes what the suite verifies today, not planned or
+aspirational coverage. Adding a scenario here without a matching test (or vice versa) is a doc bug;
+fix the mismatch instead of leaving it for later.
+
 ## Execution context
 
 - MongoDB, Redis, PostgreSQL: testcontainers (session-scoped).
@@ -21,36 +26,36 @@ Source: `e2e/test_orchestrator_main_workflow.py`
 
 ### 1.1.1 Happy path
 
-Pipelines in Mongo → schedules created in correct namespace/task queue, Redis slots written, `orchestrated_schedules` updated.
-
-### 1.2.1 No-op tick
-
-Mongo, Redis, and Temporal already match desired state → no new schedules or slot writes.
+Pipelines in Mongo → Temporal schedule created in the `laborious` namespace, `orchestrated_schedules` entries written.
 
 ### 1.3.1 Create-only
 
-New pipeline only → schedules created, timestamps inserted.
-
-### 1.3.2 Update-only
-
-Existing pipeline with newer `updated_at` → schedule updated in Temporal.
+New pipeline only → Temporal schedule created for it.
 
 ### 1.3.3 Delete-only
 
-Pipeline removed from Mongo → schedule deleted from Temporal.
-
-### 1.4.1 Conflict ordering
-
-Pipeline update and slot delete on same OPC server → slot insert before delete (production ordering).
+Pipeline removed from Mongo → previously-created Temporal schedule deleted.
 
 ### 1.5.1 Empty pipelines
 
-No active pipelines → orphan schedules removed, no new orchestration writes.
+No active pipelines → no `orchestrated_schedules` documents written, even with a pre-existing
+unrelated Temporal schedule present.
 
 ### 1.6.1 TTL index bootstrap
 
 First run creates TTL index on notification collection used by scouter pipelines.
 
+### 1.7.1 Simple metrics parallel windows
+
+Two `simple_metrics` pipelines for the same `model_id` (60 min and 1440 min windows) → one
+Temporal schedule per window, each with its own `frequency` as interval.
+
+### 1.7.2 Adaptive frequency escalation/recovery round trip
+
+Laborious-style write (`adaptive_state.escalated=true` + BSON-datetime `updated_at`) on one
+window → next run switches only that schedule to `frequency_alert`; the recovery write switches it
+back to `frequency`.
+
 ---
 
 ## 2. Alerts workflow
@@ -65,13 +70,20 @@ ERROR notification → one SMTP message, one `log_report` row, Redis cache key.
 
 Second run with same data and cache seeded → no extra email or log row.
 
-### A.1.3 Persistent escalation
-
-Alert past `notification_ttl` with cache cleared → new email sent.
-
 ### A.2.1 Group filtering
 
-Receiver group `levels` / `ignore_models` honored.
+Receiver group `levels` / `ignore_models_id` honored, both halves now covered:
+the query-time union/intersection across active groups (SIENTIAPDE-2011, see
+`LoadNotificationPackage` below) and the per-group precise slice
+(SIENTIAPDE-2012) — a group only receives notifications matching its own
+`levels`/`ignore_models_id`, even when the query-time filter widened the fetch
+to satisfy another active group's broader configuration.
+
+### A.2.2 Model_id allow-list
+
+Receiver group `model_ids` (SIENTIAPDE-2149) honored: a group configuring it only
+receives notifications from those models, even when another active group with
+no such restriction widens the query-time fetch to include every model.
 
 ### A.3.1 Empty queue
 
@@ -103,7 +115,11 @@ No notifications → no SMTP, no Postgres row.
 
 - No prior Redis timestamp → all matching notifications returned, max timestamp stored.
 - Prior timestamp → only newer notifications returned.
+- Query-time filter (SIENTIAPDE-2011) → union of active groups' `levels`, intersection of their `ignore_models_id`; a model ignored by only one of several active groups is not excluded from the fetch.
+- Model_id allow-list union at query time (SIENTIAPDE-2149) → when every active group sets `model_ids`, the fetch is restricted to the union of every group's allow-list; a model outside that union is never fetched at all, even for a group that would otherwise want it via a different axis.
 - Empty Mongo → no Redis timestamp write.
+- Batch size cap (SIENTIAPDE-2013) → with 5 pending notifications and a batch size of 2, each run returns exactly the next-oldest 2 (sorted ascending by timestamp), and the Redis cursor advances only to what was actually returned, so three successive runs drain the backlog in order instead of skipping anything.
+- Batch size 0 means unlimited (SIENTIAPDE-2013) → the configured default (`0`) does not cap the query; 12 pending notifications all come back in one run, verified against a real MongoDB instance rather than assumed from pymongo's documented convention.
 
 ### ProcessNotifications — `e2e/test_subworkflow_process_notifications.py`
 

e2e/test_alerts_main_workflow.py

@@ -12,6 +12,7 @@ from e2e.helpers import (
     count_log_report_rows,
     default_notification,
     default_receiver_group,
+    extract_html_body,
     fetch_log_report,
     load_scenario_input,
     make_workflow_id,
@@ -92,6 +93,162 @@ async def test_scenario_a_1_2_duplicate_suppressed(
     assert count_log_report_rows(postgres_engine, 'Alerts') == 0
 
 
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_a_2_1_per_group_levels_and_ignore_models_id(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    smtp_server,
+):
+    """
+    A.2.1 (SIENTIAPDE-2012): each receiver group gets only its own precise
+    slice of notifications, on top of SIENTIAPDE-2011's query-time fetch.
+
+    Two active groups widen the query-time union beyond Alerts' ERROR-only
+    default: group-errors (levels=['ERROR']) and group-warnings
+    (levels=['WARNING'], ignore_models_id=['model-x']). Because the query-time
+    filter only unions/intersects across ALL active groups, both ERROR and
+    WARNING notifications get fetched for every group, and model-x is not
+    excluded at the query level either (group-errors has no ignore_models_id
+    restriction, so the intersection is empty). The per-group filter added in
+    this subtask is what actually narrows each group back down to what it
+    configured.
+    """
+    seed_receiver_groups(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_receiver_group(
+                group_name='group-errors', contents=['core_alerts'], levels=['ERROR']
+            ),
+            default_receiver_group(
+                group_name='group-warnings',
+                contents=['core_alerts'],
+                levels=['WARNING'],
+                ignore_models_id=['model-x'],
+            ),
+        ],
+    )
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(
+                notification_id='err-a', level='ERROR', model_name='model-a', model_id='model-a'
+            ),
+            default_notification(
+                notification_id='warn-a',
+                level='WARNING',
+                model_name='model-a',
+                model_id='model-a',
+                timestamp=datetime(2024, 6, 1, 12, 1, 0, tzinfo=UTC),
+            ),
+            default_notification(
+                notification_id='warn-x',
+                level='WARNING',
+                model_name='model-x',
+                model_id='model-x',
+                timestamp=datetime(2024, 6, 1, 12, 2, 0, tzinfo=UTC),
+            ),
+        ],
+    )
+
+    await start_and_await_workflow(
+        temporal_env.client,
+        Alerts.run,
+        load_scenario_input('alerts_happy_path.json'),
+        make_workflow_id('alerts-2012'),
+    )
+
+    # One email per group -- each group ended up with exactly one notification.
+    assert len(smtp_server.messages) == 2
+    bodies = [extract_html_body(message) for message in smtp_server.messages]
+    combined = '\n'.join(bodies)
+
+    # group-errors (levels=['ERROR']): received err-a, never warn-a/warn-x
+    # despite both being fetched by the widened query-time union.
+    errors_body = next(b for b in bodies if 'Errors detected:' in b)
+    assert 'Warnings detected:' not in errors_body
+
+    # group-warnings (levels=['WARNING'], ignore_models_id=['model-x']):
+    # received warn-a only -- warn-x was fetched (query-time intersection is
+    # empty, since group-errors doesn't ignore any model) but dropped here by
+    # the per-group ignore_models_id check.
+    warnings_body = next(b for b in bodies if 'Warnings detected:' in b)
+    assert 'Errors detected:' not in warnings_body
+    assert 'model-a' in combined
+    assert 'model-x' not in combined
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_a_2_2_model_ids_allow_list(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    smtp_server,
+):
+    """
+    A.2.2 (SIENTIAPDE-2149): a group configuring `model_ids` only receives
+    notifications from those models, on top of the existing query-time/per-group
+    filtering (SIENTIAPDE-2011/2012).
+
+    group-allow (model_ids=['model-a']) coexists with group-everything (no
+    model_ids restriction). Because the query-time filter only restricts to an
+    allow-list union when EVERY active group has one, group-everything's lack
+    of a model_ids restriction means the query itself fetches both models --
+    the per-group filter added by this ticket is what narrows group-allow back
+    down to model-a only.
+    """
+    seed_receiver_groups(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_receiver_group(
+                group_name='group-allow', contents=['core_alerts'], model_ids=['model-a']
+            ),
+            default_receiver_group(group_name='group-everything', contents=['core_alerts']),
+        ],
+    )
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(
+                notification_id='err-a', level='ERROR', model_name='model-a', model_id='model-a'
+            ),
+            default_notification(
+                notification_id='err-b',
+                level='ERROR',
+                model_name='model-b',
+                model_id='model-b',
+                timestamp=datetime(2024, 6, 1, 12, 1, 0, tzinfo=UTC),
+            ),
+        ],
+    )
+
+    await start_and_await_workflow(
+        temporal_env.client,
+        Alerts.run,
+        load_scenario_input('alerts_happy_path.json'),
+        make_workflow_id('alerts-2149'),
+    )
+
+    # One email per group -- group-allow got only err-a, group-everything got both.
+    assert len(smtp_server.messages) == 2
+    bodies = [extract_html_body(message) for message in smtp_server.messages]
+
+    # Distinguish the two emails by content rather than order: only
+    # group-everything's email mentions model-b.
+    allow_body = next(b for b in bodies if 'model-b' not in b)
+    everything_body = next(b for b in bodies if 'model-b' in b)
+    assert 'model-a' in allow_body
+    assert 'model-a' in everything_body
+
+
 @pytest.mark.e2e
 @pytest.mark.asyncio
 async def test_scenario_a_3_1_empty_queue(

e2e/test_orchestrator_main_workflow.py

@@ -1,5 +1,7 @@
 """E2E tests for the Orchestrator main workflow."""
 
+from datetime import UTC, datetime, timedelta
+
 import pytest
 from pymongo import MongoClient
 from redis import Redis
@@ -26,6 +28,13 @@ async def _schedule_ids(host: str, namespace: str) -> list[str]:
     return [schedule.id async for schedule in await client.list_schedules()]
 
 
+async def _schedule_every(host: str, schedule_id: str) -> timedelta:
+    """Return the interval of a laborious-namespace schedule."""
+    client = await Client.connect(host, namespace='laborious')
+    description = await client.get_schedule_handle(schedule_id).describe()
+    return description.schedule.spec.intervals[0].every
+
+
 def _apply_orchestrator_seeds(
     scenario: dict,
     mongo_uri: str,
@@ -224,3 +233,83 @@ async def test_scenario_1_6_1_ttl_index_bootstrap(
         assert any('expireAfterSeconds' in info for info in indexes.values())
     finally:
         mongo.close()
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_7_1_simple_metrics_parallel_windows(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    temporal_host: str,
+    redis_client: Redis,
+):
+    """1.7.1: One pipelines doc per window gives one schedule per window, each with its own interval."""
+    scenario = load_scenario_input('orchestrator_simple_metrics_windows')
+    _apply_orchestrator_seeds(scenario, mongo_uri, redis_client)
+
+    await start_and_await_workflow(
+        temporal_env.client,
+        Orchestrator.run,
+        scenario['workflow_input'],
+        make_workflow_id('orchestrator-sm-windows'),
+    )
+
+    short, long = 'model-sm-simple_metrics-60m', 'model-sm-simple_metrics-1440m'
+    assert await _schedule_every(temporal_host, short) == timedelta(hours=1)
+    assert await _schedule_every(temporal_host, long) == timedelta(days=1)
+
+
+def _laborious_adaptive_write(mongo_uri: str, schedule_name: str, escalated: bool) -> None:
+    """Replicates laborious `update_simple_metrics_adaptive_state` on an escalated flip."""
+    mongo = MongoClient(mongo_uri)
+    try:
+        mongo[E2E_DATABASE]['pipelines'].update_one(
+            {'schedule_name': schedule_name},
+            {
+                '$set': {
+                    'adaptive_state': {'escalated': escalated, 'healthy_streak': 0},
+                    # BSON datetime, as laborious writes it; a string would break the tick.
+                    'updated_at': datetime.now(UTC),
+                }
+            },
+        )
+    finally:
+        mongo.close()
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_scenario_1_7_2_simple_metrics_adaptive_frequency_round_trip(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    temporal_host: str,
+    redis_client: Redis,
+):
+    """1.7.2: Escalation written by laborious switches only that window to frequency_alert; recovery switches it back."""
+    scenario = load_scenario_input('orchestrator_simple_metrics_windows')
+    _apply_orchestrator_seeds(scenario, mongo_uri, redis_client)
+    short, long = 'model-sm-simple_metrics-60m', 'model-sm-simple_metrics-1440m'
+
+    async def run(tag: str) -> None:
+        await start_and_await_workflow(
+            temporal_env.client,
+            Orchestrator.run,
+            scenario['workflow_input'],
+            make_workflow_id(f'orchestrator-sm-adaptive-{tag}'),
+        )
+
+    await run('create')
+    assert await _schedule_every(temporal_host, short) == timedelta(hours=1)
+
+    _laborious_adaptive_write(mongo_uri, short, escalated=True)
+    await run('escalate')
+    assert await _schedule_every(temporal_host, short) == timedelta(minutes=5)
+    assert await _schedule_every(temporal_host, long) == timedelta(days=1)
+
+    _laborious_adaptive_write(mongo_uri, short, escalated=False)
+    await run('recover')
+    assert await _schedule_every(temporal_host, short) == timedelta(hours=1)

e2e/test_reports_main_workflow.py

@@ -1,6 +1,6 @@
 """E2E tests for the Reports main workflow."""
 
-from email.message import EmailMessage, Message
+from datetime import UTC, datetime
 
 import pytest
 from temporalio.testing import WorkflowEnvironment
@@ -10,6 +10,7 @@ from e2e.helpers import (
     count_log_report_rows,
     default_notification,
     default_receiver_group,
+    extract_html_body,
     load_scenario_input,
     make_workflow_id,
     seed_notifications,
@@ -19,41 +20,6 @@ from e2e.helpers import (
 from orchestrator.workflows.reports import Reports
 
 
-def _extract_html_body(message: EmailMessage | Message) -> str:
-    """
-    Return the text/html portion of an email message as a decoded string.
-
-    Walks every part looking for the first text/html payload, decoding it
-    according to the part's transfer encoding and charset. Falls back to
-    the message's own ``get_content``/raw payload when no HTML part is
-    present so callers can still inspect plain-text reports.
-
-    Args:
-        message: Captured email message returned by the test SMTP server.
-
-    Return:
-        str: HTML body content, or an empty string when nothing decodable
-            is found.
-    """
-    if message.is_multipart():
-        for part in message.walk():
-            if part.get_content_type() != 'text/html':
-                continue
-            payload = part.get_payload(decode=True)
-            if payload is None:
-                continue
-            charset = part.get_content_charset() or 'utf-8'
-            return payload.decode(charset, errors='replace')
-    payload = message.get_payload(decode=True)
-    if payload is not None:
-        charset = message.get_content_charset() or 'utf-8'
-        return payload.decode(charset, errors='replace')
-    try:
-        return message.get_content()
-    except (AttributeError, KeyError):
-        return str(message.get_payload() or '')
-
-
 @pytest.mark.e2e
 @pytest.mark.asyncio
 async def test_scenario_r_1_1_happy_path(
@@ -79,13 +45,13 @@ async def test_scenario_r_1_1_happy_path(
                 notification_id='r-warn',
                 level='WARNING',
                 model_name='m2',
-                timestamp='2024-06-01 12:01:00+0000',
+                timestamp=datetime(2024, 6, 1, 12, 1, 0, tzinfo=UTC),
             ),
             default_notification(
                 notification_id='r-info',
                 level='INFO',
                 model_name='m3',
-                timestamp='2024-06-01 12:02:00+0000',
+                timestamp=datetime(2024, 6, 1, 12, 2, 0, tzinfo=UTC),
             ),
         ],
     )
@@ -98,7 +64,7 @@ async def test_scenario_r_1_1_happy_path(
     )
 
     assert len(smtp_server.messages) == 1
-    body = _extract_html_body(smtp_server.messages[0])
+    body = extract_html_body(smtp_server.messages[0])
     assert 'Errors detected:' in body
     assert 'Warnings detected:' in body
     assert 'Infos detected:' in body
@@ -137,7 +103,7 @@ async def test_scenario_r_1_2_error_section_only(
     )
 
     assert smtp_server.messages, 'Expected at least one report email'
-    body = _extract_html_body(smtp_server.messages[0])
+    body = extract_html_body(smtp_server.messages[0])
     assert 'Errors detected:' in body
     assert 'Warnings detected:' not in body
     assert 'Infos detected:' not in body

e2e/test_subworkflow_load_notification_package.py

@@ -1,5 +1,7 @@
 """E2E tests for LoadNotificationPackage subworkflow."""
 
+from datetime import UTC, datetime
+
 import pytest
 from redis import Redis
 from temporalio.testing import WorkflowEnvironment
@@ -42,10 +44,12 @@ async def test_load_package_without_prior_timestamp(
     """
     No Redis timestamp → returns notifications and stores max timestamp.
 
-    Notifications are seeded with the exact production format produced by
-    ``sientia_do.notifications.models.Notification`` (string in
-    ``DATETIME_FORMAT_WITH_TZ``, e.g. ``"2024-06-01 11:00:00+0000"``). The
-    cached "last timestamp" must mirror that representation.
+    Notifications are seeded with the exact production representation written
+    by ``sientia_do.notifications.models.Notification`` since SIENTIAPDE-2007:
+    a native ``datetime`` (BSON Date), not a formatted string. The cached
+    "last timestamp" written back to Redis by ``put_last_data_timestamp`` is a
+    string formatted per ``DATETIME_FORMAT_MS_WITH_TZ`` (microseconds
+    included) — that is the format this test asserts against.
     """
     seed_receiver_groups(mongo_uri, E2E_DATABASE, [default_receiver_group()])
     seed_notifications(
@@ -54,11 +58,11 @@ async def test_load_package_without_prior_timestamp(
         [
             default_notification(
                 notification_id='n1',
-                timestamp='2024-06-01 10:00:00+0000',
+                timestamp=datetime(2024, 6, 1, 10, 0, 0, tzinfo=UTC),
             ),
             default_notification(
                 notification_id='n2',
-                timestamp='2024-06-01 11:00:00+0000',
+                timestamp=datetime(2024, 6, 1, 11, 0, 0, tzinfo=UTC),
             ),
         ],
     )
@@ -75,7 +79,7 @@ async def test_load_package_without_prior_timestamp(
 
     assert len(result['notification_package']) == 2
     stored = redis_client.get('notification_last_timestamp:Alerts') or ''
-    assert stored.strip('"') == '2024-06-01 11:00:00+0000'
+    assert stored.strip('"') == '2024-06-01 11:00:00.000000+0000'
 
 
 @pytest.mark.e2e
@@ -90,27 +94,27 @@ async def test_load_package_with_prior_timestamp(
     """
     Existing timestamp → only newer notifications returned.
 
-    Both the seeded "last timestamp" (Redis) and the notification timestamps
-    (Mongo) follow the production format used by
-    ``sientia_do.notifications.models.Notification`` (string in
-    ``DATETIME_FORMAT_WITH_TZ``). ``load_latest_data`` will translate the
-    Redis value into a Python ``datetime`` and apply ``{$gt: <Date>}``
-    against the Mongo string timestamps; this exercise surfaces the real
-    BSON comparison semantics rather than a synthetic ideal.
+    The seeded "last timestamp" (Redis) follows the format
+    ``put_last_data_timestamp`` actually writes (``DATETIME_FORMAT_MS_WITH_TZ``,
+    microseconds included); the notification timestamps (Mongo) are native
+    ``datetime`` values, matching current production behavior since
+    SIENTIAPDE-2007. ``load_latest_data`` parses the Redis string back into a
+    ``datetime`` and applies ``{$gt: <Date>}`` against the Mongo BSON dates —
+    this exercises the real comparison semantics rather than a synthetic ideal.
     """
     seed_receiver_groups(mongo_uri, E2E_DATABASE, [default_receiver_group()])
-    seed_last_timestamp(redis_client, 'Alerts', '2024-06-01 10:30:00+0000')
+    seed_last_timestamp(redis_client, 'Alerts', '2024-06-01 10:30:00.000000+0000')
     seed_notifications(
         mongo_uri,
         E2E_DATABASE,
         [
             default_notification(
                 notification_id='old',
-                timestamp='2024-06-01 10:00:00+0000',
+                timestamp=datetime(2024, 6, 1, 10, 0, 0, tzinfo=UTC),
             ),
             default_notification(
                 notification_id='new',
-                timestamp='2024-06-01 11:00:00+0000',
+                timestamp=datetime(2024, 6, 1, 11, 0, 0, tzinfo=UTC),
             ),
         ],
     )
@@ -129,6 +133,129 @@ async def test_load_package_with_prior_timestamp(
     assert ids == {'new'}
 
 
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_load_package_query_time_filter_union_and_intersection(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    redis_client: Redis,
+):
+    """
+    SIENTIAPDE-2011: query-time filter unions active groups' levels and
+    intersects their ignore_models_id, instead of fetching everything.
+
+    Two active receiver groups narrow the fetch differently:
+      - group_a: levels=['ERROR'], ignore_models_id=['model-x']
+      - group_b: levels=['WARNING'], ignore_models_id=['model-x', 'model-y']
+
+    The Mongo query must return exactly what the UNION of levels allows
+    (ERROR or WARNING, never INFO) and must exclude only models in the
+    INTERSECTION of every active group's ignore list (model-x, ignored by
+    both). A model ignored by only one group (model-y) must still be
+    fetched, since the other group still wants it — per-group correctness
+    for that case is SIENTIAPDE-2012's job, not this query-time filter's.
+    """
+    seed_receiver_groups(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_receiver_group(
+                group_name='group_a', levels=['ERROR'], ignore_models_id=['model-x']
+            ),
+            default_receiver_group(
+                group_name='group_b',
+                levels=['WARNING'],
+                ignore_models_id=['model-x', 'model-y'],
+            ),
+        ],
+    )
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(
+                notification_id='error-included', level='ERROR', model_id='model-a'
+            ),
+            default_notification(
+                notification_id='warning-included', level='WARNING', model_id='model-a'
+            ),
+            default_notification(notification_id='info-excluded', level='INFO', model_id='model-a'),
+            default_notification(
+                notification_id='modelx-excluded', level='ERROR', model_id='model-x'
+            ),
+            default_notification(
+                notification_id='modely-included', level='WARNING', model_id='model-y'
+            ),
+        ],
+    )
+
+    input_data = load_scenario_input('subworkflow_load_notification_package.json')
+    _build_metadata(input_data)
+
+    result = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-filter'),
+    )
+
+    ids = {n['notification_id'] for n in result['notification_package']}
+    assert ids == {'error-included', 'warning-included', 'modely-included'}
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_load_package_query_time_filter_model_ids_union(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    redis_client: Redis,
+):
+    """
+    SIENTIAPDE-2149: query-time filter unions active groups' `model_ids`
+    allow-list when every active group sets one, excluding any model outside
+    that union from the fetch entirely -- not just from a single group's
+    per-group slice.
+
+    group_a wants only model-a, group_b wants only model-b -- both restrict
+    levels to ERROR so only the model_ids axis is under test. model-c is
+    outside the union of both allow-lists and must never be fetched.
+    """
+    seed_receiver_groups(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_receiver_group(group_name='group_a', levels=['ERROR'], model_ids=['model-a']),
+            default_receiver_group(group_name='group_b', levels=['ERROR'], model_ids=['model-b']),
+        ],
+    )
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(notification_id='a-included', level='ERROR', model_id='model-a'),
+            default_notification(notification_id='b-included', level='ERROR', model_id='model-b'),
+            default_notification(notification_id='c-excluded', level='ERROR', model_id='model-c'),
+        ],
+    )
+
+    input_data = load_scenario_input('subworkflow_load_notification_package.json')
+    _build_metadata(input_data)
+
+    result = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-model-ids-union'),
+    )
+
+    ids = {n['notification_id'] for n in result['notification_package']}
+    assert ids == {'a-included', 'b-included'}
+
+
 @pytest.mark.e2e
 @pytest.mark.asyncio
 async def test_load_package_empty_mongo_no_redis_write(
@@ -152,3 +279,114 @@ async def test_load_package_empty_mongo_no_redis_write(
     )
 
     assert redis_client.get('notification_last_timestamp:Alerts') is None
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_load_package_batch_size_caps_and_returns_oldest_first(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    redis_client: Redis,
+    test_activities,
+):
+    """
+    SIENTIAPDE-2013: a capped run returns the oldest pending documents first,
+    and the incremental cursor picks up the remainder on the next run.
+
+    Five notifications exist, but the batch size is capped at 2. Each run must
+    return exactly the next-oldest 2 (never an arbitrary subset), and the
+    Redis cursor must advance only to the max timestamp of what was actually
+    returned -- not the true max across all 5 -- or the skipped documents
+    would never be fetched again.
+    """
+    test_activities.notification_queue_batch_size = 2
+
+    seed_receiver_groups(mongo_uri, E2E_DATABASE, [default_receiver_group()])
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(
+                notification_id=f'n{i}',
+                timestamp=datetime(2024, 6, 1, 12, i, 0, tzinfo=UTC),
+            )
+            for i in range(1, 6)
+        ],
+    )
+
+    input_data = load_scenario_input('subworkflow_load_notification_package.json')
+    _build_metadata(input_data)
+
+    first_run = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-batch-1'),
+    )
+    assert [n['notification_id'] for n in first_run['notification_package']] == ['n1', 'n2']
+
+    second_run = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-batch-2'),
+    )
+    assert [n['notification_id'] for n in second_run['notification_package']] == ['n3', 'n4']
+
+    third_run = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-batch-3'),
+    )
+    assert [n['notification_id'] for n in third_run['notification_package']] == ['n5']
+
+
+@pytest.mark.e2e
+@pytest.mark.asyncio
+async def test_load_package_batch_size_zero_means_unlimited(
+    temporal_env: WorkflowEnvironment,
+    orchestrator_worker,
+    stub_workers,
+    mongo_uri: str,
+    test_activities,
+):
+    """
+    SIENTIAPDE-2013: notification_queue_batch_size=0 (the config default) must
+    NOT cap the query. This exercises pymongo's own documented convention --
+    `Collection.find(..., limit=0)` is equivalent to no limit at all -- against
+    a real MongoDB instance rather than trusting the convention on faith.
+
+    12 notifications is deliberately more than any other test in this file
+    seeds, so a regression that silently caps at some smaller default (e.g. if
+    `limit=0` were ever misread as "return nothing", or pymongo's own behavior
+    changed) would fail loudly here instead of passing by coincidence.
+    """
+    test_activities.notification_queue_batch_size = 0
+
+    seed_receiver_groups(mongo_uri, E2E_DATABASE, [default_receiver_group()])
+    seed_notifications(
+        mongo_uri,
+        E2E_DATABASE,
+        [
+            default_notification(
+                notification_id=f'n{i}',
+                timestamp=datetime(2024, 6, 1, 12, i, 0, tzinfo=UTC),
+            )
+            for i in range(1, 13)
+        ],
+    )
+
+    input_data = load_scenario_input('subworkflow_load_notification_package.json')
+    _build_metadata(input_data)
+
+    result = await start_and_await_workflow(
+        temporal_env.client,
+        LoadNotificationPackage.run,
+        input_data,
+        make_workflow_id('load-pkg-batch-unlimited'),
+    )
+
+    assert len(result['notification_package']) == 12

orchestrator/activities/activities.py

@@ -84,6 +84,7 @@ class Activities(TemporalManager, SlotManager, Formatters, MongoDB, Email, Postg
             connection_string=mongodb_config['connection_string'],
             database_name=mongodb_config['database_name'],
             ttl_index_seconds=mongodb_config['ttl_index_seconds'],
+            notification_queue_batch_size=mongodb_config['notification_queue_batch_size'],
             logger=logger,
             notification_handler=notification_handler,
             metrics_controller=metrics_controller,

orchestrator/activities/formatters.py

@@ -7,15 +7,20 @@ with workflow.unsafe.imports_passed_through():
     from math import ceil
     from typing import Any, TypedDict, cast
 
-    from pandas import DataFrame
+    from pandas import DataFrame, to_datetime
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
     from sientia_do.notifications.models import NotificationLevel
     from sientia_do.observability.sientia_monitoring import MetricsController, SientiaMonitoring
-    from sientia_do.temporal.constants import DATETIME_FORMAT_MS_WITH_TZ, now
+    from sientia_do.temporal.constants import (
+        DATETIME_FORMAT_MS_WITH_TZ,
+        DATETIME_FORMAT_WITH_TZ,
+        now,
+    )
 
     from orchestrator.utils.orchestrator_functions import (
         build_tag_config,
         drift,
+        frequency_alert_warning,
         gather_read_tags,
         minimal_retrain,
         pi_web_api_scouter,
@@ -365,6 +370,8 @@ class Formatters(SientiaMonitoring):
         Compares the current schedule configuration in Temporal with the new schedule
         configuration to determine which schedules need to be created, updated, or deleted.
         Uses timestamp comparison to identify schedules that have changed.
+        Also sends a SIMPLE_METRICS_FREQUENCY_ALERT_BELOW_FLOOR WARNING for laborious schedules
+        whose frequency_alert is below 1m (the value is kept) or unparseable (it is ignored).
 
         Args:
             input_data (dict[str, Any]): The input data containing:
@@ -405,6 +412,25 @@ class Formatters(SientiaMonitoring):
                 schedules, current_schedules, to_update, to_create, namespace, metadata
             )
 
+        # Fires whenever the schedule is created or updated (each `updated_at` change,
+        # including escalation flips).
+        changed = {**to_create[self.laborious_namespace], **to_update[self.laborious_namespace]}
+        for schedule_name, schedule in changed.items():
+            warning = frequency_alert_warning(schedule.get('frequency_alert'))
+            if warning:
+                self.send_notification(
+                    metadata={
+                        **metadata,
+                        'schedule_name': schedule_name,
+                        'model_id': schedule.get('model_id'),
+                        'model_name': schedule.get('model_name'),
+                    },
+                    notification_id='SIMPLE_METRICS_FREQUENCY_ALERT_BELOW_FLOOR',
+                    message=warning,
+                    block='create_schedule_config',
+                    level=NotificationLevel.WARNING,
+                )
+
         for namespace, schedules in current_schedule_config.items():
             for schedule_name in schedules:
                 if schedule_name not in schedule_config[namespace]:
@@ -755,9 +781,24 @@ class Formatters(SientiaMonitoring):
                 key = f'{notification_id}:{trigger}'
 
                 if key not in data:
+                    # ``notification['timestamp']`` started out as a native
+                    # ``datetime``/BSON Date in Mongo, but by the time it gets here
+                    # it has crossed at least one Temporal activity boundary:
+                    # Temporal's default data converter serializes ``datetime``
+                    # values to plain ISO-8601 strings and does not restore them on
+                    # decode, and pymongo returns naive datetimes for BSON Date
+                    # fields (no tzinfo), so the ISO string may carry no UTC offset
+                    # at all. Normalizing with ``utc=True`` and re-formatting to
+                    # ``DATETIME_FORMAT_WITH_TZ`` (same pattern as
+                    # ``SlotManager.put_last_data_timestamp``) keeps the value a
+                    # fixed-format string that ``export_data_to_postgres``'s
+                    # ``timestamp_conversion`` can parse after the next boundary
+                    # crossing, regardless of what shape it arrived in here.
                     data[key] = {
                         'status': group_config['status'],
-                        'timestamp': notification['timestamp'],
+                        'timestamp': to_datetime(notification['timestamp'], utc=True).strftime(
+                            DATETIME_FORMAT_WITH_TZ
+                        ),
                         'groups': [group_name],
                         'message': notification['message'],
                         'level': notification['level'],
@@ -805,6 +846,9 @@ class Formatters(SientiaMonitoring):
                     - group_name (str): Name of the receiver group
                     - contents (list): Content types to include (must contain "reports")
                     - ignore (list, optional): Notification IDs to exclude
+                    - levels (list, optional): Notification levels to include
+                    - ignore_models_id (list, optional): Model IDs to exclude
+                    - model_ids (list, optional): Model IDs to allow (SIENTIAPDE-2149)
 
         Returns:
             dict[str, Any]: Filtered receiver groups with their notifications, keyed by group_name.
@@ -828,6 +872,9 @@ class Formatters(SientiaMonitoring):
             already_added_keys = []
 
             ignore_list = receiver_group.get('ignore', [])
+            levels_list = receiver_group.get('levels')
+            ignore_models_id_list = receiver_group.get('ignore_models_id') or []
+            model_ids_list = receiver_group.get('model_ids')
 
             for notification in notification_package:
                 alert_type = 'reports'
@@ -840,6 +887,12 @@ class Formatters(SientiaMonitoring):
                     alert_type in receiver_group['contents']
                     and notification_id not in ignore_list
                     and key not in already_added_keys
+                    and (not levels_list or notification.get('level') in levels_list)
+                    and (
+                        not ignore_models_id_list
+                        or notification.get('model_id') not in ignore_models_id_list
+                    )
+                    and (not model_ids_list or notification.get('model_id') in model_ids_list)
                 ):
                     receiver_groups[group_name]['notifications'].append(notification)
                     already_added_keys.append(key)

orchestrator/activities/mongo_db.py

@@ -31,6 +31,8 @@ class MongoDB(SientiaMonitoring):
         ttl_index_seconds (int): TTL index duration in seconds
         logger (Logger): Application logger instance
         notification_handler (NotificationHandler): Notification management handler
+        notification_queue_batch_size (int): Hard cap on documents fetched per
+            `load_latest_data` run (SIENTIAPDE-2013). `0` means no limit.
     """
 
     def __init__(
@@ -41,6 +43,7 @@ class MongoDB(SientiaMonitoring):
         logger: Logger,
         notification_handler: NotificationHandler,
         metrics_controller: MetricsController,
+        notification_queue_batch_size: int = 0,
     ):
         SientiaMonitoring.__init__(
             self,
@@ -61,6 +64,7 @@ class MongoDB(SientiaMonitoring):
         )
 
         self.ttl_index_seconds = ttl_index_seconds
+        self.notification_queue_batch_size = notification_queue_batch_size
 
         # Initialize MongoDB client here (omitted for brevity)
         logger.info('MongoDB connection initialized')
@@ -238,7 +242,12 @@ class MongoDB(SientiaMonitoring):
             for pipeline in updated_pipelines
             if pipeline['success']
         ]
-        data_filter = {'$or': argument} if argument else {}
+        if not argument:
+            # Every update failed: stamp nothing so they are retried next tick (same guard as
+            # create_pipelines_timestamps). An empty filter would match every tracked schedule.
+            self.info('No successful pipeline updates to timestamp', metadata=metadata)
+            return
+        data_filter = {'$or': argument}
 
         try:
             self.mongo_db_repository.update_many(
@@ -339,7 +348,13 @@ class MongoDB(SientiaMonitoring):
             for pipeline in deleted_pipelines
             if pipeline['success']
         ]
-        data_filter = {'$or': argument} if argument else {}
+        if not argument:
+            # Every delete failed: remove nothing (same guard as create_pipelines_timestamps).
+            # An empty filter would wipe orchestrated_schedules and make the next tick delete
+            # and recreate every orchestrated schedule.
+            self.info('No successful pipeline deletions to untrack', metadata=metadata)
+            return
+        data_filter = {'$or': argument}
 
         try:
             self.mongo_db_repository.delete_many('orchestrated_schedules', data_filter, metadata)
@@ -447,6 +462,10 @@ class MongoDB(SientiaMonitoring):
         filtering by timestamp to enable incremental data processing. It
         handles connection management and provides comprehensive error reporting.
 
+        The query is capped at ``notification_queue_batch_size`` documents and sorted by
+        ``timestamp`` ascending, so a capped run returns the oldest pending documents first --
+        the caller advances its cursor only to what was actually returned, so nothing is skipped.
+
         Args:
             input_data (dict[str, Any]): Activity input parameters.
                 Required fields:
@@ -488,7 +507,13 @@ class MongoDB(SientiaMonitoring):
 
             self.debug(f'Data filter: {data_filter}', metadata=metadata)
 
-            data = self.mongo_db_repository.find(collection_name, data_filter, metadata)
+            data = self.mongo_db_repository.find(
+                collection_name,
+                data_filter,
+                metadata,
+                limit=self.notification_queue_batch_size,
+                sort=[('timestamp', 1)],
+            )
 
             self.debug(f'Collected: {data}', metadata=metadata)
 

orchestrator/activities/slot_manager.py

@@ -7,7 +7,7 @@ with workflow.unsafe.imports_passed_through():
     from logging import Logger
     from typing import Any
 
-    from pandas import DataFrame
+    from pandas import DataFrame, to_datetime
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
     from sientia_do.notifications.models import NotificationLevel
     from sientia_do.observability.sientia_monitoring import MetricsController, SientiaMonitoring
@@ -331,18 +331,33 @@ class SlotManager(SientiaMonitoring):
             self.warning('No data to insert', metadata=metadata)
             return None
 
-        last_data_timestamp = data['timestamp'].max()
+        # ``timestamp`` started out as a native ``datetime``/BSON Date in Mongo,
+        # but by the time it gets here it has crossed a Temporal activity
+        # boundary (this activity's ``input_data`` is whatever
+        # ``load_latest_data`` returned): Temporal's default data converter
+        # serializes ``datetime`` values to plain ISO-8601 strings and does not
+        # restore them on decode, so ``data['timestamp']`` is a ``str`` in a
+        # real workflow run even though unit tests mocking the activity in
+        # isolation never see that conversion. pymongo also returns naive
+        # datetimes for BSON Date fields (no tzinfo), so the ISO string may
+        # carry no UTC offset at all. ``utc=True`` normalizes every case
+        # (native datetime/Timestamp, offset-aware or naive ISO string) into a
+        # UTC-aware ``Timestamp`` — naive values are BSON Dates, always UTC.
+        last_data_timestamp = to_datetime(data['timestamp'], utc=True).max()
 
         self.debug(f'Last collected timestamp to insert: {last_data_timestamp}', metadata=metadata)
 
-        # ``timestamp`` is a native ``datetime``/``Timestamp`` (BSON Date from
-        # Mongo), which the Redis repository's plain ``json.dumps`` cannot
-        # serialize. Store it as a string formatted per
-        # ``DATETIME_FORMAT_MS_WITH_TZ`` instead.
-        last_data_timestamp = last_data_timestamp.strftime(DATETIME_FORMAT_MS_WITH_TZ)
+        # The Redis repository's plain ``json.dumps`` cannot serialize a
+        # ``Timestamp``/``datetime`` directly, so store it as a string
+        # formatted per ``DATETIME_FORMAT_MS_WITH_TZ`` instead. Kept as a
+        # separate variable (rather than reassigning ``last_data_timestamp``)
+        # since it's a different type — mypy infers a local's type from its
+        # first assignment and flags a same-name reassignment to an
+        # incompatible type as an error.
+        last_data_timestamp_str = last_data_timestamp.strftime(DATETIME_FORMAT_MS_WITH_TZ)
 
         try:
-            self.redis_repository.set(key, last_data_timestamp, ttl=60 * 60 * 5)
+            self.redis_repository.set(key, last_data_timestamp_str, ttl=60 * 60 * 5)
         except Exception as e:
             self.send_notification(
                 metadata=metadata,
@@ -354,7 +369,7 @@ class SlotManager(SientiaMonitoring):
             )
             raise e
 
-        return last_data_timestamp
+        return last_data_timestamp_str
 
     @activity.defn(name='filter_notification_alerts')
     def filter_notification_alerts(self, input_data: dict[str, Any]) -> dict[str, Any]:
@@ -381,6 +396,9 @@ class SlotManager(SientiaMonitoring):
                     - group_name (str): Name of the receiver group
                     - contents (list): Alert types to include (core_alerts, persistent_alerts)
                     - ignore (list, optional): Notification IDs to exclude
+                    - levels (list, optional): Notification levels to include
+                    - ignore_models_id (list, optional): Model IDs to exclude
+                    - model_ids (list, optional): Model IDs to allow (SIENTIAPDE-2149)
                 - notification_ttl (int): Seconds before considering notification persistent
 
         Returns:
@@ -400,48 +418,125 @@ class SlotManager(SientiaMonitoring):
 
         for receiver_group in sending_configs:
             group_name = receiver_group['group_name']
-            receiver_groups[group_name] = {**receiver_group, 'notifications': []}
-            receiver_groups[group_name]['notifications'] = []
-
-            already_added_keys = []
-
-            ignore_list = receiver_group.get('ignore', [])
-
-            for notification in notification_package:
-                alert_type = 'do_nothing'
-                notification_id = notification['notification_id']
-                # Check if notification was recently sent
-                key = f'{notification["trigger"]}:{notification_id}'
-
-                last_sent = self.redis_repository.get(key)
-
-                if last_sent is None:
-                    alert_type = 'core_alerts'
-
-                else:
-                    last_sent = datetime.strptime(last_sent, DATETIME_FORMAT_MS_WITH_TZ)
-
-                    # Check if "notification_ttl" seconds has passed since last sent
-                    if (now() - last_sent) > timedelta(seconds=notification_ttl):
-                        alert_type = 'persistent_alerts'
-
-                # Check if this group must be notified
-                if (
-                    alert_type in receiver_group['contents']
-                    and notification_id not in ignore_list
-                    and key not in already_added_keys
-                ):
-                    receiver_groups[group_name]['notifications'].append(notification)
-                    already_added_keys.append(key)
+            notifications = self._filter_group_notifications(
+                receiver_group, notification_package, notification_ttl
+            )
+            receiver_groups[group_name] = {**receiver_group, 'notifications': notifications}
 
         # Remove groups with no notifications
-        receiver_groups = {
+        return {
             group_name: group
             for group_name, group in receiver_groups.items()
             if group['notifications']
         }
 
-        return receiver_groups
+    def _filter_group_notifications(
+        self,
+        receiver_group: dict[str, Any],
+        notification_package: list[dict[str, Any]],
+        notification_ttl: int,
+    ) -> list[dict[str, Any]]:
+        """
+        Select the notifications this receiver group should get.
+
+        Args:
+            receiver_group (dict[str, Any]): The receiver group configuration.
+            notification_package (list[dict[str, Any]]): Candidate notifications.
+            notification_ttl (int): Seconds before considering a notification persistent.
+
+        Returns:
+            list[dict[str, Any]]: Notifications matching this group's filters.
+        """
+        already_added_keys: list[str] = []
+        ignore_list = receiver_group.get('ignore', [])
+        levels_list = receiver_group.get('levels')
+        ignore_models_id_list = receiver_group.get('ignore_models_id') or []
+        model_ids_list = receiver_group.get('model_ids')
+
+        notifications = []
+        for notification in notification_package:
+            notification_id = notification['notification_id']
+            key = f'{notification["trigger"]}:{notification_id}'
+            alert_type = self._determine_alert_type(key, notification_ttl)
+
+            if (
+                alert_type in receiver_group['contents']
+                and self._check_keys_and_notif_id(
+                    notification_id, ignore_list, key, already_added_keys
+                )
+                and self._check_levels_and_model_filters(
+                    notification, levels_list, ignore_models_id_list, model_ids_list
+                )
+            ):
+                notifications.append(notification)
+                already_added_keys.append(key)
+
+        return notifications
+
+    def _determine_alert_type(self, key: str, notification_ttl: int) -> str:
+        """
+        Classify a notification as new, persistent, or not-yet-due.
+
+        Args:
+            key (str): The Redis cache key for this notification.
+            notification_ttl (int): Seconds before considering a notification persistent.
+
+        Returns:
+            str: 'core_alerts', 'persistent_alerts', or 'do_nothing'.
+        """
+        last_sent = self.redis_repository.get(key)
+        if last_sent is None:
+            return 'core_alerts'
+
+        last_sent = datetime.strptime(last_sent, DATETIME_FORMAT_MS_WITH_TZ)
+        if (now() - last_sent) > timedelta(seconds=notification_ttl):
+            return 'persistent_alerts'
+
+        return 'do_nothing'
+
+    @staticmethod
+    def _check_levels_and_model_filters(
+        notification, levels_list, ignore_models_id_list, model_ids_list
+    ):
+        """
+        Check the notification's level and model_id against a group's filters.
+
+        Args:
+            notification (dict): The notification to check.
+            levels_list (list, optional): List of levels to include. If None, all levels are included.
+            ignore_models_id_list (list, optional): List of model_ids to exclude. If None, no models are excluded.
+            model_ids_list (list, optional): List of model_ids to allow (SIENTIAPDE-2149).
+                If None/empty, all models are allowed. Independent of ignore_models_id_list --
+                both apply together when both are configured.
+
+        Returns:
+            bool: True if the notification passes the checks, False otherwise.
+        """
+        level_check = not levels_list or notification.get('level') in levels_list
+        ignore_models_id_check = (
+            not ignore_models_id_list or notification.get('model_id') not in ignore_models_id_list
+        )
+        model_ids_check = not model_ids_list or notification.get('model_id') in model_ids_list
+        return level_check and ignore_models_id_check and model_ids_check
+
+    @staticmethod
+    def _check_keys_and_notif_id(
+        notification_id, ignore_list, key, already_added_keys: list[str]
+    ) -> bool:
+        """
+        Check if the key has already been added to the list of already added keys.
+
+        Args:
+            notification_id (str): The ID of the notification.
+            ignore_list (list): List of notification IDs to ignore.
+            key (str): The key to check.
+            already_added_keys (list[str]): List of keys that have already been added.
+
+        Returns:
+            bool: True if the key has not been added yet, False otherwise.
+
+        """
+        return notification_id not in ignore_list and key not in already_added_keys
 
     @activity.defn(name='store_notification_cache')
     def store_notification_cache(self, input_data: dict[str, Any]) -> None:

orchestrator/utils/connectors_config.py

@@ -21,7 +21,8 @@ def build_mongodb_config():
     Build MongoDB configuration from environment variables.
 
     Returns:
-        dict: MongoDB configuration with connection string, database name, and TTL index seconds.
+        dict: MongoDB configuration with connection string, database name, TTL index seconds,
+            and the notification queue's per-run batch size cap.
     """
     username = getenv('MONGODB_USERNAME', 'root')
     password = getenv('MONGODB_PASSWORD', 'wKZDbMNU1c')
@@ -32,6 +33,7 @@ def build_mongodb_config():
         'connection_string': connection_string,
         'database_name': getenv('MONGODB_DATABASE_NAME', 'sientia'),
         'ttl_index_seconds': int(getenv('MONGODB_TTL_INDEX_HOURS', '1')) * 3600,
+        'notification_queue_batch_size': int(getenv('NOTIFICATION_QUEUE_BATCH_SIZE', '0')),
     }
 
 

orchestrator/utils/orchestrator_functions.py

@@ -85,6 +85,9 @@ def simple_metrics(config: dict[str, Any]):
             - interval_minutes (int, optional): Computation interval in minutes (default: 60)
             - metrics (list[str], optional): List of metrics to compute
                 (default: ['rmse', 'mse', 'mae', 'r2'])
+            - frequency_alert (str, optional): Frequency used while `adaptive_state.escalated`
+                is true; same format as `frequency`. Absent = adaptive frequency off.
+            - adaptive_state (dict, optional): `{escalated, healthy_streak}`, written by laborious.
             - Additional fields from common_config
 
     Returns:
@@ -92,17 +95,56 @@ def simple_metrics(config: dict[str, Any]):
             actual data), target table, time interval, and metrics list. Results stored in
             'simple_metrics' table
     """
+    base = common_config(config)
+    frequency_alert = config.get('frequency_alert')
+    # SIENTIAPDE-1995: laborious flips `adaptive_state.escalated` on a threshold breach (and back
+    # after the window is renewed without one) and bumps `updated_at`, which makes this tick
+    # rewrite the schedule.
+    state = config.get('adaptive_state')
+    # Hand-written docs: a malformed state must not raise and kill the tick for every model.
+    if frequency_alert and isinstance(state, dict) and state.get('escalated') is True:
+        try:
+            # Temporal parses the schedule frequency on create/update; an unparseable value
+            # would fail it, so keep `frequency` (create_schedule_config still warns).
+            parse_frequency(str(frequency_alert))
+            base['frequency'] = frequency_alert
+        except ValueError:
+            pass
     return {
-        **common_config(config),
+        **base,
         'schema': 'sientia_data',
         'predictions_table_name': 'predictions',
         'data_table_name': 'laborious_data',
         'target_table_name': 'simple_metrics',
         'interval_minutes': config.get('interval_minutes', 60),
         'metrics': config.get('metrics', ['rmse', 'mse', 'mae', 'r2']),
+        # Forwarded so create_schedule_config can warn on a value below the floor.
+        'frequency_alert': frequency_alert,
     }
 
 
+MIN_FREQUENCY_ALERT_SECONDS = 60
+
+
+def frequency_alert_warning(frequency_alert: str | None) -> str | None:
+    """
+    Warning message when `frequency_alert` is below the 1-minute floor or does not parse,
+    otherwise None. Warn only (SIENTIAPDE-1995 decision 7): a value below the floor is kept;
+    an unparseable one is ignored by `simple_metrics()`, which keeps `frequency`.
+    """
+    if not frequency_alert:
+        return None
+    try:
+        # str(): a non-string value (e.g. a number typed into the doc) must warn, not raise
+        # AttributeError and fail the whole create_schedule_config activity.
+        seconds = parse_frequency(str(frequency_alert))
+    except ValueError:
+        return f'frequency_alert={frequency_alert!r} is not a valid frequency'
+    if seconds < MIN_FREQUENCY_ALERT_SECONDS:
+        return f'frequency_alert={frequency_alert} is below the 1m floor; kept as configured'
+    return None
+
+
 def minimal_retrain(config: dict[str, Any]):
     """
     Build minimal retrain configuration from pipeline config.

orchestrator/workflows/subworkflows/load_notification_package.py

@@ -68,18 +68,23 @@ class LoadNotificationPackage:
         # Load notification package from collection "notification_queue", using a
         # static filter
 
+        sending_configs = await sending_configs_handler
+
+        base_data_filter = self._compute_query_time_filter(
+            sending_configs, input_data['base_data_filter']
+        )
+
         notification_package = await workflow.start_local_activity_method(
             Activities.load_latest_data,
             {
                 **metadata,
                 'collection_name': 'notification_queue',
                 'last_data_timestamp': last_timestamp,
-                'base_data_filter': input_data['base_data_filter'],
+                'base_data_filter': base_data_filter,
             },
             start_to_close_timeout=timedelta(seconds=60),
             retry_policy=retry_policy,
         )
-        sending_configs = await sending_configs_handler
 
         if not sending_configs or not notification_package:
             return {
@@ -106,3 +111,92 @@ class LoadNotificationPackage:
             'notification_package': notification_package,
             'sending_configs': sending_configs,
         }
+
+    def _compute_query_time_filter(
+        self, sending_configs: list[dict[str, Any]], default_filter: dict[str, Any]
+    ) -> dict[str, Any]:
+        """
+        Compute the Mongo query-time filter from all active receiver groups.
+
+        - levels: union of every group's `levels`. If ANY active group has no
+          restriction (missing/empty `levels`), that group wants every level, so
+          the query can't restrict levels at all -> falls back to whatever level
+          restriction `default_filter` itself carries (e.g. ERROR-only for Alerts).
+
+          When every active group DOES restrict `levels`, the union REPLACES
+          `default_filter`'s own level restriction rather than ANDing with it —
+          this is deliberate, not an oversight: SIENTIAPDE-2011's design doc and
+          ticket text both call for Alerts' hard-coded `level: ERROR` to become
+          "whatever levels the active groups actually care about", falling back
+          to ERROR-only only when no group narrows further. Until SIENTIAPDE-2012
+          (per-group `levels` filtering) lands, this means a single active group
+          configuring `levels` to include WARNING/INFO widens the Alerts fetch for
+          every active group, not just that one — an accepted interim gap, not a
+          bug in this function.
+        - ignore_models_id: intersection of every group's `ignore_models_id`. A model
+          is only safe to drop from the fetch if EVERY active group ignores it.
+          A group with no `ignore_models_id` is treated as "ignores nothing" (empty
+          set), which naturally zeroes out the intersection via set math.
+        - model_ids (SIENTIAPDE-2149): union of every group's `model_ids` allow-list,
+          mirroring `levels`' own union-or-nothing logic. If ANY active group has no
+          restriction (missing/empty `model_ids`), that group wants every model, so
+          the query can't restrict to an allow-list at all. `model_ids` and
+          `ignore_models_id` are independent axes and can both narrow the same
+          `model_id` field at once (Mongo supports multiple operators on one field).
+        """
+        if not sending_configs:
+            return default_filter
+
+        levels_union = self._union_when_all_restrict(
+            [group.get('levels') for group in sending_configs]
+        )
+        if levels_union is not None:
+            query_filter = {**default_filter, 'level': {'$in': levels_union}}
+        else:
+            query_filter = dict(default_filter)
+
+        # `or []`, not a `.get(..., [])` default: a document with `ignore_models_id`
+        # explicitly set to `null` (no schema validation on this collection) makes
+        # `.get()` return None, and `set(None)` would raise TypeError.
+        sending_configs_ignore_models_id = [
+            set(group.get('ignore_models_id') or []) for group in sending_configs
+        ]
+        models_intersection = set.intersection(*sending_configs_ignore_models_id)
+
+        model_ids_union = self._union_when_all_restrict(
+            [group.get('model_ids') for group in sending_configs]
+        )
+
+        model_id_filter: dict[str, Any] = {}
+        if model_ids_union is not None:
+            model_id_filter['$in'] = model_ids_union
+        if models_intersection:
+            model_id_filter['$nin'] = sorted(models_intersection)
+        if model_id_filter:
+            query_filter['model_id'] = model_id_filter
+
+        return query_filter
+
+    @staticmethod
+    def _union_when_all_restrict(per_group_values: list[list[str] | None]) -> list[str] | None:
+        """
+        Union a list of per-group optional restrictions, but only when every group has one.
+
+        Shared by `levels` and `model_ids` (SIENTIAPDE-2149) in `_compute_query_time_filter`,
+        which both follow the same rule: if ANY active group is unrestricted on that axis
+        (missing/empty value), that group wants everything, so the union can't restrict the
+        query either — the caller falls back to no filter on that axis in that case (`None`).
+
+        Args:
+            per_group_values: Each active group's raw value for the axis being unioned
+                (e.g. `group.get('levels')` for every group).
+
+        Returns:
+            Sorted union of every group's values, or None if any group was unrestricted.
+        """
+        if not all(per_group_values):
+            return None
+        union: set[str] = set()
+        for values in per_group_values:
+            union.update(values or [])
+        return sorted(union)

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "orchestrator"
-version = "0.6.3"
+version = "0.8.1"
 description = "Sientia DataOps Orchestrator - ML Model Orchestration System"
 readme = "README.md"
 requires-python = ">=3.11"

requirements-local.txt

@@ -4,5 +4,5 @@ sqlalchemy
 redis
 pymongo
 jinja2
-git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.12.2
+git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.12.5
 prometheus-client

requirements.txt

@@ -4,5 +4,5 @@ sqlalchemy
 redis
 pymongo
 jinja2
-sientia_do>=1.12.2
+sientia_do>=1.12.5
 prometheus-client

tests/orchestrator/activities/test_activities.py

@@ -27,6 +27,7 @@ def test___init__(
         'connection_string': 'mongodb://localhost:27017',
         'database_name': 'test_db',
         'ttl_index_seconds': 3600,
+        'notification_queue_batch_size': 5000,
     }
 
     redis_config = {'host': 'localhost', 'port': 6379, 'username': 'admin', 'password': 'password'}
@@ -89,6 +90,7 @@ def test___init__(
         connection_string='mongodb://localhost:27017',
         database_name='test_db',
         ttl_index_seconds=3600,
+        notification_queue_batch_size=5000,
         logger=logger,
         notification_handler=notification_handler,
         metrics_controller=mock_metrics_controller.return_value,

tests/orchestrator/activities/test_formatters.py

@@ -716,7 +716,7 @@ def test_format_log_report(formatters):
         [
             {
                 'status': 'sent',
-                'timestamp': '2021-01-01',
+                'timestamp': '2021-01-01 00:00:00+0000',
                 'groups': ['test_receiver_group', 'test_receiver_group2'],
                 'message': 'test_message',
                 'level': 'test_level',
@@ -766,3 +766,310 @@ def test_filter_notification_reports(formatters):
             ],
         }
     }
+
+
+def test_filter_notification_reports_levels_restriction(formatters):
+    # SIENTIAPDE-2012: a group with `levels` set only receives notifications
+    # matching one of those levels.
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'level': 'ERROR',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'level': 'INFO',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['reports'], 'levels': ['ERROR']},
+        ],
+    }
+
+    response = formatters.filter_notification_reports(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['reports'],
+            'levels': ['ERROR'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_1',
+                    'notification_id': 'test_notification_id_1',
+                    'level': 'ERROR',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_reports_ignore_models_id_restriction(formatters):
+    # SIENTIAPDE-2012: a group with `ignore_models_id` set does not receive
+    # notifications from those models, but still receives everything else.
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['reports'],
+                'ignore_models_id': ['model_x'],
+            },
+        ],
+    }
+
+    response = formatters.filter_notification_reports(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['reports'],
+            'ignore_models_id': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_2',
+                    'notification_id': 'test_notification_id_2',
+                    'model_id': 'model_y',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_reports_model_ids_allow_list_restriction(formatters):
+    # SIENTIAPDE-2149: a group with `model_ids` set only receives notifications
+    # from those models, unlike `ignore_models_id` which is a deny-list.
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['reports'],
+                'model_ids': ['model_x'],
+            },
+        ],
+    }
+
+    response = formatters.filter_notification_reports(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['reports'],
+            'model_ids': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_1',
+                    'notification_id': 'test_notification_id_1',
+                    'model_id': 'model_x',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_reports_model_ids_and_ignore_models_id_together(formatters):
+    # SIENTIAPDE-2149: the allow-list and deny-list apply together (AND), not
+    # as alternatives -- a model must pass both checks.
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+            {
+                'trigger': 'test_trigger_3',
+                'notification_id': 'test_notification_id_3',
+                'model_id': 'model_z',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['reports'],
+                'model_ids': ['model_x', 'model_y'],
+                'ignore_models_id': ['model_x'],
+            },
+        ],
+    }
+
+    response = formatters.filter_notification_reports(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['reports'],
+            'model_ids': ['model_x', 'model_y'],
+            'ignore_models_id': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_2',
+                    'notification_id': 'test_notification_id_2',
+                    'model_id': 'model_y',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_reports_model_ids_explicit_none_does_not_raise(formatters):
+    # Same defensive concern as ignore_models_id: receiver_groups has no schema
+    # validation, so model_ids can be explicitly null instead of omitted.
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['reports'], 'model_ids': None},
+        ],
+    }
+
+    response = formatters.filter_notification_reports(input_data)
+
+    assert response['test_group_1']['notifications'] == input_data['notification_package']
+
+
+def test_filter_notification_reports_no_levels_or_ignore_models_id_passes_everything(
+    formatters,
+):
+    # SIENTIAPDE-2012: a group configuring neither field is unaffected -- same
+    # behavior as before this subtask (only `contents`/`ignore` apply).
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'level': 'ERROR',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'level': 'INFO',
+                'model_id': 'model_y',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['reports']},
+        ],
+    }
+
+    response = formatters.filter_notification_reports(input_data)
+
+    assert response['test_group_1']['notifications'] == input_data['notification_package']
+
+
+def test_filter_notification_reports_ignore_models_id_explicit_none_does_not_raise(
+    formatters,
+):
+    # receiver_groups has no schema validation -- ignore_models_id can be
+    # explicitly null (key present, value None) instead of omitted. Must not
+    # crash (see the equivalent SIENTIAPDE-2011 fix in load_notification_package).
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['reports'], 'ignore_models_id': None},
+        ],
+    }
+
+    response = formatters.filter_notification_reports(input_data)
+
+    assert response['test_group_1']['notifications'] == input_data['notification_package']
+
+
+def _schedule_config_with_frequency_alert(frequency_alert):
+    return {
+        'metadata': {},
+        'current_schedule_config': {'laborious': {}},
+        'schedule_config': {
+            'laborious': {
+                'm1-simple_metrics-60m': {
+                    'workflow_type': 'simple_metrics',
+                    'model_id': 'm1',
+                    'model_name': 'm1',
+                    'frequency': '1h',
+                    'frequency_alert': frequency_alert,
+                    'updated_at': '2026-10-07',
+                }
+            },
+            'scouter': {},
+        },
+    }
+
+
+def test_create_schedule_config_warns_on_frequency_alert_below_floor(formatters):
+    result = formatters.create_schedule_config(_schedule_config_with_frequency_alert('30s'))
+
+    # The configured value is kept.
+    assert result['to_create']['laborious']['m1-simple_metrics-60m']['frequency_alert'] == '30s'
+    formatters.send_notification.assert_called_once()
+    kwargs = formatters.send_notification.call_args.kwargs
+    assert kwargs['notification_id'] == 'SIMPLE_METRICS_FREQUENCY_ALERT_BELOW_FLOOR'
+    assert kwargs['level'] == NotificationLevel.WARNING
+    assert kwargs['metadata']['schedule_name'] == 'm1-simple_metrics-60m'
+
+
+def test_create_schedule_config_warns_on_frequency_alert_below_floor_on_update(formatters):
+    input_data = _schedule_config_with_frequency_alert('30s')
+    input_data['current_schedule_config'] = {'laborious': {'m1-simple_metrics-60m': '2026-10-06'}}
+
+    result = formatters.create_schedule_config(input_data)
+
+    assert 'm1-simple_metrics-60m' in result['to_update']['laborious']
+    formatters.send_notification.assert_called_once()
+    kwargs = formatters.send_notification.call_args.kwargs
+    assert kwargs['notification_id'] == 'SIMPLE_METRICS_FREQUENCY_ALERT_BELOW_FLOOR'
+
+
+def test_create_schedule_config_no_warning_at_or_above_floor(formatters):
+    formatters.create_schedule_config(_schedule_config_with_frequency_alert('5m'))
+
+    formatters.send_notification.assert_not_called()

tests/orchestrator/activities/test_mongo_db.py

@@ -1,7 +1,7 @@
 from datetime import UTC, datetime
 from unittest.mock import ANY, AsyncMock, MagicMock, patch
 
-from pytest import fixture
+from pytest import fixture, mark
 from sientia_do.notifications.models import NotificationLevel
 from sientia_do.temporal.constants import DATETIME_FORMAT_MS_WITH_TZ
 
@@ -31,13 +31,14 @@ def test___init__(mongo_mock):
     notification_handler = MagicMock()
     metrics_controller = AsyncMock()
 
-    MongoDB(
+    mongo = MongoDB(
         connection_string='mongodb://localhost:27017',
         database_name='test_db',
         ttl_index_seconds=3600,
         logger=logger,
         notification_handler=notification_handler,
         metrics_controller=metrics_controller,
+        notification_queue_batch_size=5000,
     )
 
     mongo_mock.assert_called_once_with(
@@ -47,6 +48,22 @@ def test___init__(mongo_mock):
         notification_handler=notification_handler,
         metrics_controller=metrics_controller,
     )
+    assert mongo.notification_queue_batch_size == 5000
+
+
+@patch('orchestrator.activities.mongo_db.MongoDBRepository')
+def test___init___notification_queue_batch_size_defaults_to_zero(mongo_mock):
+    """0 means no limit, matching the underlying find()'s own convention."""
+    mongo = MongoDB(
+        connection_string='mongodb://localhost:27017',
+        database_name='test_db',
+        ttl_index_seconds=3600,
+        logger=MagicMock(),
+        notification_handler=MagicMock(),
+        metrics_controller=AsyncMock(),
+    )
+
+    assert mongo.notification_queue_batch_size == 0
 
 
 @patch('orchestrator.activities.mongo_db.SientiaMonitoring')
@@ -248,6 +265,41 @@ def test_update_pipelines_timestamps_success(now_mock, mongo_db):
     )
 
 
+@patch('orchestrator.activities.mongo_db.now')
+@mark.parametrize(
+    'pipelines', [[], [{'schedule_name': 'test1', 'namespace': 'test1', 'success': False}]]
+)
+def test_update_pipelines_timestamps_all_failed_stamps_nothing(now_mock, mongo_db, pipelines):
+    input_data = {'updated_pipelines': pipelines}
+    mongo_db.mongo_db_repository.update_many = MagicMock()
+
+    mongo_db.update_pipelines_timestamps(input_data)
+
+    # An empty filter would match and stamp every tracked schedule, so the failed one
+    # would never be retried.
+    mongo_db.mongo_db_repository.update_many.assert_not_called()
+
+
+@patch('orchestrator.activities.mongo_db.now')
+def test_update_pipelines_timestamps_mixed_stamps_only_successes(now_mock, mongo_db):
+    input_data = {
+        'updated_pipelines': [
+            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
+            {'schedule_name': 'test2', 'namespace': 'test2', 'success': False},
+        ]
+    }
+    mongo_db.mongo_db_repository.update_many = MagicMock()
+
+    mongo_db.update_pipelines_timestamps(input_data)
+
+    mongo_db.mongo_db_repository.update_many.assert_called_once_with(
+        'orchestrated_schedules',
+        {'$or': [{'schedule_name': 'test1', 'namespace': 'test1'}]},
+        {'$set': {'updated_at': now_mock.return_value}},
+        {},
+    )
+
+
 @patch('orchestrator.activities.mongo_db.now')
 def test_update_pipelines_timestamps_failure(now_mock, mongo_db):
     input_data = {
@@ -345,6 +397,38 @@ def test_delete_pipelines_timestamps_success(now_mock, mongo_db):
     )
 
 
+@mark.parametrize(
+    'pipelines', [[], [{'schedule_name': 'test1', 'namespace': 'test1', 'success': False}]]
+)
+def test_delete_pipelines_timestamps_all_failed_deletes_nothing(mongo_db, pipelines):
+    input_data = {'deleted_pipelines': pipelines}
+    mongo_db.mongo_db_repository.delete_many = MagicMock()
+
+    mongo_db.delete_pipelines_timestamps(input_data)
+
+    # An empty filter would wipe orchestrated_schedules; the next tick's normalize_schedules
+    # would then delete and recreate every orchestrated Temporal schedule.
+    mongo_db.mongo_db_repository.delete_many.assert_not_called()
+
+
+def test_delete_pipelines_timestamps_mixed_deletes_only_successes(mongo_db):
+    input_data = {
+        'deleted_pipelines': [
+            {'schedule_name': 'test1', 'namespace': 'test1', 'success': True},
+            {'schedule_name': 'test2', 'namespace': 'test2', 'success': False},
+        ]
+    }
+    mongo_db.mongo_db_repository.delete_many = MagicMock()
+
+    mongo_db.delete_pipelines_timestamps(input_data)
+
+    mongo_db.mongo_db_repository.delete_many.assert_called_once_with(
+        'orchestrated_schedules',
+        {'$or': [{'schedule_name': 'test1', 'namespace': 'test1'}]},
+        {},
+    )
+
+
 @patch('orchestrator.activities.mongo_db.now')
 def test_delete_pipelines_timestamps_failure(now_mock, mongo_db):
     input_data = {
@@ -480,6 +564,8 @@ def test_load_latest_data_none_last_data_timestamp(mongo_db):
         'test_collection',
         {'level': 'ERROR'},
         {'workflow_name': 'test_pipeline', 'schedule_name': 'test_schedule'},
+        limit=0,
+        sort=[('timestamp', 1)],
     )
 
     assert result == [
@@ -487,6 +573,29 @@ def test_load_latest_data_none_last_data_timestamp(mongo_db):
     ]
 
 
+def test_load_latest_data_passes_configured_batch_size_as_limit(mongo_db):
+    """SIENTIAPDE-2013: the configured notification_queue_batch_size caps the query."""
+    mongo_db.notification_queue_batch_size = 100
+    mongo_db.mongo_db_repository.find = MagicMock(return_value=[])
+
+    mongo_db.load_latest_data(
+        {
+            'metadata': {'workflow_name': 'test_pipeline', 'schedule_name': 'test_schedule'},
+            'collection_name': 'test_collection',
+            'last_data_timestamp': None,
+            'base_data_filter': {'level': 'ERROR'},
+        }
+    )
+
+    mongo_db.mongo_db_repository.find.assert_called_once_with(
+        'test_collection',
+        {'level': 'ERROR'},
+        {'workflow_name': 'test_pipeline', 'schedule_name': 'test_schedule'},
+        limit=100,
+        sort=[('timestamp', 1)],
+    )
+
+
 def test_load_latest_data_not_none_last_data_timestamp(mongo_db):
     """Test load_latest_data compares native datetime against native datetime in $gt"""
 
@@ -516,6 +625,8 @@ def test_load_latest_data_not_none_last_data_timestamp(mongo_db):
             'timestamp': {'$gt': datetime(2023, 1, 1, 12, 0, 0, tzinfo=UTC)},
         },
         {'workflow_name': 'test_pipeline', 'schedule_name': 'test_schedule'},
+        limit=0,
+        sort=[('timestamp', 1)],
     )
 
     assert result == [
@@ -529,7 +640,7 @@ def test_load_latest_data_run_boundary_no_skip_or_duplicate(mongo_db):
 
     boundary_timestamp = datetime(2023, 1, 1, 12, 0, 0, tzinfo=UTC)
 
-    def fake_find(collection_name, data_filter, metadata):
+    def fake_find(collection_name, data_filter, metadata, limit=0, sort=None):
         gt = data_filter.get('timestamp', {}).get('$gt')
         all_docs = [
             {

tests/orchestrator/activities/test_slot_manager.py

@@ -389,6 +389,293 @@ def test_filter_notification_alerts(slot_manager):
     }
 
 
+def test_filter_notification_alerts_levels_restriction(slot_manager):
+    # SIENTIAPDE-2012: a group with `levels` set only receives notifications
+    # matching one of those levels.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None, None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'level': 'ERROR',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'level': 'WARNING',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['core_alerts'], 'levels': ['ERROR']},
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['core_alerts'],
+            'levels': ['ERROR'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_1',
+                    'notification_id': 'test_notification_id_1',
+                    'level': 'ERROR',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_alerts_ignore_models_id_restriction(slot_manager):
+    # SIENTIAPDE-2012: a group with `ignore_models_id` set does not receive
+    # notifications from those models, but still receives everything else.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None, None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['core_alerts'],
+                'ignore_models_id': ['model_x'],
+            },
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['core_alerts'],
+            'ignore_models_id': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_2',
+                    'notification_id': 'test_notification_id_2',
+                    'model_id': 'model_y',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_alerts_model_ids_allow_list_restriction(slot_manager):
+    # SIENTIAPDE-2149: a group with `model_ids` set only receives notifications
+    # from those models, unlike `ignore_models_id` which is a deny-list.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None, None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['core_alerts'],
+                'model_ids': ['model_x'],
+            },
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['core_alerts'],
+            'model_ids': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_1',
+                    'notification_id': 'test_notification_id_1',
+                    'model_id': 'model_x',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_alerts_model_ids_and_ignore_models_id_together(slot_manager):
+    # SIENTIAPDE-2149: the allow-list and deny-list apply together (AND), not
+    # as alternatives -- a model must pass both checks.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None, None, None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'model_id': 'model_y',
+            },
+            {
+                'trigger': 'test_trigger_3',
+                'notification_id': 'test_notification_id_3',
+                'model_id': 'model_z',
+            },
+        ],
+        'sending_configs': [
+            {
+                'group_name': 'test_group_1',
+                'contents': ['core_alerts'],
+                'model_ids': ['model_x', 'model_y'],
+                'ignore_models_id': ['model_x'],
+            },
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    # model_x: allowed but ignored -> excluded. model_y: allowed and not
+    # ignored -> included. model_z: not in the allow-list -> excluded.
+    assert response == {
+        'test_group_1': {
+            'group_name': 'test_group_1',
+            'contents': ['core_alerts'],
+            'model_ids': ['model_x', 'model_y'],
+            'ignore_models_id': ['model_x'],
+            'notifications': [
+                {
+                    'trigger': 'test_trigger_2',
+                    'notification_id': 'test_notification_id_2',
+                    'model_id': 'model_y',
+                },
+            ],
+        },
+    }
+
+
+def test_filter_notification_alerts_model_ids_explicit_none_does_not_raise(slot_manager):
+    # Same defensive concern as ignore_models_id: receiver_groups has no schema
+    # validation, so model_ids can be explicitly null instead of omitted.
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['core_alerts'], 'model_ids': None},
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response['test_group_1']['notifications'] == input_data['notification_package']
+
+
+def test_filter_notification_alerts_no_levels_or_ignore_models_id_passes_everything(
+    slot_manager,
+):
+    # SIENTIAPDE-2012: a group configuring neither field is unaffected -- same
+    # behavior as before this subtask (only `contents`/`ignore` apply).
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None, None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'level': 'ERROR',
+                'model_id': 'model_x',
+            },
+            {
+                'trigger': 'test_trigger_2',
+                'notification_id': 'test_notification_id_2',
+                'level': 'INFO',
+                'model_id': 'model_y',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['core_alerts']},
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response['test_group_1']['notifications'] == input_data['notification_package']
+
+
+def test_filter_notification_alerts_ignore_models_id_explicit_none_does_not_raise(
+    slot_manager,
+):
+    # receiver_groups has no schema validation -- ignore_models_id can be
+    # explicitly null (key present, value None) instead of omitted. Must not
+    # crash (see the equivalent SIENTIAPDE-2011 fix in load_notification_package).
+    slot_manager.redis_repository.get = MagicMock(side_effect=[None])
+
+    input_data = {
+        **metadata,
+        'notification_package': [
+            {
+                'trigger': 'test_trigger_1',
+                'notification_id': 'test_notification_id_1',
+                'model_id': 'model_x',
+            },
+        ],
+        'sending_configs': [
+            {'group_name': 'test_group_1', 'contents': ['core_alerts'], 'ignore_models_id': None},
+        ],
+        'notification_ttl': 300,
+        'mail_type': 'test_mail_type',
+    }
+
+    response = slot_manager.filter_notification_alerts(input_data)
+
+    assert response['test_group_1']['notifications'] == input_data['notification_package']
+
+
 def test_store_notification_cache(slot_manager):
     """Test store_notification_cache"""
     test_data = {

tests/orchestrator/utils/test_connectors_config.py

@@ -64,10 +64,12 @@ def test_build_mongo_db_config_with_env_vars():
     environ['MONGODB_URL'] = 'localhost:27018'
     environ['MONGODB_DATABASE_NAME'] = 'test_db'
     environ['MONGODB_TTL_INDEX_HOURS'] = '2'
+    environ['NOTIFICATION_QUEUE_BATCH_SIZE'] = '2000'
     assert build_mongodb_config() == {
         'connection_string': 'mongodb://sientia1:sientia1@localhost:27018',
         'database_name': 'test_db',
         'ttl_index_seconds': 7200,
+        'notification_queue_batch_size': 2000,
     }
 
 
@@ -77,10 +79,12 @@ def test_build_mongo_db_config_with_defaults():
     environ.pop('MONGODB_DATABASE_NAME', None)
     environ.pop('MONGODB_URL', None)
     environ.pop('MONGODB_TTL_INDEX_HOURS', None)
+    environ.pop('NOTIFICATION_QUEUE_BATCH_SIZE', None)
     assert build_mongodb_config() == {
         'connection_string': 'mongodb://root:wKZDbMNU1c@localhost:27018',
         'database_name': 'sientia',
         'ttl_index_seconds': 3600,
+        'notification_queue_batch_size': 0,
     }
 
 

tests/orchestrator/utils/test_orchestrator_functions.py

@@ -1,10 +1,13 @@
 from unittest.mock import call, patch
 
+from pytest import mark
+
 from orchestrator.utils.orchestrator_functions import (
     base_scouter,
     build_tag_config,
     common_config,
     drift,
+    frequency_alert_warning,
     gather_read_tags,
     minimal_retrain,
     overlap_filter_config,
@@ -110,6 +113,7 @@ def test_simple_metrics():
         'target_table_name': 'simple_metrics',
         'interval_minutes': 120,
         'metrics': ['rmse', 'mse'],
+        'frequency_alert': None,
         'execution_timeout_seconds': 300,
         'task_timeout_seconds': 300,
         'runtime': 'legacy',
@@ -117,6 +121,61 @@ def test_simple_metrics():
     assert result == expected
 
 
+def _simple_metrics_config(**fields):
+    return {
+        'workflow_type': 'simple_metrics',
+        'schedule_name': 'm1-simple_metrics-60m',
+        'model_id': 'm1',
+        'model': {'name': 'm1'},
+        'frequency': '1h',
+        **fields,
+    }
+
+
+def test_simple_metrics_escalated_uses_frequency_alert():
+    config = _simple_metrics_config(
+        frequency_alert='5m', adaptive_state={'escalated': True, 'healthy_streak': 0}
+    )
+    result = simple_metrics(config)
+    assert result['frequency'] == '5m'
+    assert result['frequency_alert'] == '5m'
+
+
+def test_simple_metrics_not_escalated_keeps_frequency():
+    config = _simple_metrics_config(
+        frequency_alert='5m', adaptive_state={'escalated': False, 'healthy_streak': 2}
+    )
+    assert simple_metrics(config)['frequency'] == '1h'
+
+
+def test_simple_metrics_escalated_without_frequency_alert_keeps_frequency():
+    # Removing frequency_alert turns adaptive mode off even if a stale state says escalated.
+    config = _simple_metrics_config(adaptive_state={'escalated': True, 'healthy_streak': 0})
+    assert simple_metrics(config)['frequency'] == '1h'
+
+
+def test_simple_metrics_frequency_alert_without_state_keeps_frequency():
+    assert simple_metrics(_simple_metrics_config(frequency_alert='5m'))['frequency'] == '1h'
+
+
+def test_simple_metrics_escalated_unparseable_frequency_alert_keeps_frequency():
+    # An unparseable frequency would make the Temporal schedule create/update fail.
+    config = _simple_metrics_config(frequency_alert='5min', adaptive_state={'escalated': True})
+    assert simple_metrics(config)['frequency'] == '1h'
+
+
+def test_simple_metrics_escalated_below_floor_frequency_alert_is_applied():
+    # Decision 7: below the floor only warns, the value is kept.
+    config = _simple_metrics_config(frequency_alert='30s', adaptive_state={'escalated': True})
+    assert simple_metrics(config)['frequency'] == '30s'
+
+
+@mark.parametrize('adaptive_state', ['yes', {'escalated': 'false'}])
+def test_simple_metrics_malformed_adaptive_state_keeps_frequency(adaptive_state):
+    config = _simple_metrics_config(frequency_alert='5m', adaptive_state=adaptive_state)
+    assert simple_metrics(config)['frequency'] == '1h'
+
+
 def test_minimal_retrain():
     config = {
         'workflow_type': 'minimal_retrain',
@@ -762,3 +821,19 @@ def test_pi_web_api_scouter_with_no_timeout():
         'runtime': 'legacy',
     }
     assert result == expected
+
+
+@mark.parametrize(
+    'frequency_alert, warns',
+    [
+        (None, False),
+        ('5m', False),
+        ('1m', False),  # the floor itself is allowed
+        ('60s', False),
+        ('30s', True),
+        ('5min', True),  # unparseable
+        (5, True),  # not a string: must warn, never raise
+    ],
+)
+def test_frequency_alert_warning(frequency_alert, warns):
+    assert (frequency_alert_warning(frequency_alert) is not None) is warns

tests/orchestrator/workflows/subworkflows/test_load_notification_package.py

@@ -32,16 +32,19 @@ async def test_run(workflow_mock, load_notification_package):
         'base_data_filter': {'level': 'ERROR'},
     }
 
+    # Await order is now: last_timestamp -> sending_configs -> notification_package
+    # (fixed ordering bug: sending_configs must resolve before load_latest_data is
+    # called, so its levels/ignore_models_id can feed the query-time filter).
     workflow_mock.start_local_activity_method.side_effect = [
         '2023-01-01 12:00:00',
         [
             {
-                'id': '1',
+                'id_r': '1',
             }
         ],
         [
             {
-                'id_r': '1',
+                'id': '1',
             }
         ],
     ]
@@ -158,7 +161,9 @@ async def test_run_has_data(workflow_mock, load_notification_package):
         'mail_type': 'test_mail_type',
     }
 
-    workflow_mock.start_local_activity_method.side_effect = ['2023-01-01 12:00:00', ['data'], []]
+    # sending_configs (awaited 2nd) is empty here on purpose, to exercise the
+    # early-return branch below regardless of notification_package having data.
+    workflow_mock.start_local_activity_method.side_effect = ['2023-01-01 12:00:00', [], ['data']]
 
     output = await load_notification_package.run(input_data)
 
@@ -169,3 +174,162 @@ async def test_run_has_data(workflow_mock, load_notification_package):
     }
 
     workflow_mock.start_activity_method.assert_not_called()
+
+
+def test_compute_query_time_filter_empty_active_groups(load_notification_package):
+    default_filter = {'level': 'ERROR'}
+
+    result = load_notification_package._compute_query_time_filter([], default_filter)
+
+    assert result == default_filter
+
+
+def test_compute_query_time_filter_mixed_groups_no_levels_restriction(load_notification_package):
+    # One active group restricts levels/ignore_models_id, the other has neither set
+    # (the common case today) -> that group wants everything, so no levels
+    # restriction can be applied at query time; falls back to default_filter.
+    sending_configs = [
+        {'group_name': 'group_a', 'levels': ['ERROR'], 'ignore_models_id': ['model_a']},
+        {'group_name': 'group_b'},
+    ]
+    default_filter = {'level': 'ERROR'}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert result == {'level': 'ERROR'}
+
+
+def test_compute_query_time_filter_union_replaces_default_level_restriction(
+    load_notification_package,
+):
+    # default_filter already restricts level (e.g. Alerts' ERROR-only default),
+    # but every active group specifies levels of its own -> the union REPLACES
+    # default_filter's level restriction, it does not AND with it. This is
+    # deliberate per SIENTIAPDE-2011's design doc and ticket text ("Alerts'
+    # hard-coded level: ERROR should become whatever levels the active groups
+    # actually care about"), not a bug -- this test locks the behavior in so a
+    # future "fix" reintroducing AND semantics fails loudly instead of passing
+    # silently.
+    sending_configs = [
+        {'group_name': 'group_a', 'levels': ['WARNING'], 'ignore_models_id': []},
+    ]
+    default_filter = {'level': 'ERROR'}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert result == {'level': {'$in': ['WARNING']}}
+
+
+def test_compute_query_time_filter_model_ignored_by_only_one_group_not_excluded(
+    load_notification_package,
+):
+    # model_a is ignored by group_a but group_b still wants it -> must stay in
+    # the fetch (the per-group filter is what drops it for group_a specifically).
+    sending_configs = [
+        {'group_name': 'group_a', 'levels': ['ERROR'], 'ignore_models_id': ['model_a']},
+        {'group_name': 'group_b', 'levels': ['ERROR'], 'ignore_models_id': []},
+    ]
+    default_filter = {}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert result == {'level': {'$in': ['ERROR']}}
+    assert 'model_id' not in result
+
+
+def test_compute_query_time_filter_model_ids_union_when_all_groups_restrict(
+    load_notification_package,
+):
+    # SIENTIAPDE-2149: every active group sets a model_ids allow-list -> the
+    # union applies at query time, mirroring how `levels` already behaves.
+    sending_configs = [
+        {'group_name': 'group_a', 'model_ids': ['model_a']},
+        {'group_name': 'group_b', 'model_ids': ['model_b']},
+    ]
+    default_filter = {}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert result == {'model_id': {'$in': ['model_a', 'model_b']}}
+
+
+def test_compute_query_time_filter_model_ids_no_restriction_when_any_group_missing(
+    load_notification_package,
+):
+    # One active group has no model_ids restriction -> it wants every model,
+    # so the query can't restrict to an allow-list at all.
+    sending_configs = [
+        {'group_name': 'group_a', 'model_ids': ['model_a']},
+        {'group_name': 'group_b'},
+    ]
+    default_filter = {}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert 'model_id' not in result
+
+
+def test_compute_query_time_filter_model_ids_union_and_ignore_models_id_intersection_together(
+    load_notification_package,
+):
+    # SIENTIAPDE-2149: the allow-list union ($in) and the ignore-list
+    # intersection ($nin) are independent axes and can both narrow the same
+    # model_id field at once.
+    sending_configs = [
+        {
+            'group_name': 'group_a',
+            'model_ids': ['model_a', 'model_b'],
+            'ignore_models_id': ['model_a'],
+        },
+        {
+            'group_name': 'group_b',
+            'model_ids': ['model_b', 'model_c'],
+            'ignore_models_id': ['model_a'],
+        },
+    ]
+    default_filter = {}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert result == {
+        'model_id': {'$in': ['model_a', 'model_b', 'model_c'], '$nin': ['model_a']},
+    }
+
+
+def test_compute_query_time_filter_union_levels_and_intersection_models(
+    load_notification_package,
+):
+    # Every active group restricts levels -> union of levels applies.
+    # model_a is ignored by every active group -> safe to exclude at query time.
+    sending_configs = [
+        {'group_name': 'group_a', 'levels': ['ERROR'], 'ignore_models_id': ['model_a', 'model_b']},
+        {'group_name': 'group_b', 'levels': ['WARNING', 'ERROR'], 'ignore_models_id': ['model_a']},
+    ]
+    default_filter = {}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert result == {
+        'level': {'$in': ['ERROR', 'WARNING']},
+        'model_id': {'$nin': ['model_a']},
+    }
+
+
+def test_compute_query_time_filter_ignore_models_id_explicit_none_does_not_raise(
+    load_notification_package,
+):
+    # receiver_groups has no schema validation, so a document can have
+    # ignore_models_id explicitly set to null (key present, value None) rather
+    # than omitted. `group.get('ignore_models_id', [])` would still return None
+    # in that case (the default only applies when the key is missing), and
+    # `set(None)` raises TypeError -- must not crash the whole activity.
+    sending_configs = [
+        {'group_name': 'group_a', 'levels': ['ERROR'], 'ignore_models_id': None},
+        {'group_name': 'group_b', 'levels': ['ERROR'], 'ignore_models_id': ['model_a']},
+    ]
+    default_filter = {}
+
+    result = load_notification_package._compute_query_time_filter(sending_configs, default_filter)
+
+    assert result == {'level': {'$in': ['ERROR']}}
+    assert 'model_id' not in result