sientia-dataops-orchestrator_temporal/e2e/scenarios.md

E2E Scenario Documentation — Orchestrator

Functional reference for orchestrator E2E scenarios. Tests live under e2e/, use @pytest.mark.e2e, and run with:

pytest e2e/ --override-ini testpaths=e2e -m e2e

Every section below maps 1:1 to an actual test function (named in each section, or as a bullet under a Source: heading) — this file describes what the suite verifies today, not planned or aspirational coverage. Adding a scenario here without a matching test (or vice versa) is a doc bug; fix the mismatch instead of leaving it for later.

Execution context

• MongoDB, Redis, PostgreSQL: testcontainers (session-scoped).
• SMTP: in-process aiosmtpd (e2e/smtp_test_server.py).
• Temporal: WorkflowEnvironment.start_local() with stub workers on scouter / laborious.
• Production code under orchestrator/** is not mocked; only Logger may be a MagicMock.

---

1. Orchestrator workflow

Source: e2e/test_orchestrator_main_workflow.py

1.1.1 Happy path

Pipelines in Mongo → Temporal schedule created in the laborious namespace, orchestrated_schedules entries written.

1.3.1 Create-only

New pipeline only → Temporal schedule created for it.

1.3.3 Delete-only

Pipeline removed from Mongo → previously-created Temporal schedule deleted.

1.5.1 Empty pipelines

No active pipelines → no orchestrated_schedules documents written, even with a pre-existing unrelated Temporal schedule present.

1.6.1 TTL index bootstrap

First run creates TTL index on notification collection used by scouter pipelines.

1.7.1 Simple metrics parallel windows

Two simple_metrics pipelines for the same model_id (60 min and 1440 min windows) → one Temporal schedule per window, each with its own frequency as interval.

1.7.2 Adaptive frequency escalation/recovery round trip

Laborious-style write (adaptive_state.escalated=true + BSON-datetime updated_at) on one window → next run switches only that schedule to frequency_alert; the recovery write switches it back to frequency.

1.8.1 Drift window and chunk_period

Two drift pipelines on frequency='1m': one without interval_minutes and with an invalid chunk_period='hour' → workflow input interval=1 (frequency, no overlap), chunk_period='min' and exactly one DRIFT_CHUNK_PERIOD_INVALID notification; one with interval_minutes=60 and chunk_period='s' → both forwarded unchanged, no warning (SIENTIAPDE-2212).

---

2. Alerts workflow

Source: e2e/test_alerts_main_workflow.py

A.1.1 Happy path

ERROR notification → one SMTP message, one log_report row, Redis cache key.

A.1.2 TTL duplicate suppression

Second run with same data and cache seeded → no extra email or log row.

A.2.1 Group filtering

Receiver group levels / ignore_models_id honored, both halves now covered: the query-time union/intersection across active groups (SIENTIAPDE-2011, see LoadNotificationPackage below) and the per-group precise slice (SIENTIAPDE-2012) — a group only receives notifications matching its own levels/ignore_models_id, even when the query-time filter widened the fetch to satisfy another active group's broader configuration.

A.2.2 Model_id allow-list

Receiver group model_ids (SIENTIAPDE-2149) honored: a group configuring it only receives notifications from those models, even when another active group with no such restriction widens the query-time fetch to include every model.

A.3.1 Empty queue

No notifications → no SMTP, no Postgres row.

---

3. Reports workflow

Source: e2e/test_reports_main_workflow.py

R.1.1 Happy path

Mixed ERROR/WARNING/INFO → one HTML email with all section headings.

R.1.2 Per-level rendering

Single-level notifications → only matching section in HTML body.

R.2.1 Empty queue

No notifications → no SMTP, no Postgres row.

---

4. Subworkflows

LoadNotificationPackage — e2e/test_subworkflow_load_notification_package.py

• No prior Redis timestamp → all matching notifications returned, max timestamp stored.
• Prior timestamp → only newer notifications returned.
• Query-time filter (SIENTIAPDE-2011) → union of active groups' levels, intersection of their ignore_models_id; a model ignored by only one of several active groups is not excluded from the fetch.
• Model_id allow-list union at query time (SIENTIAPDE-2149) → when every active group sets model_ids, the fetch is restricted to the union of every group's allow-list; a model outside that union is never fetched at all, even for a group that would otherwise want it via a different axis.
• Empty Mongo → no Redis timestamp write.
• Batch size cap (SIENTIAPDE-2013) → with 5 pending notifications and a batch size of 2, each run returns exactly the next-oldest 2 (sorted ascending by timestamp), and the Redis cursor advances only to what was actually returned, so three successive runs drain the backlog in order instead of skipping anything.
• Batch size 0 means unlimited (SIENTIAPDE-2013) → the configured default (0) does not cap the query; 12 pending notifications all come back in one run, verified against a real MongoDB instance rather than assumed from pymongo's documented convention.

ProcessNotifications — e2e/test_subworkflow_process_notifications.py

• Full round trip: HTML → SMTP → log_report in Postgres.
• Empty receiver groups → {}.