sientia-dataops-orchestrator_temporal/e2e/scenarios.md

E2E Scenario Documentation — Orchestrator

Functional reference for orchestrator E2E scenarios. Tests live under e2e/, use @pytest.mark.e2e, and run with:

pytest e2e/ --override-ini testpaths=e2e -m e2e

Execution context

• MongoDB, Redis, PostgreSQL: testcontainers (session-scoped).
• SMTP: in-process aiosmtpd (e2e/smtp_test_server.py).
• Temporal: WorkflowEnvironment.start_local() with stub workers on scouter / laborious.
• Production code under orchestrator/** is not mocked; only Logger may be a MagicMock.

---

1. Orchestrator workflow

Source: e2e/test_orchestrator_main_workflow.py

1.1.1 Happy path

Pipelines in Mongo → schedules created in correct namespace/task queue, Redis slots written, orchestrated_schedules updated.

1.2.1 No-op tick

Mongo, Redis, and Temporal already match desired state → no new schedules or slot writes.

1.3.1 Create-only

New pipeline only → schedules created, timestamps inserted.

1.3.2 Update-only

Existing pipeline with newer updated_at → schedule updated in Temporal.

1.3.3 Delete-only

Pipeline removed from Mongo → schedule deleted from Temporal.

1.4.1 Conflict ordering

Pipeline update and slot delete on same OPC server → slot insert before delete (production ordering).

1.5.1 Empty pipelines

No active pipelines → orphan schedules removed, no new orchestration writes.

1.6.1 TTL index bootstrap

First run creates TTL index on notification collection used by scouter pipelines.

---

2. Alerts workflow

Source: e2e/test_alerts_main_workflow.py

A.1.1 Happy path

ERROR notification → one SMTP message, one log_report row, Redis cache key.

A.1.2 TTL duplicate suppression

Second run with same data and cache seeded → no extra email or log row.

A.1.3 Persistent escalation

Alert past notification_ttl with cache cleared → new email sent.

A.2.1 Group filtering

Receiver group levels / ignore_models honored.

A.3.1 Empty queue

No notifications → no SMTP, no Postgres row.

---

3. Reports workflow

Source: e2e/test_reports_main_workflow.py

R.1.1 Happy path

Mixed ERROR/WARNING/INFO → one HTML email with all section headings.

R.1.2 Per-level rendering

Single-level notifications → only matching section in HTML body.

R.2.1 Empty queue

No notifications → no SMTP, no Postgres row.

---

4. Subworkflows

LoadNotificationPackage — e2e/test_subworkflow_load_notification_package.py

• No prior Redis timestamp → all matching notifications returned, max timestamp stored.
• Prior timestamp → only newer notifications returned.
• Empty Mongo → no Redis timestamp write.

ProcessNotifications — e2e/test_subworkflow_process_notifications.py

• Full round trip: HTML → SMTP → log_report in Postgres.
• Empty receiver groups → {}.