d46bd55cb4
Also fixes the test_activities fixture, which built mongodb_config by hand and was missing the new required notification_queue_batch_size key -- every e2e test would have hit a KeyError without this. New test seeds 5 notifications with a batch size of 2 and confirms three successive runs return them oldest-first in order (n1,n2 / n3,n4 / n5), proving the cursor only advances to what was actually returned rather than skipping documents a capped query didn't fetch. Verified against real testcontainers Mongo/Redis/Temporal, not mocks. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3.7 KiB
E2E Scenario Documentation — Orchestrator
Functional reference for orchestrator E2E scenarios. Tests live under e2e/, use @pytest.mark.e2e, and run with:
pytest e2e/ --override-ini testpaths=e2e -m e2e
Execution context
- MongoDB, Redis, PostgreSQL: testcontainers (session-scoped).
- SMTP: in-process
aiosmtpd(e2e/smtp_test_server.py). - Temporal:
WorkflowEnvironment.start_local()with stub workers onscouter/laborious. - Production code under
orchestrator/**is not mocked; onlyLoggermay be aMagicMock.
1. Orchestrator workflow
Source: e2e/test_orchestrator_main_workflow.py
1.1.1 Happy path
Pipelines in Mongo → schedules created in correct namespace/task queue, Redis slots written, orchestrated_schedules updated.
1.2.1 No-op tick
Mongo, Redis, and Temporal already match desired state → no new schedules or slot writes.
1.3.1 Create-only
New pipeline only → schedules created, timestamps inserted.
1.3.2 Update-only
Existing pipeline with newer updated_at → schedule updated in Temporal.
1.3.3 Delete-only
Pipeline removed from Mongo → schedule deleted from Temporal.
1.4.1 Conflict ordering
Pipeline update and slot delete on same OPC server → slot insert before delete (production ordering).
1.5.1 Empty pipelines
No active pipelines → orphan schedules removed, no new orchestration writes.
1.6.1 TTL index bootstrap
First run creates TTL index on notification collection used by scouter pipelines.
2. Alerts workflow
Source: e2e/test_alerts_main_workflow.py
A.1.1 Happy path
ERROR notification → one SMTP message, one log_report row, Redis cache key.
A.1.2 TTL duplicate suppression
Second run with same data and cache seeded → no extra email or log row.
A.1.3 Persistent escalation
Alert past notification_ttl with cache cleared → new email sent.
A.2.1 Group filtering
Receiver group levels / ignore_models_id honored. The query-time half
(union of levels, intersection of ignore_models_id across active groups)
is covered at the subworkflow level — see LoadNotificationPackage below.
The per-group half (each group only receiving its own precise slice) is not
yet implemented in production code (SIENTIAPDE-2012) and so is not yet
exercised here.
A.3.1 Empty queue
No notifications → no SMTP, no Postgres row.
3. Reports workflow
Source: e2e/test_reports_main_workflow.py
R.1.1 Happy path
Mixed ERROR/WARNING/INFO → one HTML email with all section headings.
R.1.2 Per-level rendering
Single-level notifications → only matching section in HTML body.
R.2.1 Empty queue
No notifications → no SMTP, no Postgres row.
4. Subworkflows
LoadNotificationPackage — e2e/test_subworkflow_load_notification_package.py
- No prior Redis timestamp → all matching notifications returned, max timestamp stored.
- Prior timestamp → only newer notifications returned.
- Query-time filter (SIENTIAPDE-2011) → union of active groups'
levels, intersection of theirignore_models_id; a model ignored by only one of several active groups is not excluded from the fetch. - Empty Mongo → no Redis timestamp write.
- Batch size cap (SIENTIAPDE-2013) → with 5 pending notifications and a batch size of 2, each run returns exactly the next-oldest 2 (sorted ascending by timestamp), and the Redis cursor advances only to what was actually returned, so three successive runs drain the backlog in order instead of skipping anything.
ProcessNotifications — e2e/test_subworkflow_process_notifications.py
- Full round trip: HTML → SMTP →
log_reportin Postgres. - Empty receiver groups →
{}.