112 lines
2.8 KiB
Markdown
112 lines
2.8 KiB
Markdown
# E2E Scenario Documentation — Orchestrator
|
|
|
|
Functional reference for orchestrator E2E scenarios. Tests live under `e2e/`, use `@pytest.mark.e2e`, and run with:
|
|
|
|
```bash
|
|
pytest e2e/ --override-ini testpaths=e2e -m e2e
|
|
```
|
|
|
|
## Execution context
|
|
|
|
- MongoDB, Redis, PostgreSQL: testcontainers (session-scoped).
|
|
- SMTP: in-process `aiosmtpd` (`e2e/smtp_test_server.py`).
|
|
- Temporal: `WorkflowEnvironment.start_local()` with stub workers on `scouter` / `laborious`.
|
|
- Production code under `orchestrator/**` is not mocked; only `Logger` may be a `MagicMock`.
|
|
|
|
---
|
|
|
|
## 1. Orchestrator workflow
|
|
|
|
Source: `e2e/test_orchestrator_main_workflow.py`
|
|
|
|
### 1.1.1 Happy path
|
|
|
|
Pipelines in Mongo → schedules created in correct namespace/task queue, Redis slots written, `orchestrated_schedules` updated.
|
|
|
|
### 1.2.1 No-op tick
|
|
|
|
Mongo, Redis, and Temporal already match desired state → no new schedules or slot writes.
|
|
|
|
### 1.3.1 Create-only
|
|
|
|
New pipeline only → schedules created, timestamps inserted.
|
|
|
|
### 1.3.2 Update-only
|
|
|
|
Existing pipeline with newer `updated_at` → schedule updated in Temporal.
|
|
|
|
### 1.3.3 Delete-only
|
|
|
|
Pipeline removed from Mongo → schedule deleted from Temporal.
|
|
|
|
### 1.4.1 Conflict ordering
|
|
|
|
Pipeline update and slot delete on same OPC server → slot insert before delete (production ordering).
|
|
|
|
### 1.5.1 Empty pipelines
|
|
|
|
No active pipelines → orphan schedules removed, no new orchestration writes.
|
|
|
|
### 1.6.1 TTL index bootstrap
|
|
|
|
First run creates TTL index on notification collection used by scouter pipelines.
|
|
|
|
---
|
|
|
|
## 2. Alerts workflow
|
|
|
|
Source: `e2e/test_alerts_main_workflow.py`
|
|
|
|
### A.1.1 Happy path
|
|
|
|
ERROR notification → one SMTP message, one `log_report` row, Redis cache key.
|
|
|
|
### A.1.2 TTL duplicate suppression
|
|
|
|
Second run with same data and cache seeded → no extra email or log row.
|
|
|
|
### A.1.3 Persistent escalation
|
|
|
|
Alert past `notification_ttl` with cache cleared → new email sent.
|
|
|
|
### A.2.1 Group filtering
|
|
|
|
Receiver group `levels` / `ignore_models` honored.
|
|
|
|
### A.3.1 Empty queue
|
|
|
|
No notifications → no SMTP, no Postgres row.
|
|
|
|
---
|
|
|
|
## 3. Reports workflow
|
|
|
|
Source: `e2e/test_reports_main_workflow.py`
|
|
|
|
### R.1.1 Happy path
|
|
|
|
Mixed ERROR/WARNING/INFO → one HTML email with all section headings.
|
|
|
|
### R.1.2 Per-level rendering
|
|
|
|
Single-level notifications → only matching section in HTML body.
|
|
|
|
### R.2.1 Empty queue
|
|
|
|
No notifications → no SMTP, no Postgres row.
|
|
|
|
---
|
|
|
|
## 4. Subworkflows
|
|
|
|
### LoadNotificationPackage — `e2e/test_subworkflow_load_notification_package.py`
|
|
|
|
- No prior Redis timestamp → all matching notifications returned, max timestamp stored.
|
|
- Prior timestamp → only newer notifications returned.
|
|
- Empty Mongo → no Redis timestamp write.
|
|
|
|
### ProcessNotifications — `e2e/test_subworkflow_process_notifications.py`
|
|
|
|
- Full round trip: HTML → SMTP → `log_report` in Postgres.
|
|
- Empty receiver groups → `{}`.
|