474c2ef42c
Enhance orchestration configuration and documentation. Added `RUNTIME` variable to `.env.example`, updated `.gitignore` to exclude `openspec/` and `.cursor/`, and modified `README.md` to clarify queue naming conventions and runtime handling. Refactored activities to use synchronous database and email handling, improving performance and consistency. Updated test cases to reflect these changes and ensure compatibility with new activity definitions.
112 lines
2.8 KiB
Markdown
112 lines
2.8 KiB
Markdown
# E2E Scenario Documentation — Orchestrator
|
|
|
|
Functional reference for orchestrator E2E scenarios. Tests live under `e2e/`, use `@pytest.mark.e2e`, and run with:
|
|
|
|
```bash
|
|
pytest e2e/ --override-ini testpaths=e2e -m e2e
|
|
```
|
|
|
|
## Execution context
|
|
|
|
- MongoDB, Redis, PostgreSQL: testcontainers (session-scoped).
|
|
- SMTP: in-process `aiosmtpd` (`e2e/smtp_test_server.py`).
|
|
- Temporal: `WorkflowEnvironment.start_local()` with stub workers on `scouter` / `laborious`.
|
|
- Production code under `orchestrator/**` is not mocked; only `Logger` may be a `MagicMock`.
|
|
|
|
---
|
|
|
|
## 1. Orchestrator workflow
|
|
|
|
Source: `e2e/test_orchestrator_main_workflow.py`
|
|
|
|
### 1.1.1 Happy path
|
|
|
|
Pipelines in Mongo → schedules created in correct namespace/task queue, Redis slots written, `orchestrated_schedules` updated.
|
|
|
|
### 1.2.1 No-op tick
|
|
|
|
Mongo, Redis, and Temporal already match desired state → no new schedules or slot writes.
|
|
|
|
### 1.3.1 Create-only
|
|
|
|
New pipeline only → schedules created, timestamps inserted.
|
|
|
|
### 1.3.2 Update-only
|
|
|
|
Existing pipeline with newer `updated_at` → schedule updated in Temporal.
|
|
|
|
### 1.3.3 Delete-only
|
|
|
|
Pipeline removed from Mongo → schedule deleted from Temporal.
|
|
|
|
### 1.4.1 Conflict ordering
|
|
|
|
Pipeline update and slot delete on same OPC server → slot insert before delete (production ordering).
|
|
|
|
### 1.5.1 Empty pipelines
|
|
|
|
No active pipelines → orphan schedules removed, no new orchestration writes.
|
|
|
|
### 1.6.1 TTL index bootstrap
|
|
|
|
First run creates TTL index on notification collection used by scouter pipelines.
|
|
|
|
---
|
|
|
|
## 2. Alerts workflow
|
|
|
|
Source: `e2e/test_alerts_main_workflow.py`
|
|
|
|
### A.1.1 Happy path
|
|
|
|
ERROR notification → one SMTP message, one `log_report` row, Redis cache key.
|
|
|
|
### A.1.2 TTL duplicate suppression
|
|
|
|
Second run with same data and cache seeded → no extra email or log row.
|
|
|
|
### A.1.3 Persistent escalation
|
|
|
|
Alert past `notification_ttl` with cache cleared → new email sent.
|
|
|
|
### A.2.1 Group filtering
|
|
|
|
Receiver group `levels` / `ignore_models` honored.
|
|
|
|
### A.3.1 Empty queue
|
|
|
|
No notifications → no SMTP, no Postgres row.
|
|
|
|
---
|
|
|
|
## 3. Reports workflow
|
|
|
|
Source: `e2e/test_reports_main_workflow.py`
|
|
|
|
### R.1.1 Happy path
|
|
|
|
Mixed ERROR/WARNING/INFO → one HTML email with all section headings.
|
|
|
|
### R.1.2 Per-level rendering
|
|
|
|
Single-level notifications → only matching section in HTML body.
|
|
|
|
### R.2.1 Empty queue
|
|
|
|
No notifications → no SMTP, no Postgres row.
|
|
|
|
---
|
|
|
|
## 4. Subworkflows
|
|
|
|
### LoadNotificationPackage — `e2e/test_subworkflow_load_notification_package.py`
|
|
|
|
- No prior Redis timestamp → all matching notifications returned, max timestamp stored.
|
|
- Prior timestamp → only newer notifications returned.
|
|
- Empty Mongo → no Redis timestamp write.
|
|
|
|
### ProcessNotifications — `e2e/test_subworkflow_process_notifications.py`
|
|
|
|
- Full round trip: HTML → SMTP → `log_report` in Postgres.
|
|
- Empty receiver groups → `{}`.
|