# E2E Scenario Documentation — Orchestrator

Functional reference for orchestrator E2E scenarios. Tests live under `e2e/`, use `@pytest.mark.e2e`, and run with:

```bash
pytest e2e/ --override-ini testpaths=e2e -m e2e
```

**Every section below maps 1:1 to an actual test function** (named in each section, or as a bullet
under a `Source:` heading) — this file describes what the suite verifies today, not planned or
aspirational coverage. Adding a scenario here without a matching test (or vice versa) is a doc bug;
fix the mismatch instead of leaving it for later.

## Execution context

- MongoDB, Redis, PostgreSQL: testcontainers (session-scoped).
- SMTP: in-process `aiosmtpd` (`e2e/smtp_test_server.py`).
- Temporal: `WorkflowEnvironment.start_local()` with stub workers on `scouter` / `laborious`.
- Production code under `orchestrator/**` is not mocked; only `Logger` may be a `MagicMock`.

---

## 1. Orchestrator workflow

Source: `e2e/test_orchestrator_main_workflow.py`

### 1.1.1 Happy path

Pipelines in Mongo → Temporal schedule created in the `laborious` namespace, `orchestrated_schedules` entries written.

### 1.3.1 Create-only

New pipeline only → Temporal schedule created for it.

### 1.3.3 Delete-only

Pipeline removed from Mongo → previously-created Temporal schedule deleted.

### 1.5.1 Empty pipelines

No active pipelines → no `orchestrated_schedules` documents written, even with a pre-existing
unrelated Temporal schedule present.

### 1.6.1 TTL index bootstrap

First run creates TTL index on notification collection used by scouter pipelines.

---

## 2. Alerts workflow

Source: `e2e/test_alerts_main_workflow.py`

### A.1.1 Happy path

ERROR notification → one SMTP message, one `log_report` row, Redis cache key.

### A.1.2 TTL duplicate suppression

Second run with same data and cache seeded → no extra email or log row.

### A.2.1 Group filtering

Receiver group `levels` / `ignore_models_id` honored, both halves now covered:
the query-time union/intersection across active groups (SIENTIAPDE-2011, see
`LoadNotificationPackage` below) and the per-group precise slice
(SIENTIAPDE-2012) — a group only receives notifications matching its own
`levels`/`ignore_models_id`, even when the query-time filter widened the fetch
to satisfy another active group's broader configuration.

### A.3.1 Empty queue

No notifications → no SMTP, no Postgres row.

---

## 3. Reports workflow

Source: `e2e/test_reports_main_workflow.py`

### R.1.1 Happy path

Mixed ERROR/WARNING/INFO → one HTML email with all section headings.

### R.1.2 Per-level rendering

Single-level notifications → only matching section in HTML body.

### R.2.1 Empty queue

No notifications → no SMTP, no Postgres row.

---

## 4. Subworkflows

### LoadNotificationPackage — `e2e/test_subworkflow_load_notification_package.py`

- No prior Redis timestamp → all matching notifications returned, max timestamp stored.
- Prior timestamp → only newer notifications returned.
- Query-time filter (SIENTIAPDE-2011) → union of active groups' `levels`, intersection of their `ignore_models_id`; a model ignored by only one of several active groups is not excluded from the fetch.
- Empty Mongo → no Redis timestamp write.

### ProcessNotifications — `e2e/test_subworkflow_process_notifications.py`

- Full round trip: HTML → SMTP → `log_report` in Postgres.
- Empty receiver groups → `{}`.