Code import - branch 1.3.0

e2e/scenarios.md

@@ -0,0 +1,138 @@
+# Scouter E2E scenario catalog
+
+## Execution context
+
+- **MongoDB**, **Redis**, and **PostgreSQL** run as session-scoped testcontainers with autouse cleanup between tests.
+- **PI Web API** is an in-process HTTP server (`e2e/pi_web_api_test_server.py`) speaking the wire format consumed by `PIWebAPIClient`.
+- **Temporal** uses `WorkflowEnvironment.start_local()` and a single worker on `scouter-test-queue`.
+- **Production code is not mocked** (except `Logger` and optional notification insert spy).
+
+---
+
+## 0. Harness smoke tests
+
+Diagnostic-only checks under `e2e/test_harness_smoke.py`. They are not business scenarios; they exist to fail fast when the harness itself (Docker / containers / Temporal worker wiring) is broken, before the numbered suite runs.
+
+### 0.0.1 Postgres schema ready (`test_postgres_schema_ready`)
+
+Confirms the autouse fixture executed `db_schema.sql` and `sientia_data.laborious_data` exists in the Postgres testcontainer.
+
+### 0.0.2 Activities construct (`test_activities_construct`)
+
+Confirms the production `Activities` instance initializes against the Mongo/Redis/Postgres testcontainers without hanging (no `patch(...)` involved).
+
+### 0.0.3 Temporal PI happy path (`test_temporal_pi_happy_path`)
+
+End-to-end liveness check: `WorkflowEnvironment.start_local()` + worker + in-process PI server + `PIWebAPIScouter` complete without raising. Functional assertions for this flow live in scenario **2.1.1**.
+
+---
+
+## 1. Scouter workflow
+
+### 1.1.1 Happy path
+
+Seed `raw_<schedule>` with multiple documents, run `Scouter`, assert Postgres rows and Redis `last_data_timestamp:scouter:<schedule>`.
+
+### 1.2.1 Incremental load
+
+Pre-seed Redis timestamp; seed older and newer Mongo docs; assert only newer rows export and timestamp advances.
+
+### 1.3.1 Empty Mongo early exit
+
+Empty `raw_<schedule>`; workflow exits without Postgres rows or Redis timestamp key.
+
+### 1.3.2 No Redis timestamp first run
+
+No prior Redis key; all seeded Mongo docs load and timestamp is written after success.
+
+---
+
+## 2. PIWebAPIScouter workflow
+
+### 2.1.1 Happy path
+
+PI server `success` mode with two tags; assert Postgres rows, Redis hold key, one HTTP request recorded.
+
+### 2.1.2 Multiple tags
+
+Five tags with `avg` / `mdn` / `max` / `min` / `lts`; assert five distinct `variable` values and exact aggregated numbers in Postgres.
+
+### 2.1.3 Debug data package
+
+`debug_data_package=True`; assert `data_package_pi_web_api_scouter_*` Redis key with `data` and `held_data`.
+
+### 2.2.1 Empty response early exit
+
+Server `empty` mode; zero Postgres rows for `model_id`, one request recorded.
+
+### 2.3.1 PI Web API connection error
+
+Server `error` mode (HTTP 5xx); workflow fails; `PI_WEB_API_REQUEST_ERROR` notification in Mongo.
+
+### 2.3.2 PI Web API timeout
+
+Server `timeout` mode; workflow fails; `PI_WEB_API_REQUEST_ERROR` notification sent.
+
+### 2.3.3 Invalid endpoint
+
+Workflow uses `/invalid/endpoint` (404); workflow fails; `PI_WEB_API_REQUEST_ERROR` notification sent.
+
+---
+
+## 3. CoreScouter subworkflow
+
+### 3.1.1 Complete processing success
+
+Single tag, no filters; Postgres row and `held_data_*` Redis key; no `data_package_*` key.
+
+### 3.1.2 Null values filter discard
+
+`NULL_VALUES_FILTER` DISCARD; one WARNING notification; only valid row in Postgres.
+
+### 3.1.3 Null values filter warn
+
+`NULL_VALUES_FILTER` WARN; notification sent; both rows in Postgres.
+
+### 3.1.4 Out of bounds filter discard
+
+`OUT_OF_BOUNDS_FILTER` DISCARD; in-range row only; WARNING notification.
+
+### 3.1.5 Aggregation avg
+
+Three points; Postgres `value` equals arithmetic mean (20.0).
+
+### 3.1.6 Aggregation mdn
+
+Median equals 5.0 in Postgres.
+
+### 3.1.7 Aggregation max
+
+Maximum equals 15.0 in Postgres.
+
+### 3.1.8 Aggregation min
+
+Minimum equals 30.0 in Postgres.
+
+### 3.1.9 Aggregation lts
+
+Last-by-timestamp value equals 300.0 in Postgres.
+
+### 3.1.10 Fill missing tags
+
+`fill_missing_tags=True`; `held_data_*` contains missing tag keys with `None`.
+
+### 3.1.11 Debug data package
+
+`debug_data_package=True`; `data_package_*` Redis key decodes to dict with `data` and `held_data`.
+
+### 3.2.1 Empty after grouping early exit
+
+Empty column-oriented `data`; no Postgres rows; no populated `held_data_*`.
+
+### 3.3.1 Invalid aggregation function
+
+`aggr_func= bogus`; `AGGREGATION_ISSUES` ERROR notification; bogus tag absent from Postgres.
+
+### 3.3.2 Postgres export failure surfaces
+
+Drop `value` column before run; workflow fails; ERROR notification in Mongo.