gitea_admin/sientia-dataops-orchestrator_temporal

Files

Eduardo Rios 14b854148a SIENTIAPDE-2011: add e2e coverage for query-time filter, fix stale e2e fixtures

e2e fixtures (default_notification, seed_last_timestamp, default_receiver_group)
still assumed the pre-SIENTIAPDE-2007 string-timestamp representation and the
old ignore_models field name, so the load_notification_package e2e suite was
failing independently of this ticket's changes:

- default_notification now seeds `timestamp` as a native datetime (matches
  production since SIENTIAPDE-2007's Notification.timestamp = now()).
- seed_last_timestamp now uses DATETIME_FORMAT_MS_WITH_TZ (matches what
  put_last_data_timestamp actually writes to Redis today).
- default_receiver_group's ignore_models renamed to ignore_models_id, per
  this ticket's decision.
- test_reports_main_workflow.py's explicit timestamp= call sites updated to
  match.

Also fixes a real production bug in put_last_data_timestamp (slot_manager.py)
surfaced by running the e2e suite against a real Temporal environment: Temporal's
default data converter serializes datetime values to ISO-8601 strings across
the activity boundary, and pymongo returns naive datetimes for BSON Date
fields, so `data['timestamp'].max().strftime(...)` crashed on a plain str.
Unit tests never caught this because they mock the activity boundary and
never cross real (de)serialization. Fixed by normalizing through
`pandas.to_datetime(..., utc=True)`, which handles native datetime/Timestamp
and offset-aware/naive ISO strings uniformly.

Adds test_load_package_query_time_filter_union_and_intersection, a new e2e
test exercising the actual SIENTIAPDE-2011 behavior end-to-end: union of
active groups' levels, intersection of their ignore_models_id, and the
requirement that a model ignored by only one of several active groups must
not be excluded from the fetch.

Note: test_reports_main_workflow.py / test_alerts_main_workflow.py still have
a separate, pre-existing failure in process_notifications.py (same family of
bug: a notification's timestamp field parsed with a fixed strptime format
after crossing the same Temporal serialization boundary) — confirmed present
on the unmodified baseline too, unrelated to this ticket, tracked separately.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-09-14 12:15:23 -03:00

3.4 KiB

Raw Permalink Blame History

E2E Scenario Documentation — Orchestrator

Functional reference for orchestrator E2E scenarios. Tests live under e2e/, use @pytest.mark.e2e, and run with:

pytest e2e/ --override-ini testpaths=e2e -m e2e

Execution context

MongoDB, Redis, PostgreSQL: testcontainers (session-scoped).
SMTP: in-process aiosmtpd (e2e/smtp_test_server.py).
Temporal: WorkflowEnvironment.start_local() with stub workers on scouter / laborious.
Production code under orchestrator/** is not mocked; only Logger may be a MagicMock.

1. Orchestrator workflow

Source: e2e/test_orchestrator_main_workflow.py

1.1.1 Happy path

Pipelines in Mongo → schedules created in correct namespace/task queue, Redis slots written, orchestrated_schedules updated.

1.2.1 No-op tick

Mongo, Redis, and Temporal already match desired state → no new schedules or slot writes.

1.3.1 Create-only

New pipeline only → schedules created, timestamps inserted.

1.3.2 Update-only

Existing pipeline with newer updated_at → schedule updated in Temporal.

1.3.3 Delete-only

Pipeline removed from Mongo → schedule deleted from Temporal.

1.4.1 Conflict ordering

Pipeline update and slot delete on same OPC server → slot insert before delete (production ordering).

1.5.1 Empty pipelines

No active pipelines → orphan schedules removed, no new orchestration writes.

1.6.1 TTL index bootstrap

First run creates TTL index on notification collection used by scouter pipelines.

2. Alerts workflow

Source: e2e/test_alerts_main_workflow.py

A.1.1 Happy path

ERROR notification → one SMTP message, one log_report row, Redis cache key.

A.1.2 TTL duplicate suppression

Second run with same data and cache seeded → no extra email or log row.

A.1.3 Persistent escalation

Alert past notification_ttl with cache cleared → new email sent.

A.2.1 Group filtering

Receiver group levels / ignore_models_id honored. The query-time half (union of levels, intersection of ignore_models_id across active groups) is covered at the subworkflow level — see LoadNotificationPackage below. The per-group half (each group only receiving its own precise slice) is not yet implemented in production code (SIENTIAPDE-2012) and so is not yet exercised here.

A.3.1 Empty queue

No notifications → no SMTP, no Postgres row.

3. Reports workflow

Source: e2e/test_reports_main_workflow.py

R.1.1 Happy path

Mixed ERROR/WARNING/INFO → one HTML email with all section headings.

R.1.2 Per-level rendering

Single-level notifications → only matching section in HTML body.

R.2.1 Empty queue