Add e2e tag-database scenarios and cover worker/email/formatter gaps
- Add e2e scenarios 1.9.1-1.9.4 for PI Web API through the tag
database: scouter and output tag resolution, mixed-connector schedule
hold, and OPC slot unchanged by PI connectors. Seed connectors and
connector_tags from a new scenario input.
- Tighten the happy-path e2e to assert both schedules and that no PI
output config is produced when the tag database is unused.
- Drop dead `if self.server` / `if notification_handler` guards in the
email reconnect and worker shutdown paths, since both are always set.
- Log the traceback inside the message on worker failure, as the
sientia_do Logger.error does not accept exc_info.
- Add unit tests for the worker failure exit, Prometheus startup
failure, email reconnect without a password, and formatter, mongo,
temporal manager and orchestrator edge cases.
New, separate model_id-based filter distinct from the existing
ignore_models_id deny-list (SIENTIAPDE-2012): a group configuring
model_ids only receives notifications from those models. Both axes
apply together (AND) when a group sets both.
Design decisions made while implementing (matching the levels/
ignore_models_id split from SIENTIAPDE-2011/2012, not yet confirmed
with the team beyond what's in the ticket):
- Query-time filter: union of every active group's model_ids, applied
only when ALL active groups set one -- mirrors levels' exact
union-or-nothing logic in _compute_query_time_filter. Combines
cleanly with ignore_models_id's intersection $nin on the same
model_id field.
- Per-group filter: extended _check_levels_and_ignore_models_id (now
_check_levels_and_model_filters) in slot_manager.py, and the inline
condition in formatters.py's filter_notification_reports.
- Empty/missing model_ids means no restriction, same convention as
levels and ignore_models_id.
Covered by new unit tests (slot_manager, formatters,
_compute_query_time_filter) and two new e2e tests against real
Mongo/Redis/Temporal: per-group allow-list interacting with another
unrestricted group, and the query-time union excluding a model outside
every active group's allow-list.
Not yet verified on the real cluster -- blocked on the dev cluster's
package index not auto-syncing new sientia_do releases (separate issue,
being investigated for SIENTIAPDE-2013).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends filter_notification_alerts (slot_manager.py) and
filter_notification_reports (formatters.py) to check
receiver_group.get('levels') and receiver_group.get('ignore_models_id'),
mirroring the existing contents/ignore pattern exactly. Closes the gap
SIENTIAPDE-2011 deliberately left open: the query-time union/intersection
filter widens what gets fetched for the union of all active groups, but until
now nothing narrowed each group back down to its own configured slice.
Both checks short-circuit via `not levels_list or ...` / `not
ignore_models_id_list or ...` rather than accessing notification['level']/
['model_id'] directly, so groups that don't configure either field (today's
common case, and every existing unit test fixture) never evaluate the
right-hand side and can't raise KeyError on notifications missing those keys.
ignore_models_id reads via `.get(...) or []`, defensively handling an
explicit `null` in a receiver_groups document from the start (same fix
SIENTIAPDE-2011 needed a follow-up commit for).
Tests:
- 8 new unit tests (4 per file): levels restriction, ignore_models_id
restriction, neither-configured passthrough, explicit-None safety.
- New e2e test (test_scenario_a_2_1_per_group_levels_and_ignore_models_id)
against real Mongo/Redis/Temporal: two active groups configured
differently: SIENTIAPDE-2011's query-time filter widens the fetch to the
union of both, and this subtask's per-group filter narrows each group back
to its own levels/ignore_models_id, including the case where a model is
ignored by only one of the active groups.
- Deduplicated the two e2e suites' identical _extract_html_body helper into
e2e/helpers.py (extract_html_body), now shared by both.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
e2e fixtures (default_notification, seed_last_timestamp, default_receiver_group)
still assumed the pre-SIENTIAPDE-2007 string-timestamp representation and the
old ignore_models field name, so the load_notification_package e2e suite was
failing independently of this ticket's changes:
- default_notification now seeds `timestamp` as a native datetime (matches
production since SIENTIAPDE-2007's Notification.timestamp = now()).
- seed_last_timestamp now uses DATETIME_FORMAT_MS_WITH_TZ (matches what
put_last_data_timestamp actually writes to Redis today).
- default_receiver_group's ignore_models renamed to ignore_models_id, per
this ticket's decision.
- test_reports_main_workflow.py's explicit timestamp= call sites updated to
match.
Also fixes a real production bug in put_last_data_timestamp (slot_manager.py)
surfaced by running the e2e suite against a real Temporal environment: Temporal's
default data converter serializes datetime values to ISO-8601 strings across
the activity boundary, and pymongo returns naive datetimes for BSON Date
fields, so `data['timestamp'].max().strftime(...)` crashed on a plain str.
Unit tests never caught this because they mock the activity boundary and
never cross real (de)serialization. Fixed by normalizing through
`pandas.to_datetime(..., utc=True)`, which handles native datetime/Timestamp
and offset-aware/naive ISO strings uniformly.
Adds test_load_package_query_time_filter_union_and_intersection, a new e2e
test exercising the actual SIENTIAPDE-2011 behavior end-to-end: union of
active groups' levels, intersection of their ignore_models_id, and the
requirement that a model ignored by only one of several active groups must
not be excluded from the fetch.
Note: test_reports_main_workflow.py / test_alerts_main_workflow.py still have
a separate, pre-existing failure in process_notifications.py (same family of
bug: a notification's timestamp field parsed with a fixed strptime format
after crossing the same Temporal serialization boundary) — confirmed present
on the unmodified baseline too, unrelated to this ticket, tracked separately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Enhance orchestration configuration and documentation. Added `RUNTIME` variable to `.env.example`, updated `.gitignore` to exclude `openspec/` and `.cursor/`, and modified `README.md` to clarify queue naming conventions and runtime handling. Refactored activities to use synchronous database and email handling, improving performance and consistency. Updated test cases to reflect these changes and ensure compatibility with new activity definitions.
