SIENTIAPDE-2011: add e2e coverage for query-time filter, fix stale e2e fixtures

e2e fixtures (default_notification, seed_last_timestamp, default_receiver_group)
still assumed the pre-SIENTIAPDE-2007 string-timestamp representation and the
old ignore_models field name, so the load_notification_package e2e suite was
failing independently of this ticket's changes:

- default_notification now seeds `timestamp` as a native datetime (matches
  production since SIENTIAPDE-2007's Notification.timestamp = now()).
- seed_last_timestamp now uses DATETIME_FORMAT_MS_WITH_TZ (matches what
  put_last_data_timestamp actually writes to Redis today).
- default_receiver_group's ignore_models renamed to ignore_models_id, per
  this ticket's decision.
- test_reports_main_workflow.py's explicit timestamp= call sites updated to
  match.

Also fixes a real production bug in put_last_data_timestamp (slot_manager.py)
surfaced by running the e2e suite against a real Temporal environment: Temporal's
default data converter serializes datetime values to ISO-8601 strings across
the activity boundary, and pymongo returns naive datetimes for BSON Date
fields, so `data['timestamp'].max().strftime(...)` crashed on a plain str.
Unit tests never caught this because they mock the activity boundary and
never cross real (de)serialization. Fixed by normalizing through
`pandas.to_datetime(..., utc=True)`, which handles native datetime/Timestamp
and offset-aware/naive ISO strings uniformly.

Adds test_load_package_query_time_filter_union_and_intersection, a new e2e
test exercising the actual SIENTIAPDE-2011 behavior end-to-end: union of
active groups' levels, intersection of their ignore_models_id, and the
requirement that a model ignored by only one of several active groups must
not be excluded from the fetch.

Note: test_reports_main_workflow.py / test_alerts_main_workflow.py still have
a separate, pre-existing failure in process_notifications.py (same family of
bug: a notification's timestamp field parsed with a fixed strptime format
after crossing the same Temporal serialization boundary) — confirmed present
on the unmodified baseline too, unrelated to this ticket, tracked separately.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Eduardo Rios
2026-09-14 12:15:23 -03:00
parent 1711f3b0d1
commit 14b854148a
5 changed files with 143 additions and 47 deletions

View File

@@ -29,7 +29,6 @@ MONGO_COLLECTIONS = (
)
DATETIME_FORMAT_MS_WITH_TZ = '%Y-%m-%d %H:%M:%S.%f%z'
DATETIME_FORMAT_WITH_TZ = '%Y-%m-%d %H:%M:%S%z'
_TIMESTAMP_MARKER_PATTERN = re.compile(r'^@now(?:([+-])(\d+)([smhd]))?$')
_TIMESTAMP_UNIT_TO_KWARG = {'s': 'seconds', 'm': 'minutes', 'h': 'hours', 'd': 'days'}
@@ -222,17 +221,19 @@ def seed_last_timestamp(redis_client: Redis, mail_type: str, value: str) -> None
Set notification_last_timestamp for a mail type, JSON-encoded.
Values must be JSON-encoded so ``redis_repository.get`` (which calls
``json.loads`` on the raw payload) can deserialize them. The value
must follow the exact format ``sientia_do.notifications.models.Notification``
writes into ``notification_queue.timestamp``: ``DATETIME_FORMAT_WITH_TZ``
(e.g. ``"2026-05-22 16:47:02+0000"``) — no microseconds and no colon in
the timezone offset.
``json.loads`` on the raw payload) can deserialize them. The value must
follow the exact format ``put_last_data_timestamp``
(``orchestrator/activities/slot_manager.py``) writes into Redis today:
``DATETIME_FORMAT_MS_WITH_TZ`` (e.g. ``"2026-05-22 16:47:02.000000+0000"``)
— microseconds included, no colon in the timezone offset. ``load_latest_data``
parses this same format back into a ``datetime`` for the ``$gt`` comparison,
so the two sides must agree exactly.
Args:
redis_client: Redis client connected to the test instance.
mail_type: Mail type identifier (e.g. ``"Alerts"``, ``"Reports"``).
value: Timestamp string in ``DATETIME_FORMAT_WITH_TZ``
(e.g. ``"2024-06-01 10:30:00+0000"``).
value: Timestamp string in ``DATETIME_FORMAT_MS_WITH_TZ``
(e.g. ``"2024-06-01 10:30:00.000000+0000"``).
"""
redis_client.set(f'notification_last_timestamp:{mail_type}', json.dumps(value))
@@ -295,34 +296,35 @@ def default_notification(
*,
notification_id: str,
level: str = 'ERROR',
timestamp: str | None = None,
timestamp: datetime | None = None,
model_name: str = 'model-a',
model_id: str = '1',
) -> dict[str, Any]:
"""
Build a minimal notification_queue document mirroring production layout.
The ``timestamp`` field is stored as a string in ``DATETIME_FORMAT_WITH_TZ``
because that is exactly what ``sientia_do.notifications.models.Notification``
writes into ``notification_queue`` in production (``now().strftime(
DATETIME_FORMAT_WITH_TZ)``). Tests intentionally use this same format so we
The ``timestamp`` field is stored as a native, timezone-aware ``datetime``
(BSON Date) because that is exactly what
``sientia_do.notifications.models.Notification`` writes into
``notification_queue`` in production since SIENTIAPDE-2007
(``self.timestamp = now()`` — a real ``datetime``, no more string
formatting). Tests intentionally use this same representation so we
surface, rather than hide, real production behavior in downstream
activities.
activities (e.g. ``put_last_data_timestamp``'s ``DataFrame(...)['timestamp']
.max().strftime(...)``, which requires a real ``datetime``, not a string).
Args:
notification_id: Unique identifier for the notification.
level: Notification level (e.g. ``"ERROR"``, ``"WARNING"``).
timestamp: Optional production-format timestamp string. ``None`` falls
back to a fixed sample value.
timestamp: Optional timezone-aware datetime. ``None`` falls back to a
fixed sample value.
model_name: Model name attached to the notification.
model_id: Model id attached to the notification.
Return:
dict[str, Any]: A notification document ready for insertion.
"""
ts = timestamp if timestamp is not None else datetime(
2024, 6, 1, 12, 0, 0, tzinfo=UTC
).strftime(DATETIME_FORMAT_WITH_TZ)
ts = timestamp if timestamp is not None else datetime(2024, 6, 1, 12, 0, 0, tzinfo=UTC)
return {
'notification_id': notification_id,
'level': level,
@@ -343,7 +345,7 @@ def default_receiver_group(
members: list[str] | None = None,
levels: list[str] | None = None,
contents: list[str] | None = None,
ignore_models: list[str] | None = None,
ignore_models_id: list[str] | None = None,
) -> dict[str, Any]:
"""Minimal active receiver_groups document."""
return {
@@ -352,5 +354,5 @@ def default_receiver_group(
'members': members or ['admin@example.com'],
'levels': levels or ['ERROR', 'WARNING', 'INFO'],
'contents': contents or ['core_alerts', 'persistent_alerts', 'reports'],
'ignore_models': ignore_models or [],
'ignore_models_id': ignore_models_id or [],
}