SIENTIAPDE-2011: add e2e coverage for query-time filter, fix stale e2e fixtures
e2e fixtures (default_notification, seed_last_timestamp, default_receiver_group) still assumed the pre-SIENTIAPDE-2007 string-timestamp representation and the old ignore_models field name, so the load_notification_package e2e suite was failing independently of this ticket's changes: - default_notification now seeds `timestamp` as a native datetime (matches production since SIENTIAPDE-2007's Notification.timestamp = now()). - seed_last_timestamp now uses DATETIME_FORMAT_MS_WITH_TZ (matches what put_last_data_timestamp actually writes to Redis today). - default_receiver_group's ignore_models renamed to ignore_models_id, per this ticket's decision. - test_reports_main_workflow.py's explicit timestamp= call sites updated to match. Also fixes a real production bug in put_last_data_timestamp (slot_manager.py) surfaced by running the e2e suite against a real Temporal environment: Temporal's default data converter serializes datetime values to ISO-8601 strings across the activity boundary, and pymongo returns naive datetimes for BSON Date fields, so `data['timestamp'].max().strftime(...)` crashed on a plain str. Unit tests never caught this because they mock the activity boundary and never cross real (de)serialization. Fixed by normalizing through `pandas.to_datetime(..., utc=True)`, which handles native datetime/Timestamp and offset-aware/naive ISO strings uniformly. Adds test_load_package_query_time_filter_union_and_intersection, a new e2e test exercising the actual SIENTIAPDE-2011 behavior end-to-end: union of active groups' levels, intersection of their ignore_models_id, and the requirement that a model ignored by only one of several active groups must not be excluded from the fetch. Note: test_reports_main_workflow.py / test_alerts_main_workflow.py still have a separate, pre-existing failure in process_notifications.py (same family of bug: a notification's timestamp field parsed with a fixed strptime format after crossing the same Temporal serialization boundary) — confirmed present on the unmodified baseline too, unrelated to this ticket, tracked separately. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -7,7 +7,7 @@ with workflow.unsafe.imports_passed_through():
|
||||
from logging import Logger
|
||||
from typing import Any
|
||||
|
||||
from pandas import DataFrame
|
||||
from pandas import DataFrame, to_datetime
|
||||
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
|
||||
from sientia_do.notifications.models import NotificationLevel
|
||||
from sientia_do.observability.sientia_monitoring import MetricsController, SientiaMonitoring
|
||||
@@ -331,14 +331,25 @@ class SlotManager(SientiaMonitoring):
|
||||
self.warning('No data to insert', metadata=metadata)
|
||||
return None
|
||||
|
||||
last_data_timestamp = data['timestamp'].max()
|
||||
# ``timestamp`` started out as a native ``datetime``/BSON Date in Mongo,
|
||||
# but by the time it gets here it has crossed a Temporal activity
|
||||
# boundary (this activity's ``input_data`` is whatever
|
||||
# ``load_latest_data`` returned): Temporal's default data converter
|
||||
# serializes ``datetime`` values to plain ISO-8601 strings and does not
|
||||
# restore them on decode, so ``data['timestamp']`` is a ``str`` in a
|
||||
# real workflow run even though unit tests mocking the activity in
|
||||
# isolation never see that conversion. pymongo also returns naive
|
||||
# datetimes for BSON Date fields (no tzinfo), so the ISO string may
|
||||
# carry no UTC offset at all. ``utc=True`` normalizes every case
|
||||
# (native datetime/Timestamp, offset-aware or naive ISO string) into a
|
||||
# UTC-aware ``Timestamp`` — naive values are BSON Dates, always UTC.
|
||||
last_data_timestamp = to_datetime(data['timestamp'], utc=True).max()
|
||||
|
||||
self.debug(f'Last collected timestamp to insert: {last_data_timestamp}', metadata=metadata)
|
||||
|
||||
# ``timestamp`` is a native ``datetime``/``Timestamp`` (BSON Date from
|
||||
# Mongo), which the Redis repository's plain ``json.dumps`` cannot
|
||||
# serialize. Store it as a string formatted per
|
||||
# ``DATETIME_FORMAT_MS_WITH_TZ`` instead.
|
||||
# The Redis repository's plain ``json.dumps`` cannot serialize a
|
||||
# ``Timestamp``/``datetime`` directly, so store it as a string
|
||||
# formatted per ``DATETIME_FORMAT_MS_WITH_TZ`` instead.
|
||||
last_data_timestamp = last_data_timestamp.strftime(DATETIME_FORMAT_MS_WITH_TZ)
|
||||
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user