New, separate model_id-based filter distinct from the existing
ignore_models_id deny-list (SIENTIAPDE-2012): a group configuring
model_ids only receives notifications from those models. Both axes
apply together (AND) when a group sets both.
Design decisions made while implementing (matching the levels/
ignore_models_id split from SIENTIAPDE-2011/2012, not yet confirmed
with the team beyond what's in the ticket):
- Query-time filter: union of every active group's model_ids, applied
only when ALL active groups set one -- mirrors levels' exact
union-or-nothing logic in _compute_query_time_filter. Combines
cleanly with ignore_models_id's intersection $nin on the same
model_id field.
- Per-group filter: extended _check_levels_and_ignore_models_id (now
_check_levels_and_model_filters) in slot_manager.py, and the inline
condition in formatters.py's filter_notification_reports.
- Empty/missing model_ids means no restriction, same convention as
levels and ignore_models_id.
Covered by new unit tests (slot_manager, formatters,
_compute_query_time_filter) and two new e2e tests against real
Mongo/Redis/Temporal: per-group allow-list interacting with another
unrestricted group, and the query-time union excluding a model outside
every active group's allow-list.
Not yet verified on the real cluster -- blocked on the dev cluster's
package index not auto-syncing new sientia_do releases (separate issue,
being investigated for SIENTIAPDE-2013).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A cold start or notification storm had no backstop: load_latest_data
fetched the entire matching backlog in one query. Confirmed live during
SIENTIAPDE-2012's cluster verification -- an empty Redis cursor plus a
real recurring error let a single run's query exceed Temporal's 4MB
message limit and get stuck retrying forever.
Threads a new notification_queue_batch_size config (env var
NOTIFICATION_QUEUE_BATCH_SIZE, default 5000) through to the repository's
find(), hard cap per run, no drain loop -- the incremental
last_data_timestamp cursor picks up any remainder next run, per the
parent ticket's design decision. Sorts by timestamp ascending so a
capped run always returns the oldest pending documents first; without
that, a capped batch has no guarantee of hitting the oldest documents,
and the cursor would advance past whatever it skipped.
Requires sientia_do>=1.13.0 (adds limit/sort params to
MongoDBRepository.find in both the sync and async client) -- bumped the
pin here per the parent ticket's decision to do it in the same PR.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
filter_notification_alerts tripped Sonar's cognitive-complexity gate
(16 > 15 allowed) from its nested loop/if structure. Split into
_filter_group_notifications (per-group notification selection) and
_determine_alert_type (Redis TTL classification), leaving the public
method as a simple loop over receiver groups. Behavior and call order
are unchanged -- verified against the existing filter_notification_alerts
unit tests by inspection (sientia_do isn't installable in this local
environment, so the real suite can't run here; CI will confirm).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The parameter-level annotation on _check_keys_and_notif_id alone
wasn't enough -- mypy needs the type at the empty list's declaration
site inside filter_notification_alerts. Also reformatted the helper's
now-longer signature to satisfy ruff.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes CI formatting failure introduced by the manual Sonar cognitive-
complexity fix (extracting the ignore-list/dedup check into its own
helper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes CI formatting failure introduced by the manual Sonar cognitive-
complexity fix (extracting the levels/ignore_models_id check into its
own helper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends filter_notification_alerts (slot_manager.py) and
filter_notification_reports (formatters.py) to check
receiver_group.get('levels') and receiver_group.get('ignore_models_id'),
mirroring the existing contents/ignore pattern exactly. Closes the gap
SIENTIAPDE-2011 deliberately left open: the query-time union/intersection
filter widens what gets fetched for the union of all active groups, but until
now nothing narrowed each group back down to its own configured slice.
Both checks short-circuit via `not levels_list or ...` / `not
ignore_models_id_list or ...` rather than accessing notification['level']/
['model_id'] directly, so groups that don't configure either field (today's
common case, and every existing unit test fixture) never evaluate the
right-hand side and can't raise KeyError on notifications missing those keys.
ignore_models_id reads via `.get(...) or []`, defensively handling an
explicit `null` in a receiver_groups document from the start (same fix
SIENTIAPDE-2011 needed a follow-up commit for).
Tests:
- 8 new unit tests (4 per file): levels restriction, ignore_models_id
restriction, neither-configured passthrough, explicit-None safety.
- New e2e test (test_scenario_a_2_1_per_group_levels_and_ignore_models_id)
against real Mongo/Redis/Temporal: two active groups configured
differently: SIENTIAPDE-2011's query-time filter widens the fetch to the
union of both, and this subtask's per-group filter narrows each group back
to its own levels/ignore_models_id, including the case where a model is
ignored by only one of the active groups.
- Deduplicated the two e2e suites' identical _extract_html_body helper into
e2e/helpers.py (extract_html_body), now shared by both.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
notification['timestamp'] crosses at least one Temporal activity
boundary (load_latest_data -> filter_notification_reports/alerts)
before reaching format_log_report. Temporal's default data converter
serializes datetime values to plain ISO-8601 strings without
restoring them on decode, and pymongo returns naive datetimes for
BSON Date fields, so the string can carry no UTC offset at all. That
broke export_data_to_postgres's strict to_datetime(..., format=
DATETIME_FORMAT_WITH_TZ) parsing downstream with:
ValueError: time data "2024-06-01T12:00:00" doesn't match format
"%Y-%m-%d %H:%M:%S%z"
Same underlying issue as SIENTIAPDE-2007/2009, fixed here using the
same pattern as SlotManager.put_last_data_timestamp: normalize with
pandas.to_datetime(..., utc=True) and re-format to
DATETIME_FORMAT_WITH_TZ before the value crosses the next boundary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI mypy caught what my local environment couldn't (Python 3.14 has no
compatible numpy stub for the project's python_version=3.11 mypy target,
so mypy couldn't run locally at all):
slot_manager.py:353: error: Incompatible types in assignment
(expression has type "str", variable has type "Timestamp")
slot_manager.py:368: error: Incompatible return value type
(got "Timestamp", expected "str | None")
Reassigning `last_data_timestamp` from a Timestamp to its strftime() string
trips mypy's inference (it fixes an unannotated local's type from its first
assignment). Split into a separate `last_data_timestamp_str` for the string
form instead of reusing the name.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
e2e fixtures (default_notification, seed_last_timestamp, default_receiver_group)
still assumed the pre-SIENTIAPDE-2007 string-timestamp representation and the
old ignore_models field name, so the load_notification_package e2e suite was
failing independently of this ticket's changes:
- default_notification now seeds `timestamp` as a native datetime (matches
production since SIENTIAPDE-2007's Notification.timestamp = now()).
- seed_last_timestamp now uses DATETIME_FORMAT_MS_WITH_TZ (matches what
put_last_data_timestamp actually writes to Redis today).
- default_receiver_group's ignore_models renamed to ignore_models_id, per
this ticket's decision.
- test_reports_main_workflow.py's explicit timestamp= call sites updated to
match.
Also fixes a real production bug in put_last_data_timestamp (slot_manager.py)
surfaced by running the e2e suite against a real Temporal environment: Temporal's
default data converter serializes datetime values to ISO-8601 strings across
the activity boundary, and pymongo returns naive datetimes for BSON Date
fields, so `data['timestamp'].max().strftime(...)` crashed on a plain str.
Unit tests never caught this because they mock the activity boundary and
never cross real (de)serialization. Fixed by normalizing through
`pandas.to_datetime(..., utc=True)`, which handles native datetime/Timestamp
and offset-aware/naive ISO strings uniformly.
Adds test_load_package_query_time_filter_union_and_intersection, a new e2e
test exercising the actual SIENTIAPDE-2011 behavior end-to-end: union of
active groups' levels, intersection of their ignore_models_id, and the
requirement that a model ignored by only one of several active groups must
not be excluded from the fetch.
Note: test_reports_main_workflow.py / test_alerts_main_workflow.py still have
a separate, pre-existing failure in process_notifications.py (same family of
bug: a notification's timestamp field parsed with a fixed strptime format
after crossing the same Temporal serialization boundary) — confirmed present
on the unmodified baseline too, unrelated to this ticket, tracked separately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Addresses review feedback: prefer sientia_do's explicit datetime format
constant over Python's isoformat()/fromisoformat() when round-tripping
last_data_timestamp through Redis, matching the convention already used
elsewhere in mongo_db.py and slot_manager.py.
Notification.timestamp is now stored as a native BSON Date instead of a
string, so load_latest_data must parse the ISO string coming from Redis
back to datetime before building the $gt filter, or the comparison
silently breaks (BSON String vs Date always evaluates False). Also fixes
put_last_data_timestamp, which would otherwise raise TypeError trying to
json.dumps a pandas Timestamp when persisting the new watermark to Redis.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SIENTIAPDE-1646 Update RUNTIME_WORKFLOWS in TemporalManager to include 'drift' and 'simple_metrics', expanding workflow capabilities and enhancing orchestration functionality.
SIENTIAPDE-1646 Add support for runtime workflows in TemporalManager by introducing RUNTIME_WORKFLOWS constant and updating schedule creation logic to handle runtime names appropriately.
Update GITHUB_BRANCH in values.yaml to feature/SIENTIAPDE-1646, refactor mongo_db.py for cleaner code, and enhance worker.py by adding missing commas for improved syntax consistency.
Enhance orchestration configuration and documentation. Added `RUNTIME` variable to `.env.example`, updated `.gitignore` to exclude `openspec/` and `.cursor/`, and modified `README.md` to clarify queue naming conventions and runtime handling. Refactored activities to use synchronous database and email handling, improving performance and consistency. Updated test cases to reflect these changes and ensure compatibility with new activity definitions.
Enhance MongoDB activity to handle missing topic configurations gracefully by using get() method and skipping empty collections, improving robustness in pipeline processing.
Update GITHUB_BRANCH to feature/SIENTIAPDE-1445 and enhance formatters with a new registry for workflow types, improving configuration management. Refactor process_schedules to utilize the new registry and add error handling for unsupported workflow types. Update tests to validate new functionality and ensure proper integration.
Enhance orchestrator utilities and activities with improved documentation and functionality. Update README to reflect new orchestrator functions for drift and simple metrics workflows. Refactor email, formatters, mongo_db, slot_manager, and temporal_manager activities to include detailed docstrings and improve clarity on input parameters and return values. Ensure consistent metadata handling across activities for better logging and tracking.
SIENTIAPDE-1273 Refactor drift function in orchestrator utilities for improved configuration and update formatters to include drift import. Enhance tests to validate new configurations and ensure integration with existing workflows.
Implement simple metrics configuration in formatters and orchestrator utilities. Update tests to validate new simple metrics functionality and ensure proper integration with existing workflows.
Update GITHUB_BRANCH to feature/SIENTIAPDE-1273 and increment Helm chart version to 0.6.0. Enhance TemporalManager to include offset in schedule intervals for improved scheduling flexibility.
Implement drift monitoring configuration and update formatters to support drift workflows. Add drift function to orchestrator utilities and enhance tests for drift functionality.
Update image tag to 0.5.0 in values.yaml and change GITHUB_BRANCH to feature/SIENTIAPDE-1325 for specific external operation metrics. Remove unused timestamp formatting in mongo_db.py to streamline data handling.
Remove unused Couchbase configurations and related code. Refactor Activities class to eliminate Couchbase dependency, updating initialization and shutdown methods. Update README and tests to reflect these changes. Upgrade sientia-dataops-library dependency version in requirements.txt.
SIENTIAPDE-1312: Refactor build_tag_config to accept a list of tags and return notifications for missing servers. Update Formatters to handle notifications during slot configuration. Enhance tests to validate new behavior and error reporting.
Update sientia-dataops-library dependency to version 1.4.7 and enhance TemporalManager to support offset configuration for scheduling. Added offset handling in common configuration and updated related tests to validate new functionality.
Enhance README and code documentation for clarity and structure
- Added a comprehensive Table of Contents to the README for easier navigation.
- Introduced a new section on Code Quality & Validation, detailing the validation workflow and tools used.
- Improved descriptions and consistency in docstrings across various modules, including metrics, formatters, and MongoDB activities.
- Clarified function return types and arguments in the MongoDB and TemporalManager classes for better understanding.
Enhance type hinting and code structure in orchestrator activities
- Added type hints for better code clarity in `formatters.py`, `mongo_db.py`, `temporal_manager.py`, and `email_builder.py`.
- Updated `.gitignore` to include `coverage.xml` for improved coverage reporting.
- Improved readability by restructuring variable declarations and method signatures.
Refactor orchestrator activities and configuration files
- Removed unused Temporal timeout environment variables from values.yaml.
- Reorganized import statements in various activity files for better readability.
- Updated logging messages to use consistent formatting across activities.
- Enhanced test cases to ensure proper initialization and shutdown of orchestrator activities.
- Improved overall code structure and readability by applying consistent formatting and style adjustments.
SIENTIAPDE-1231: Add timeout configuration for Temporal tasks
- Introduced new timeout parameters in values.yaml for task execution.
- Updated TemporalManager to utilize execution and task timeout settings in seconds.
- Enhanced orchestrator_functions.py to include default timeout values for better orchestration control.
Update orchestration configuration and execution parameters
- Adjusted execution counts in Jupyter notebook cells for better tracking.
- Updated schedule name in the orchestration configuration to align with new requirements.
- Modified `values.yaml` to change the GitHub branch for improved versioning.
- Enhanced `Activities` and `TemporalManager` classes to include timeout parameters for task execution.
- Updated connector configuration to include new timeout settings for better orchestration control.
Enhance README and activities with advanced notification filtering features
- Updated README to include new intelligent notification filtering capabilities, including TTL-based duplicate prevention and ignore lists.
- Enhanced `Formatters` activity to support notification filtering for scheduled reports and added detailed filtering logic.
- Improved `SlotManager` to implement advanced notification filtering with persistent alert detection and group-based filtering.
- Updated `Alerts` workflow to incorporate intelligent filtering for real-time error notifications, ensuring efficient alert delivery.
Refactor orchestration and email handling
- Updated the orchestration queries to streamline model lookups and added conditions for active models.
- Changed the entry point in the local run script to use the worker module.
- Removed the deprecated samples.json file.
- Enhanced the Email activity to skip sending if the SMTP server is not configured.
- Added timestamp field handling in MongoDB activities for better data management.
- Updated connectors configuration to allow for a None SMTP server.
- Improved the common configuration function to include model configuration details.
Remove deprecated files and configurations
- Deleted coverage.sh, Dockerfile, docker-compose.yml, email.html, input_sample.json, and Makefile as part of the cleanup process.
- Updated README.md to reflect the removal of these components and provide a clearer overview of the project structure and functionality.
- Added module-level docstrings to orchestrator, activities, and workflows for improved documentation and clarity on system architecture.
SIENTIAPDE-1205: Update timestamp handling in MongoDB activities
- Modified timestamp formatting in mongo_db.py to apply UTC timezone before string conversion, ensuring consistency with DATETIME_FORMAT_MS_WITH_TZ.
SIENTIAPDE-1205: Simplify timestamp formatting in MongoDB activities
- Removed timezone conversion from timestamp handling in mongo_db.py, streamlining the formatting process while maintaining consistency with DATETIME_FORMAT_MS_WITH_TZ.
SIENTIAPDE-1205: Refactor timestamp handling in MongoDB activities
- Updated the import statement in mongo_db.py to include timezone support.
- Modified timestamp formatting to ensure UTC timezone is applied before string conversion, enhancing consistency in datetime handling.
