CI mypy caught what my local environment couldn't (Python 3.14 has no
compatible numpy stub for the project's python_version=3.11 mypy target,
so mypy couldn't run locally at all):
slot_manager.py:353: error: Incompatible types in assignment
(expression has type "str", variable has type "Timestamp")
slot_manager.py:368: error: Incompatible return value type
(got "Timestamp", expected "str | None")
Reassigning `last_data_timestamp` from a Timestamp to its strftime() string
trips mypy's inference (it fixes an unannotated local's type from its first
assignment). Split into a separate `last_data_timestamp_str` for the string
form instead of reusing the name.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
e2e fixtures (default_notification, seed_last_timestamp, default_receiver_group)
still assumed the pre-SIENTIAPDE-2007 string-timestamp representation and the
old ignore_models field name, so the load_notification_package e2e suite was
failing independently of this ticket's changes:
- default_notification now seeds `timestamp` as a native datetime (matches
production since SIENTIAPDE-2007's Notification.timestamp = now()).
- seed_last_timestamp now uses DATETIME_FORMAT_MS_WITH_TZ (matches what
put_last_data_timestamp actually writes to Redis today).
- default_receiver_group's ignore_models renamed to ignore_models_id, per
this ticket's decision.
- test_reports_main_workflow.py's explicit timestamp= call sites updated to
match.
Also fixes a real production bug in put_last_data_timestamp (slot_manager.py)
surfaced by running the e2e suite against a real Temporal environment: Temporal's
default data converter serializes datetime values to ISO-8601 strings across
the activity boundary, and pymongo returns naive datetimes for BSON Date
fields, so `data['timestamp'].max().strftime(...)` crashed on a plain str.
Unit tests never caught this because they mock the activity boundary and
never cross real (de)serialization. Fixed by normalizing through
`pandas.to_datetime(..., utc=True)`, which handles native datetime/Timestamp
and offset-aware/naive ISO strings uniformly.
Adds test_load_package_query_time_filter_union_and_intersection, a new e2e
test exercising the actual SIENTIAPDE-2011 behavior end-to-end: union of
active groups' levels, intersection of their ignore_models_id, and the
requirement that a model ignored by only one of several active groups must
not be excluded from the fetch.
Note: test_reports_main_workflow.py / test_alerts_main_workflow.py still have
a separate, pre-existing failure in process_notifications.py (same family of
bug: a notification's timestamp field parsed with a fixed strptime format
after crossing the same Temporal serialization boundary) — confirmed present
on the unmodified baseline too, unrelated to this ticket, tracked separately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Push level and model-exclusion filtering down to the Mongo query instead of
fetching the full notification_queue unfiltered on every run.
- Fix await ordering in load_notification_package so sending_configs (active
receiver_groups) resolves before load_latest_data is called, letting the
query filter be built from group data.
- Add _compute_query_time_filter: unions active groups' `levels` (falls back
to the workflow's default filter if any group has no restriction, since
that group wants every level) and intersects their `ignore_models_id`
(a model is only safe to drop from the fetch if every active group
ignores it).
- Add unit tests covering mixed groups, a model ignored by only one group
(must not be excluded from the fetch), and the empty active-groups case.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Addresses review feedback: prefer sientia_do's explicit datetime format
constant over Python's isoformat()/fromisoformat() when round-tripping
last_data_timestamp through Redis, matching the convention already used
elsewhere in mongo_db.py and slot_manager.py.
Notification.timestamp is now stored as a native BSON Date instead of a
string, so load_latest_data must parse the ISO string coming from Redis
back to datetime before building the $gt filter, or the comparison
silently breaks (BSON String vs Date always evaluates False). Also fixes
put_last_data_timestamp, which would otherwise raise TypeError trying to
json.dumps a pandas Timestamp when persisting the new watermark to Redis.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SIENTIAPDE-1646 Update RUNTIME_WORKFLOWS in TemporalManager to include 'drift' and 'simple_metrics', expanding workflow capabilities and enhancing orchestration functionality.
SIENTIAPDE-1646 Add support for runtime workflows in TemporalManager by introducing RUNTIME_WORKFLOWS constant and updating schedule creation logic to handle runtime names appropriately.
Update GITHUB_BRANCH in values.yaml to feature/SIENTIAPDE-1646, refactor mongo_db.py for cleaner code, and enhance worker.py by adding missing commas for improved syntax consistency.
Enhance orchestration configuration and documentation. Added `RUNTIME` variable to `.env.example`, updated `.gitignore` to exclude `openspec/` and `.cursor/`, and modified `README.md` to clarify queue naming conventions and runtime handling. Refactored activities to use synchronous database and email handling, improving performance and consistency. Updated test cases to reflect these changes and ensure compatibility with new activity definitions.
Fix error model retrieval in notification_generator.ipynb and add on_conflict parameter to common_config in orchestrator_functions.py for improved error handling.
Reintroduce prepare_worker import in worker.py to maintain consistency in worker setup, ensuring clarity and proper initialization for orchestrator workflows.
Refactor worker initialization by introducing prepare_worker function to streamline worker setup for orchestrator, alerts, and reports workflows, enhancing code clarity and maintainability.
Update version in init_orchestration.ipynb to 3.11.14 and enhance README.md with detailed architecture principles, execution flows, and improved filtering mechanisms in workflows. Refactor orchestrator_functions.py for clarity in filter configuration and add support for PI Web API output configuration in predictions_batch. Update tests to reflect new configurations.
Enhance MongoDB activity to handle missing topic configurations gracefully by using get() method and skipping empty collections, improving robustness in pipeline processing.
Refactor tag processing in pi_web_api_scouter function to improve clarity by renaming variables and enhancing readability of the configuration handling.
Refactor values.yaml to comment out email configuration settings for improved security and clarity. Update orchestrator_functions.py to enhance tag processing by iterating over items in read_tags, improving code readability.
Refactor gather_read_tags function to remove logger parameter and associated logging statements, streamlining the function for improved clarity and focus on processing pipelines.
Update GITHUB_BRANCH to feature/SIENTIAPDE-1445 and enhance formatters with a new registry for workflow types, improving configuration management. Refactor process_schedules to utilize the new registry and add error handling for unsupported workflow types. Update tests to validate new functionality and ensure proper integration.
SIENTIAPDE-1273 Add base_scouter and pi_web_api_scouter functions to orchestrator utilities for enhanced configuration management. Update tests to validate new functionalities and ensure proper integration with existing workflows.
SIENTIAPDE-1273 Add 'save_transform' and 'transform_table_name' parameters to predictions_batch function for enhanced data processing capabilities. Update corresponding test cases to reflect these changes.
Enhance orchestrator utilities and activities with improved documentation and functionality. Update README to reflect new orchestrator functions for drift and simple metrics workflows. Refactor email, formatters, mongo_db, slot_manager, and temporal_manager activities to include detailed docstrings and improve clarity on input parameters and return values. Ensure consistent metadata handling across activities for better logging and tracking.
SIENTIAPDE-1273 Refactor drift function in orchestrator utilities for improved configuration and update formatters to include drift import. Enhance tests to validate new configurations and ensure integration with existing workflows.
Implement simple metrics configuration in formatters and orchestrator utilities. Update tests to validate new simple metrics functionality and ensure proper integration with existing workflows.
Update GITHUB_BRANCH to feature/SIENTIAPDE-1273 and increment Helm chart version to 0.6.0. Enhance TemporalManager to include offset in schedule intervals for improved scheduling flexibility.
Implement drift monitoring configuration and update formatters to support drift workflows. Add drift function to orchestrator utilities and enhance tests for drift functionality.
Update image tag to 0.5.0 in values.yaml and change GITHUB_BRANCH to feature/SIENTIAPDE-1325 for specific external operation metrics. Remove unused timestamp formatting in mongo_db.py to streamline data handling.
Remove unused Couchbase configurations and related code. Refactor Activities class to eliminate Couchbase dependency, updating initialization and shutdown methods. Update README and tests to reflect these changes. Upgrade sientia-dataops-library dependency version in requirements.txt.
Refactor workflow calls to use subworkflow naming convention for load_notification_package and process_notifications in alerts and reports modules. Update corresponding tests to reflect these changes.
SIENTIAPDE-1312: Refactor build_tag_config to accept a list of tags and return notifications for missing servers. Update Formatters to handle notifications during slot configuration. Enhance tests to validate new behavior and error reporting.
SIENTIAPDE-1312: Update build_tag_config to ensure minimum frequency defaults to 1000 and convert frequency values to integers for accurate calculations of subscription_period_ms.
SIENTIAPDE-1312: Update build_tag_config to calculate subscription_period_ms based on minimum frequency from tags. Enhance test_build_tag_config to include frequency and validate subscription_period_ms.
Update sientia-dataops-library dependency to version 1.4.7 and enhance TemporalManager to support offset configuration for scheduling. Added offset handling in common configuration and updated related tests to validate new functionality.
Enhance README and code documentation for clarity and structure
- Added a comprehensive Table of Contents to the README for easier navigation.
- Introduced a new section on Code Quality & Validation, detailing the validation workflow and tools used.
- Improved descriptions and consistency in docstrings across various modules, including metrics, formatters, and MongoDB activities.
- Clarified function return types and arguments in the MongoDB and TemporalManager classes for better understanding.
Enhance type hinting and code structure in orchestrator activities
- Added type hints for better code clarity in `formatters.py`, `mongo_db.py`, `temporal_manager.py`, and `email_builder.py`.
- Updated `.gitignore` to include `coverage.xml` for improved coverage reporting.
- Improved readability by restructuring variable declarations and method signatures.
Refactor orchestrator activities and configuration files
- Removed unused Temporal timeout environment variables from values.yaml.
- Reorganized import statements in various activity files for better readability.
- Updated logging messages to use consistent formatting across activities.
- Enhanced test cases to ensure proper initialization and shutdown of orchestrator activities.
- Improved overall code structure and readability by applying consistent formatting and style adjustments.
