A cold start or notification storm had no backstop: load_latest_data
fetched the entire matching backlog in one query. Confirmed live during
SIENTIAPDE-2012's cluster verification -- an empty Redis cursor plus a
real recurring error let a single run's query exceed Temporal's 4MB
message limit and get stuck retrying forever.
Threads a new notification_queue_batch_size config (env var
NOTIFICATION_QUEUE_BATCH_SIZE, default 5000) through to the repository's
find(), hard cap per run, no drain loop -- the incremental
last_data_timestamp cursor picks up any remainder next run, per the
parent ticket's design decision. Sorts by timestamp ascending so a
capped run always returns the oldest pending documents first; without
that, a capped batch has no guarantee of hitting the oldest documents,
and the cursor would advance past whatever it skipped.
Requires sientia_do>=1.13.0 (adds limit/sort params to
MongoDBRepository.find in both the sync and async client) -- bumped the
pin here per the parent ticket's decision to do it in the same PR.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Manual code-review pass (code-review-python checklist) flagged that no test
exercised default_filter already restricting level (e.g. Alerts' ERROR-only
default) combined with every active group specifying its own levels. The
replace-not-AND behavior is deliberate per SIENTIAPDE-2011's design doc and
ticket text, but was previously only covered by a docstring comment. Adds
test_compute_query_time_filter_union_replaces_default_level_restriction so a
future change reintroducing AND semantics fails a test instead of passing
silently.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Code review findings on the SIENTIAPDE-2011 PR:
- [Major] _compute_query_time_filter used group.get('ignore_models_id', [])
for the intersection, whose default only applies when the key is missing.
receiver_groups has no schema validation, so a document with
ignore_models_id explicitly set to null (key present, value None) made
.get() return None, and set(None) raised TypeError, crashing the whole
activity for every mail type. Fixed with `group.get(...) or []`. Added a
regression test.
- [Minor] CLAUDE.md:48 described Alerts' base data filter as always
`level: ERROR`, which this ticket's own query-time union can now widen.
Updated the line to reflect that.
- [Blocker, not applied] A finding argued the levels union should AND with
default_filter's own level restriction instead of replacing it, to avoid
widening Alerts (ERROR-only real-time channel) when a receiver_group sets
broader levels. Both SIENTIAPDE-2011's ticket text and the parent design
doc explicitly call for the union to replace the hard-coded ERROR-only
default ("whatever levels the active groups actually care about"), so
applying that fix would revert the subtask's actual purpose. The
interim gap the finding correctly identifies (no per-group level filter
exists yet until SIENTIAPDE-2012 lands) is now documented in
_compute_query_time_filter's docstring instead of "fixed" by narrowing
query-time behavior the ticket asked for.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
notification['timestamp'] crosses at least one Temporal activity
boundary (load_latest_data -> filter_notification_reports/alerts)
before reaching format_log_report. Temporal's default data converter
serializes datetime values to plain ISO-8601 strings without
restoring them on decode, and pymongo returns naive datetimes for
BSON Date fields, so the string can carry no UTC offset at all. That
broke export_data_to_postgres's strict to_datetime(..., format=
DATETIME_FORMAT_WITH_TZ) parsing downstream with:
ValueError: time data "2024-06-01T12:00:00" doesn't match format
"%Y-%m-%d %H:%M:%S%z"
Same underlying issue as SIENTIAPDE-2007/2009, fixed here using the
same pattern as SlotManager.put_last_data_timestamp: normalize with
pandas.to_datetime(..., utc=True) and re-format to
DATETIME_FORMAT_WITH_TZ before the value crosses the next boundary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Push level and model-exclusion filtering down to the Mongo query instead of
fetching the full notification_queue unfiltered on every run.
- Fix await ordering in load_notification_package so sending_configs (active
receiver_groups) resolves before load_latest_data is called, letting the
query filter be built from group data.
- Add _compute_query_time_filter: unions active groups' `levels` (falls back
to the workflow's default filter if any group has no restriction, since
that group wants every level) and intersects their `ignore_models_id`
(a model is only safe to drop from the fetch if every active group
ignores it).
- Add unit tests covering mixed groups, a model ignored by only one group
(must not be excluded from the fetch), and the empty active-groups case.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Addresses review feedback: prefer sientia_do's explicit datetime format
constant over Python's isoformat()/fromisoformat() when round-tripping
last_data_timestamp through Redis, matching the convention already used
elsewhere in mongo_db.py and slot_manager.py.
Notification.timestamp is now stored as a native BSON Date instead of a
string, so load_latest_data must parse the ISO string coming from Redis
back to datetime before building the $gt filter, or the comparison
silently breaks (BSON String vs Date always evaluates False). Also fixes
put_last_data_timestamp, which would otherwise raise TypeError trying to
json.dumps a pandas Timestamp when persisting the new watermark to Redis.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Enhance orchestration configuration and documentation. Added `RUNTIME` variable to `.env.example`, updated `.gitignore` to exclude `openspec/` and `.cursor/`, and modified `README.md` to clarify queue naming conventions and runtime handling. Refactored activities to use synchronous database and email handling, improving performance and consistency. Updated test cases to reflect these changes and ensure compatibility with new activity definitions.
Update version in init_orchestration.ipynb to 3.11.14 and enhance README.md with detailed architecture principles, execution flows, and improved filtering mechanisms in workflows. Refactor orchestrator_functions.py for clarity in filter configuration and add support for PI Web API output configuration in predictions_batch. Update tests to reflect new configurations.
Refactor test cases in test_orchestrator_functions.py to consolidate read_tags structure and enhance clarity in API error handling, improving test robustness and readability.
Update GITHUB_BRANCH to feature/SIENTIAPDE-1445 and enhance formatters with a new registry for workflow types, improving configuration management. Refactor process_schedules to utilize the new registry and add error handling for unsupported workflow types. Update tests to validate new functionality and ensure proper integration.
SIENTIAPDE-1273 Add base_scouter and pi_web_api_scouter functions to orchestrator utilities for enhanced configuration management. Update tests to validate new functionalities and ensure proper integration with existing workflows.
SIENTIAPDE-1273 Add 'save_transform' and 'transform_table_name' parameters to predictions_batch function for enhanced data processing capabilities. Update corresponding test cases to reflect these changes.
SIENTIAPDE-1273 Refactor drift function in orchestrator utilities for improved configuration and update formatters to include drift import. Enhance tests to validate new configurations and ensure integration with existing workflows.
Implement simple metrics configuration in formatters and orchestrator utilities. Update tests to validate new simple metrics functionality and ensure proper integration with existing workflows.
Implement drift monitoring configuration and update formatters to support drift workflows. Add drift function to orchestrator utilities and enhance tests for drift functionality.
Remove unused Couchbase configurations and related code. Refactor Activities class to eliminate Couchbase dependency, updating initialization and shutdown methods. Update README and tests to reflect these changes. Upgrade sientia-dataops-library dependency version in requirements.txt.
Refactor workflow calls to use subworkflow naming convention for load_notification_package and process_notifications in alerts and reports modules. Update corresponding tests to reflect these changes.
SIENTIAPDE-1312: Refactor build_tag_config to accept a list of tags and return notifications for missing servers. Update Formatters to handle notifications during slot configuration. Enhance tests to validate new behavior and error reporting.
SIENTIAPDE-1312: Update build_tag_config to calculate subscription_period_ms based on minimum frequency from tags. Enhance test_build_tag_config to include frequency and validate subscription_period_ms.
Update sientia-dataops-library dependency to version 1.4.7 and enhance TemporalManager to support offset configuration for scheduling. Added offset handling in common configuration and updated related tests to validate new functionality.
Refactor orchestrator activities and configuration files
- Removed unused Temporal timeout environment variables from values.yaml.
- Reorganized import statements in various activity files for better readability.
- Updated logging messages to use consistent formatting across activities.
- Enhanced test cases to ensure proper initialization and shutdown of orchestrator activities.
- Improved overall code structure and readability by applying consistent formatting and style adjustments.
SIENTIAPDE-1222 Refactor server configuration handling in orchestrator_functions.py and update related tests
- Removed the security_spec field from server configuration in build_tag_config to streamline the configuration process.
- Updated test cases in test_formatters.py and test_orchestrator_functions.py to reflect the removal of security_spec, ensuring alignment with the new configuration structure and improving test accuracy.
Enhance test cases for server configuration in test_formatters.py and test_orchestrator_functions.py
- Added optional fields for certificate paths in the test cases to align with recent changes in server configuration, improving test coverage and ensuring proper handling of security configurations.
SIENTIAPDE-1222
Enhance tests for email and MongoDB activities
- Added a test for sending emails when no SMTP server is configured, ensuring proper handling of such cases.
- Updated MongoDB tests to include timestamp fields in the results, improving data accuracy in document retrieval.
- Modified connector configuration tests to reflect changes in SMTP server settings.
Add test for handling cases with no receiver groups in Alerts workflow
- Implemented a new test case to verify the behavior of the Alerts workflow when no receiver groups are provided, ensuring that the workflow executes correctly without unnecessary processing.
SIENTIAPDE-1205: Enhance predictions_batch function and update test cases
- Added 'predictions_storage_policy' to the predictions_batch function for improved configuration handling.
- Updated test cases to include 'predictions_storage_policy' in expected results, ensuring consistency in testing.
SIENTIAPDE-1205: Update timestamp formatting in MongoDB tests
- Modified timestamp handling in test_mongo_db.py to utilize DATETIME_FORMAT_MS_WITH_TZ for consistent formatting across test cases.
- Ensured all timestamp strings include UTC timezone information for improved accuracy in datetime comparisons.
refactor: standardize timestamp handling across MongoDB and SlotManager
- Replaced instances of `now()` with `date_now` for consistent timestamp usage in MongoDB and SlotManager activities.
- Updated test cases to mock the new `now()` function, ensuring accurate timestamp handling in unit tests.
- Adjusted timestamp formatting in various tests to align with the DATETIME_FORMAT_MS_WITH_TZ standard.
fix: add metadata handling in schedule configuration tests
- Enhanced the test cases in test_formatters.py and test_temporal_manager.py to include metadata in the schedule configuration inputs.
- This change improves the accuracy and traceability of the tests by ensuring that metadata is properly accounted for in the test scenarios.
feat: add shutdown tests for Activities and Email classes
- Implemented unit tests for the shutdown methods in the Activities and Email classes to ensure proper resource cleanup.
- Verified that the shutdown processes for MongoDB, PostgreSQL, and Email are called as expected during the shutdown sequence.
SIENTIAPDE-1184
refactor: improve schedule reporting in Formatters class
- Introduced a unified reporting structure for created, updated, and deleted schedules.
- Enhanced success and error report messages for clarity, including specific schedule types in notifications.
- Updated attachment handling in success and error reports to improve context and readability.
fix: enhance error reporting in Formatters class
- Updated the parse_report_schedule method to return a dictionary for error keys, including both message and attachment.
- Improved error report formatting in send_error_report method to include attachments for better context in failure notifications.
SIENTIAPDE-1174
chore: update sientia-dataops-library dependency version from 1.3.7 to 1.3.8 in requirements.txt
- Removed unused test group from test_formatters.py.
- Added new test case for handling scenarios with no log reports in test_alerts.py.
- Added new test case for handling scenarios with no groups in test_reports.py.
fix: add conditional login for SMTP server and enhance test coverage
- Updated the Email class to conditionally log in to the SMTP server only if a sender password is provided, improving security.
- Enhanced test cases for the try_send_email method to cover various scenarios, including successful email sending and handling SMTP disconnections.
Update values.yaml and tests to incorporate mail_type parameter for notification handling
- Changed GITHUB_BRANCH in values.yaml to reflect the new alert pipeline.
- Added 'mail_type' parameter to various test cases in SlotManager and notification workflows to enhance notification specificity and tracking.
- Ensured consistency in test data structures across multiple test functions.
feat: enhance email notifications and orchestrator activities for reports
- Updated email HTML structure to clearly present errors, warnings, and info notifications.
- Introduced a new method in the Formatters class to filter notification reports based on group configurations.
- Enhanced the Reports workflow to integrate the new filtering functionality and manage notification packages effectively.
- Adjusted SlotManager to store timestamps with mail type specificity for better tracking.
- Improved test coverage for the new filtering functionality in the Formatters class.
refactor: streamline email generation and enhance notification handling
- Updated email HTML structure to improve clarity and presentation of notifications.
- Enhanced notification generator to better manage execution counts and output formatting.
- Refactored Email class to improve error handling and logging during email sending.
- Adjusted EmailBuilder to efficiently process error, warning, and info models.
- Improved test coverage for email and notification functionalities to ensure reliability.
feat: enhance email and slot manager activities with new filtering and caching functionalities
- Updated Email class to improve attachment handling and logging.
- Refactored Formatters class to process notifications more effectively by grouping them based on their configurations.
- Introduced new methods in SlotManager for filtering notification alerts and storing notification cache.
- Enhanced Alerts workflow to integrate new filtering and caching activities.
- Added tests for new functionalities in SlotManager and Formatters classes to ensure reliability.
refactor: simplify EmailBuilder parameters and remove unused templates
- Removed unused template file attributes from EmailBuilder class.
- Updated parameters method to eliminate report_data argument, streamlining the function for better clarity and efficiency.
feat: enhance orchestrator activities with new email and Postgres integrations
- Added Email and Postgres classes to the Activities class for improved functionality.
- Introduced new methods in MongoDB and SlotManager for loading and managing data.
- Updated requirements.txt to include jinja2.
- Added new formatting activity for log reports in Formatters class.
- Enhanced test coverage for MongoDB and SlotManager activities.
feat: enhance test coverage for minimal_retrain workflow
- Added a new test for the minimal_retrain function to validate its configuration output.
- Updated the test_process_schedules to include a new schedule entry for minimal_retrain, ensuring comprehensive testing of the workflow integration.
