SIENTIAPDE-1646 Add support for runtime workflows in TemporalManager by introducing RUNTIME_WORKFLOWS constant and updating schedule creation logic to handle runtime names appropriately.
Update GITHUB_BRANCH in values.yaml to feature/SIENTIAPDE-1646, refactor mongo_db.py for cleaner code, and enhance worker.py by adding missing commas for improved syntax consistency.
Enhance orchestration configuration and documentation. Added `RUNTIME` variable to `.env.example`, updated `.gitignore` to exclude `openspec/` and `.cursor/`, and modified `README.md` to clarify queue naming conventions and runtime handling. Refactored activities to use synchronous database and email handling, improving performance and consistency. Updated test cases to reflect these changes and ensure compatibility with new activity definitions.
Enhance MongoDB activity to handle missing topic configurations gracefully by using get() method and skipping empty collections, improving robustness in pipeline processing.
Update GITHUB_BRANCH to feature/SIENTIAPDE-1445 and enhance formatters with a new registry for workflow types, improving configuration management. Refactor process_schedules to utilize the new registry and add error handling for unsupported workflow types. Update tests to validate new functionality and ensure proper integration.
Enhance orchestrator utilities and activities with improved documentation and functionality. Update README to reflect new orchestrator functions for drift and simple metrics workflows. Refactor email, formatters, mongo_db, slot_manager, and temporal_manager activities to include detailed docstrings and improve clarity on input parameters and return values. Ensure consistent metadata handling across activities for better logging and tracking.
SIENTIAPDE-1273 Refactor drift function in orchestrator utilities for improved configuration and update formatters to include drift import. Enhance tests to validate new configurations and ensure integration with existing workflows.
Implement simple metrics configuration in formatters and orchestrator utilities. Update tests to validate new simple metrics functionality and ensure proper integration with existing workflows.
Update GITHUB_BRANCH to feature/SIENTIAPDE-1273 and increment Helm chart version to 0.6.0. Enhance TemporalManager to include offset in schedule intervals for improved scheduling flexibility.
Implement drift monitoring configuration and update formatters to support drift workflows. Add drift function to orchestrator utilities and enhance tests for drift functionality.
Update image tag to 0.5.0 in values.yaml and change GITHUB_BRANCH to feature/SIENTIAPDE-1325 for specific external operation metrics. Remove unused timestamp formatting in mongo_db.py to streamline data handling.
Remove unused Couchbase configurations and related code. Refactor Activities class to eliminate Couchbase dependency, updating initialization and shutdown methods. Update README and tests to reflect these changes. Upgrade sientia-dataops-library dependency version in requirements.txt.
SIENTIAPDE-1312: Refactor build_tag_config to accept a list of tags and return notifications for missing servers. Update Formatters to handle notifications during slot configuration. Enhance tests to validate new behavior and error reporting.
Update sientia-dataops-library dependency to version 1.4.7 and enhance TemporalManager to support offset configuration for scheduling. Added offset handling in common configuration and updated related tests to validate new functionality.
Enhance README and code documentation for clarity and structure
- Added a comprehensive Table of Contents to the README for easier navigation.
- Introduced a new section on Code Quality & Validation, detailing the validation workflow and tools used.
- Improved descriptions and consistency in docstrings across various modules, including metrics, formatters, and MongoDB activities.
- Clarified function return types and arguments in the MongoDB and TemporalManager classes for better understanding.
Enhance type hinting and code structure in orchestrator activities
- Added type hints for better code clarity in `formatters.py`, `mongo_db.py`, `temporal_manager.py`, and `email_builder.py`.
- Updated `.gitignore` to include `coverage.xml` for improved coverage reporting.
- Improved readability by restructuring variable declarations and method signatures.
Refactor orchestrator activities and configuration files
- Removed unused Temporal timeout environment variables from values.yaml.
- Reorganized import statements in various activity files for better readability.
- Updated logging messages to use consistent formatting across activities.
- Enhanced test cases to ensure proper initialization and shutdown of orchestrator activities.
- Improved overall code structure and readability by applying consistent formatting and style adjustments.
SIENTIAPDE-1231: Add timeout configuration for Temporal tasks
- Introduced new timeout parameters in values.yaml for task execution.
- Updated TemporalManager to utilize execution and task timeout settings in seconds.
- Enhanced orchestrator_functions.py to include default timeout values for better orchestration control.
Update orchestration configuration and execution parameters
- Adjusted execution counts in Jupyter notebook cells for better tracking.
- Updated schedule name in the orchestration configuration to align with new requirements.
- Modified `values.yaml` to change the GitHub branch for improved versioning.
- Enhanced `Activities` and `TemporalManager` classes to include timeout parameters for task execution.
- Updated connector configuration to include new timeout settings for better orchestration control.
Enhance README and activities with advanced notification filtering features
- Updated README to include new intelligent notification filtering capabilities, including TTL-based duplicate prevention and ignore lists.
- Enhanced `Formatters` activity to support notification filtering for scheduled reports and added detailed filtering logic.
- Improved `SlotManager` to implement advanced notification filtering with persistent alert detection and group-based filtering.
- Updated `Alerts` workflow to incorporate intelligent filtering for real-time error notifications, ensuring efficient alert delivery.
Refactor orchestration and email handling
- Updated the orchestration queries to streamline model lookups and added conditions for active models.
- Changed the entry point in the local run script to use the worker module.
- Removed the deprecated samples.json file.
- Enhanced the Email activity to skip sending if the SMTP server is not configured.
- Added timestamp field handling in MongoDB activities for better data management.
- Updated connectors configuration to allow for a None SMTP server.
- Improved the common configuration function to include model configuration details.
Remove deprecated files and configurations
- Deleted coverage.sh, Dockerfile, docker-compose.yml, email.html, input_sample.json, and Makefile as part of the cleanup process.
- Updated README.md to reflect the removal of these components and provide a clearer overview of the project structure and functionality.
- Added module-level docstrings to orchestrator, activities, and workflows for improved documentation and clarity on system architecture.
SIENTIAPDE-1205: Update timestamp handling in MongoDB activities
- Modified timestamp formatting in mongo_db.py to apply UTC timezone before string conversion, ensuring consistency with DATETIME_FORMAT_MS_WITH_TZ.
SIENTIAPDE-1205: Simplify timestamp formatting in MongoDB activities
- Removed timezone conversion from timestamp handling in mongo_db.py, streamlining the formatting process while maintaining consistency with DATETIME_FORMAT_MS_WITH_TZ.
SIENTIAPDE-1205: Refactor timestamp handling in MongoDB activities
- Updated the import statement in mongo_db.py to include timezone support.
- Modified timestamp formatting to ensure UTC timezone is applied before string conversion, enhancing consistency in datetime handling.
SIENTIAPDE-1205: Update GITHUB_BRANCH and standardize datetime handling in MongoDB activities
- Changed GITHUB_BRANCH in values.yaml from "main" to "SIENTIAPDE-1205-alterar-opc-para-assincrono".
- Updated datetime parsing in mongo_db.py to use DATETIME_FORMAT_MS_WITH_TZ for consistent timestamp handling.
Update orchestration notebook execution count, modify image tag in values.yaml, and standardize datetime handling in MongoDB activities
- Changed execution count in init_orchestration.ipynb from 2 to 1.
- Updated image tag in values.yaml from 0.4.4 to 0.4.5.
- Replaced DATETIME_FORMAT_MS_WITH_TZ with DATETIME_FORMAT_MS in mongo_db.py for consistent timestamp handling.
refactor: standardize timestamp handling across MongoDB and SlotManager
- Replaced instances of `now()` with `date_now` for consistent timestamp usage in MongoDB and SlotManager activities.
- Updated test cases to mock the new `now()` function, ensuring accurate timestamp handling in unit tests.
- Adjusted timestamp formatting in various tests to align with the DATETIME_FORMAT_MS_WITH_TZ standard.
refactor: update date handling and remove deprecated constants
- Replaced the usage of datetime.now() with a new now() function from sientia_do.temporal.constants for consistent timestamp handling across multiple files.
- Updated the date format from DEFAULT_DATE_FORMAT to DATETIME_FORMAT_MS_WITH_TZ in various activities to ensure uniformity in timestamp formatting.
- Removed the obsolete patterns.py file that contained the DEFAULT_DATE_FORMAT constant.
refactor: remove unused schedule_handles attribute from TemporalManager
- Eliminated the schedule_handles dictionary from the TemporalManager class as it was no longer needed.
- Cleaned up related code to ensure proper functionality without the removed attribute.
refactor: enhance logging and metadata handling across activities
- Updated logging statements in Email, Formatters, MongoDB, SlotManager, and TemporalManager classes to include metadata for better traceability.
- Improved success reporting in MongoDB, SlotManager, and TemporalManager by specifying the number of successfully processed items.
- Refactored logger usage in worker.py to utilize custom_info for enhanced logging consistency.
chore: update dependencies and refactor logger imports
- Updated the sientia-dataops-library dependency version from 1.3.8 to 1.4.1 in requirements.txt.
- Changed image tag in values.yaml from 0.4.0 to 0.4.2.
- Refactored logger imports in multiple files to use the new observability module instead of the temporal module.
- Updated retry policy imports in various workflow files to reflect the new structure.
refactor: enhance shutdown procedures and documentation across activities
- Added shutdown methods to MongoDB and Email classes to ensure proper resource cleanup.
- Updated docstrings for shutdown methods to clarify their purpose and functionality.
- Enhanced documentation for various methods across multiple classes, improving clarity on parameters and return values.
- Improved the main function and other utility functions with detailed docstrings for better understanding and maintainability.
docs: enhance docstrings in Formatters class
- Improved clarity and consistency of docstrings across multiple methods in the Formatters class.
- Added detailed descriptions for method arguments and return values to facilitate better understanding and usage.
- Ensured that all public methods now have comprehensive documentation, enhancing maintainability and readability.
refactor: consolidate report handling in Formatters class
- Introduced a new method, manage_and_send_report, to streamline success and error report generation.
- Improved code readability by reducing duplication in report handling logic.
- Maintained existing functionality while enhancing the structure of report messages.
SIENTIAPDE-1184
refactor: improve schedule reporting in Formatters class
- Introduced a unified reporting structure for created, updated, and deleted schedules.
- Enhanced success and error report messages for clarity, including specific schedule types in notifications.
- Updated attachment handling in success and error reports to improve context and readability.
fix: enhance error reporting in Formatters class
- Updated the parse_report_schedule method to return a dictionary for error keys, including both message and attachment.
- Improved error report formatting in send_error_report method to include attachments for better context in failure notifications.
fix: enhance error reporting in TemporalManager class
- Added typed_search_attributes to the schedule creation for improved search capabilities.
- Included traceback in error messages when schedule deletion fails, providing better context for debugging.
fix: update notification IDs for error reporting in Formatters class
- Changed notification IDs for error reports related to schedule creation, update, and deletion to include the suffix '_ERROR' for better clarity in error handling.
fix: add conditional check for data_filter in MongoDB activity
- Updated the insert_many method to only execute if data_filter is not empty, preventing unnecessary database operations and potential errors.
feat: enhance notification filtering in Formatters and SlotManager classes
- Added logic to remove receiver groups with no notifications, improving the efficiency of notification handling.
- Implemented early return in Alerts and Reports workflows when there are no notifications or receiver groups, streamlining the processing flow.
