Enhance OPC server configuration in orchestrator_functions.py
- Added optional fields for certificate paths in the build_tag_config function to improve security configuration flexibility.
Enhance README and activities with advanced notification filtering features
- Updated README to include new intelligent notification filtering capabilities, including TTL-based duplicate prevention and ignore lists.
- Enhanced `Formatters` activity to support notification filtering for scheduled reports and added detailed filtering logic.
- Improved `SlotManager` to implement advanced notification filtering with persistent alert detection and group-based filtering.
- Updated `Alerts` workflow to incorporate intelligent filtering for real-time error notifications, ensuring efficient alert delivery.
Refactor orchestration and email handling
- Updated the orchestration queries to streamline model lookups and added conditions for active models.
- Changed the entry point in the local run script to use the worker module.
- Removed the deprecated samples.json file.
- Enhanced the Email activity to skip sending if the SMTP server is not configured.
- Added timestamp field handling in MongoDB activities for better data management.
- Updated connectors configuration to allow for a None SMTP server.
- Improved the common configuration function to include model configuration details.
SIENTIAPDE-1206: Add early return for empty receiver groups in Alerts workflow
- Implemented an early return in the Alerts class to handle cases where no receiver groups are provided, preventing unnecessary processing in the notification workflow.
Update environment variables in values.yaml and enhance predictions_batch function in orchestrator_functions.py
- Changed GITHUB_BRANCH to reflect the new task for adjusting laborious processes to capture timestamps from MLflow.
- Added a new filter configuration for handling empty data in the predictions_batch function to improve error management.
Remove deprecated files and configurations
- Deleted coverage.sh, Dockerfile, docker-compose.yml, email.html, input_sample.json, and Makefile as part of the cleanup process.
- Updated README.md to reflect the removal of these components and provide a clearer overview of the project structure and functionality.
- Added module-level docstrings to orchestrator, activities, and workflows for improved documentation and clarity on system architecture.
SIENTIAPDE-1205: Enhance predictions_batch function and update test cases
- Added 'predictions_storage_policy' to the predictions_batch function for improved configuration handling.
- Updated test cases to include 'predictions_storage_policy' in expected results, ensuring consistency in testing.
SIENTIAPDE-1205: Update timestamp handling in MongoDB activities
- Modified timestamp formatting in mongo_db.py to apply UTC timezone before string conversion, ensuring consistency with DATETIME_FORMAT_MS_WITH_TZ.
SIENTIAPDE-1205: Simplify timestamp formatting in MongoDB activities
- Removed timezone conversion from timestamp handling in mongo_db.py, streamlining the formatting process while maintaining consistency with DATETIME_FORMAT_MS_WITH_TZ.
SIENTIAPDE-1205: Refactor timestamp handling in MongoDB activities
- Updated the import statement in mongo_db.py to include timezone support.
- Modified timestamp formatting to ensure UTC timezone is applied before string conversion, enhancing consistency in datetime handling.
SIENTIAPDE-1205: Update GITHUB_BRANCH and standardize datetime handling in MongoDB activities
- Changed GITHUB_BRANCH in values.yaml from "main" to "SIENTIAPDE-1205-alterar-opc-para-assincrono".
- Updated datetime parsing in mongo_db.py to use DATETIME_FORMAT_MS_WITH_TZ for consistent timestamp handling.
Update orchestration notebook execution count, modify image tag in values.yaml, and standardize datetime handling in MongoDB activities
- Changed execution count in init_orchestration.ipynb from 2 to 1.
- Updated image tag in values.yaml from 0.4.4 to 0.4.5.
- Replaced DATETIME_FORMAT_MS_WITH_TZ with DATETIME_FORMAT_MS in mongo_db.py for consistent timestamp handling.
chore: update image tag and enhance datetime column handling in orchestrator functions
- Updated the image tag in values.yaml from 0.4.2 to 0.4.4 for the latest version.
- Added support for datetime_columns in minimal_retrain and predictions_batch functions to improve timestamp handling in orchestrator functions.
refactor: standardize timestamp handling across MongoDB and SlotManager
- Replaced instances of `now()` with `date_now` for consistent timestamp usage in MongoDB and SlotManager activities.
- Updated test cases to mock the new `now()` function, ensuring accurate timestamp handling in unit tests.
- Adjusted timestamp formatting in various tests to align with the DATETIME_FORMAT_MS_WITH_TZ standard.
refactor: update date handling and remove deprecated constants
- Replaced the usage of datetime.now() with a new now() function from sientia_do.temporal.constants for consistent timestamp handling across multiple files.
- Updated the date format from DEFAULT_DATE_FORMAT to DATETIME_FORMAT_MS_WITH_TZ in various activities to ensure uniformity in timestamp formatting.
- Removed the obsolete patterns.py file that contained the DEFAULT_DATE_FORMAT constant.
chore: update sientia-dataops-library version and modify environment variables
- Updated the sientia-dataops-library dependency version from 1.4.1 to 1.4.2 in requirements.txt.
- Changed the GITHUB_BRANCH value in values.yaml to reflect the new task SIENTIAPDE-1193.
- Enhanced the process_notifications workflow to include timestamp conversion metadata for improved data handling.
refactor: remove unused schedule_handles attribute from TemporalManager
- Eliminated the schedule_handles dictionary from the TemporalManager class as it was no longer needed.
- Cleaned up related code to ensure proper functionality without the removed attribute.
refactor: enhance logging and metadata handling across activities
- Updated logging statements in Email, Formatters, MongoDB, SlotManager, and TemporalManager classes to include metadata for better traceability.
- Improved success reporting in MongoDB, SlotManager, and TemporalManager by specifying the number of successfully processed items.
- Refactored logger usage in worker.py to utilize custom_info for enhanced logging consistency.
chore: update dependencies and refactor logger imports
- Updated the sientia-dataops-library dependency version from 1.3.8 to 1.4.1 in requirements.txt.
- Changed image tag in values.yaml from 0.4.0 to 0.4.2.
- Refactored logger imports in multiple files to use the new observability module instead of the temporal module.
- Updated retry policy imports in various workflow files to reflect the new structure.
feat: add SDK metrics service and update telemetry configuration
- Introduced a new service for SDK metrics in values.yaml, enabling metrics collection on port 9091.
- Updated the worker.py to initialize a new runtime with Prometheus metrics configuration for the SDK metrics server.
- Adjusted environment variable for GITHUB_BRANCH to reflect the current focus on testing and performance design.
refactor: enhance shutdown procedures and documentation across activities
- Added shutdown methods to MongoDB and Email classes to ensure proper resource cleanup.
- Updated docstrings for shutdown methods to clarify their purpose and functionality.
- Enhanced documentation for various methods across multiple classes, improving clarity on parameters and return values.
- Improved the main function and other utility functions with detailed docstrings for better understanding and maintainability.
docs: enhance docstrings in Formatters class
- Improved clarity and consistency of docstrings across multiple methods in the Formatters class.
- Added detailed descriptions for method arguments and return values to facilitate better understanding and usage.
- Ensured that all public methods now have comprehensive documentation, enhancing maintainability and readability.
refactor: consolidate report handling in Formatters class
- Introduced a new method, manage_and_send_report, to streamline success and error report generation.
- Improved code readability by reducing duplication in report handling logic.
- Maintained existing functionality while enhancing the structure of report messages.
SIENTIAPDE-1184
refactor: improve schedule reporting in Formatters class
- Introduced a unified reporting structure for created, updated, and deleted schedules.
- Enhanced success and error report messages for clarity, including specific schedule types in notifications.
- Updated attachment handling in success and error reports to improve context and readability.
fix: enhance error reporting in Formatters class
- Updated the parse_report_schedule method to return a dictionary for error keys, including both message and attachment.
- Improved error report formatting in send_error_report method to include attachments for better context in failure notifications.
fix: enhance error reporting in TemporalManager class
- Added typed_search_attributes to the schedule creation for improved search capabilities.
- Included traceback in error messages when schedule deletion fails, providing better context for debugging.
fix: update notification IDs for error reporting in Formatters class
- Changed notification IDs for error reports related to schedule creation, update, and deletion to include the suffix '_ERROR' for better clarity in error handling.
fix: add conditional check for data_filter in MongoDB activity
- Updated the insert_many method to only execute if data_filter is not empty, preventing unnecessary database operations and potential errors.
feat: enhance notification filtering in Formatters and SlotManager classes
- Added logic to remove receiver groups with no notifications, improving the efficiency of notification handling.
- Implemented early return in Alerts and Reports workflows when there are no notifications or receiver groups, streamlining the processing flow.
feat: integrate metrics tracking for sent emails in Email activity
- Added metrics tracking for the number of emails sent, incorporating pod ID, model name, pipeline name, and email group for better monitoring.
- Enhanced logging to provide detailed information on email sending status for each receiver group.
feat: update dependencies and enhance metrics integration
- Updated sientia-dataops-library dependency version from 1.3.5 to 1.3.7 in requirements.txt.
- Incremented image tag in values.yaml from 0.2.7 to 0.3.1.
- Added Prometheus metrics service configuration in values.yaml, enabling metrics collection.
- Implemented Prometheus client in worker.py to start a metrics server and track application health.
fix: add conditional login for SMTP server and enhance test coverage
- Updated the Email class to conditionally log in to the SMTP server only if a sender password is provided, improving security.
- Enhanced test cases for the try_send_email method to cover various scenarios, including successful email sending and handling SMTP disconnections.
fix: add timeout to SMTP connection for improved reliability
- Updated the SMTP connection in the Email class to include a timeout of 20 seconds, enhancing the robustness of email sending operations.
- Adjusted the reconnection logic to maintain consistent timeout settings during server initialization.
refactor: rename send_email method to try_send_email and update parameter type
- Renamed the send_email method to try_send_email for clarity in its purpose.
- Changed the receivers parameter type from list[str] to str to simplify the method signature.
SIENTIAPDE-1172
feat: update email configuration and enhance email sending functionality
- Changed EMAIL_SENDER in values.yaml to a new email address.
- Updated EMAIL_SENDER_PASSWORD key reference in values.yaml for improved security.
- Added PostgreSQL configuration parameters to values.yaml for database connectivity.
- Refactored Email class to include a new send_email method for better error handling during email transmission.
- Implemented reconnection logic for SMTP server disconnections to ensure reliable email delivery.
feat: enhance notification handling in Formatters and SlotManager classes
- Introduced a mechanism to prevent duplicate notifications by tracking already added keys.
- Updated notification filtering logic to ensure unique notifications are processed for each receiver group.
- Improved overall notification management in orchestrator activities.
feat: enhance email notifications and orchestrator activities for reports
- Updated email HTML structure to clearly present errors, warnings, and info notifications.
- Introduced a new method in the Formatters class to filter notification reports based on group configurations.
- Enhanced the Reports workflow to integrate the new filtering functionality and manage notification packages effectively.
- Adjusted SlotManager to store timestamps with mail type specificity for better tracking.
- Improved test coverage for the new filtering functionality in the Formatters class.
refactor: streamline email generation and enhance notification handling
- Updated email HTML structure to improve clarity and presentation of notifications.
- Enhanced notification generator to better manage execution counts and output formatting.
- Refactored Email class to improve error handling and logging during email sending.
- Adjusted EmailBuilder to efficiently process error, warning, and info models.
- Improved test coverage for email and notification functionalities to ensure reliability.
refactor: update email HTML structure and enhance notification generator
- Modified email HTML to reflect updated model names and error/warning messages.
- Adjusted notification generator to improve execution counts and streamline output handling.
- Refactored activities in orchestrator to correct parameter names and improve type hints for better clarity.
- Enhanced email builder to handle error, warning, and info models more efficiently.
feat: integrate email configuration and enhance orchestrator activities
- Added email configuration parameters to values.yaml for sender details and SMTP settings.
- Refactored Email class to correct parameter names and improve logging during initialization.
- Introduced build_email_config function to streamline email configuration retrieval.
- Updated worker.py to include email and Postgres configurations in the main orchestration process.
feat: enhance email and slot manager activities with new filtering and caching functionalities
- Updated Email class to improve attachment handling and logging.
- Refactored Formatters class to process notifications more effectively by grouping them based on their configurations.
- Introduced new methods in SlotManager for filtering notification alerts and storing notification cache.
- Enhanced Alerts workflow to integrate new filtering and caching activities.
- Added tests for new functionalities in SlotManager and Formatters classes to ensure reliability.
refactor: simplify EmailBuilder parameters and remove unused templates
- Removed unused template file attributes from EmailBuilder class.
- Updated parameters method to eliminate report_data argument, streamlining the function for better clarity and efficiency.
feat: enhance orchestrator activities with new email and Postgres integrations
- Added Email and Postgres classes to the Activities class for improved functionality.
- Introduced new methods in MongoDB and SlotManager for loading and managing data.
- Updated requirements.txt to include jinja2.
- Added new formatting activity for log reports in Formatters class.
- Enhanced test coverage for MongoDB and SlotManager activities.
fix: update workflow_type in minimal_retrain function for consistency
- Changed workflow_type from "retrain" to "minimal_retrain" in the minimal_retrain function to align with the new schedule configuration.
SIENTIAPDE-1171 feat: add minimal retrain schedule configuration to samples.json and update formatters.py
- Added a new schedule configuration for the minimal retrain pipeline in samples.json.
- Updated the Formatters class in formatters.py to handle the new minimal retrain workflow type, integrating the minimal_retrain function for streamlined processing.
feat: add minimal_retrain function to orchestrator_functions.py for retraining workflows
- Introduced minimal_retrain function to streamline retraining configuration.
- The function integrates common configuration with specific parameters for retraining workflows.
feat: add TTL index functionality to MongoDB activities and update requirements
- Updated MongoDB class to accept TTL index duration in seconds.
- Implemented create_collection_with_ttl_index method to create collections with TTL indexes.
- Added environment variable for TTL index duration in connectors_config.
- Updated tests to cover new TTL index functionality and ensure proper behavior.
- Bumped sientia-dataops-library version to 1.3.4 in requirements.txt.
