SIENTIAPDE-1184
refactor: improve schedule reporting in Formatters class
- Introduced a unified reporting structure for created, updated, and deleted schedules.
- Enhanced success and error report messages for clarity, including specific schedule types in notifications.
- Updated attachment handling in success and error reports to improve context and readability.
fix: enhance error reporting in Formatters class
- Updated the parse_report_schedule method to return a dictionary for error keys, including both message and attachment.
- Improved error report formatting in send_error_report method to include attachments for better context in failure notifications.
fix: enhance error reporting in TemporalManager class
- Added typed_search_attributes to the schedule creation for improved search capabilities.
- Included traceback in error messages when schedule deletion fails, providing better context for debugging.
fix: update notification IDs for error reporting in Formatters class
- Changed notification IDs for error reports related to schedule creation, update, and deletion to include the suffix '_ERROR' for better clarity in error handling.
fix: add conditional check for data_filter in MongoDB activity
- Updated the insert_many method to only execute if data_filter is not empty, preventing unnecessary database operations and potential errors.
feat: enhance notification filtering in Formatters and SlotManager classes
- Added logic to remove receiver groups with no notifications, improving the efficiency of notification handling.
- Implemented early return in Alerts and Reports workflows when there are no notifications or receiver groups, streamlining the processing flow.
feat: integrate metrics tracking for sent emails in Email activity
- Added metrics tracking for the number of emails sent, incorporating pod ID, model name, pipeline name, and email group for better monitoring.
- Enhanced logging to provide detailed information on email sending status for each receiver group.
feat: update dependencies and enhance metrics integration
- Updated sientia-dataops-library dependency version from 1.3.5 to 1.3.7 in requirements.txt.
- Incremented image tag in values.yaml from 0.2.7 to 0.3.1.
- Added Prometheus metrics service configuration in values.yaml, enabling metrics collection.
- Implemented Prometheus client in worker.py to start a metrics server and track application health.
fix: add conditional login for SMTP server and enhance test coverage
- Updated the Email class to conditionally log in to the SMTP server only if a sender password is provided, improving security.
- Enhanced test cases for the try_send_email method to cover various scenarios, including successful email sending and handling SMTP disconnections.
fix: add timeout to SMTP connection for improved reliability
- Updated the SMTP connection in the Email class to include a timeout of 20 seconds, enhancing the robustness of email sending operations.
- Adjusted the reconnection logic to maintain consistent timeout settings during server initialization.
refactor: rename send_email method to try_send_email and update parameter type
- Renamed the send_email method to try_send_email for clarity in its purpose.
- Changed the receivers parameter type from list[str] to str to simplify the method signature.
SIENTIAPDE-1172
feat: update email configuration and enhance email sending functionality
- Changed EMAIL_SENDER in values.yaml to a new email address.
- Updated EMAIL_SENDER_PASSWORD key reference in values.yaml for improved security.
- Added PostgreSQL configuration parameters to values.yaml for database connectivity.
- Refactored Email class to include a new send_email method for better error handling during email transmission.
- Implemented reconnection logic for SMTP server disconnections to ensure reliable email delivery.
feat: enhance notification handling in Formatters and SlotManager classes
- Introduced a mechanism to prevent duplicate notifications by tracking already added keys.
- Updated notification filtering logic to ensure unique notifications are processed for each receiver group.
- Improved overall notification management in orchestrator activities.
feat: enhance email notifications and orchestrator activities for reports
- Updated email HTML structure to clearly present errors, warnings, and info notifications.
- Introduced a new method in the Formatters class to filter notification reports based on group configurations.
- Enhanced the Reports workflow to integrate the new filtering functionality and manage notification packages effectively.
- Adjusted SlotManager to store timestamps with mail type specificity for better tracking.
- Improved test coverage for the new filtering functionality in the Formatters class.
refactor: streamline email generation and enhance notification handling
- Updated email HTML structure to improve clarity and presentation of notifications.
- Enhanced notification generator to better manage execution counts and output formatting.
- Refactored Email class to improve error handling and logging during email sending.
- Adjusted EmailBuilder to efficiently process error, warning, and info models.
- Improved test coverage for email and notification functionalities to ensure reliability.
refactor: update email HTML structure and enhance notification generator
- Modified email HTML to reflect updated model names and error/warning messages.
- Adjusted notification generator to improve execution counts and streamline output handling.
- Refactored activities in orchestrator to correct parameter names and improve type hints for better clarity.
- Enhanced email builder to handle error, warning, and info models more efficiently.
feat: integrate email configuration and enhance orchestrator activities
- Added email configuration parameters to values.yaml for sender details and SMTP settings.
- Refactored Email class to correct parameter names and improve logging during initialization.
- Introduced build_email_config function to streamline email configuration retrieval.
- Updated worker.py to include email and Postgres configurations in the main orchestration process.
feat: enhance email and slot manager activities with new filtering and caching functionalities
- Updated Email class to improve attachment handling and logging.
- Refactored Formatters class to process notifications more effectively by grouping them based on their configurations.
- Introduced new methods in SlotManager for filtering notification alerts and storing notification cache.
- Enhanced Alerts workflow to integrate new filtering and caching activities.
- Added tests for new functionalities in SlotManager and Formatters classes to ensure reliability.
refactor: simplify EmailBuilder parameters and remove unused templates
- Removed unused template file attributes from EmailBuilder class.
- Updated parameters method to eliminate report_data argument, streamlining the function for better clarity and efficiency.
feat: enhance orchestrator activities with new email and Postgres integrations
- Added Email and Postgres classes to the Activities class for improved functionality.
- Introduced new methods in MongoDB and SlotManager for loading and managing data.
- Updated requirements.txt to include jinja2.
- Added new formatting activity for log reports in Formatters class.
- Enhanced test coverage for MongoDB and SlotManager activities.
fix: update workflow_type in minimal_retrain function for consistency
- Changed workflow_type from "retrain" to "minimal_retrain" in the minimal_retrain function to align with the new schedule configuration.
SIENTIAPDE-1171 feat: add minimal retrain schedule configuration to samples.json and update formatters.py
- Added a new schedule configuration for the minimal retrain pipeline in samples.json.
- Updated the Formatters class in formatters.py to handle the new minimal retrain workflow type, integrating the minimal_retrain function for streamlined processing.
feat: add minimal_retrain function to orchestrator_functions.py for retraining workflows
- Introduced minimal_retrain function to streamline retraining configuration.
- The function integrates common configuration with specific parameters for retraining workflows.
feat: add TTL index functionality to MongoDB activities and update requirements
- Updated MongoDB class to accept TTL index duration in seconds.
- Implemented create_collection_with_ttl_index method to create collections with TTL indexes.
- Added environment variable for TTL index duration in connectors_config.
- Updated tests to cover new TTL index functionality and ensure proper behavior.
- Bumped sientia-dataops-library version to 1.3.4 in requirements.txt.
refactor: update requirements to point to local sientia-dataops-library path; modify orchestration and test files to use start_activity_method for consistency
refactor: update MongoDB notification IDs for error handling in activities; add tests for error scenarios in MongoDB and TemporalManager activities to ensure proper notification sending on failures
refactor: update notification handling in activities to include metadata in notifications, enhancing context for error reporting and success messages; update requirements to use version 1.3.0 of the sientia-dataops-library
refactor: update notification handler imports to use CoreNotificationHandler across activities and worker, enhancing consistency in notification management
refactor: update logging in Couchbase activity to use f-strings for improved readability; modify orchestration workflow to conditionally execute report handlers and replace method calls for consistency
refactor: update orchestration workflow to conditionally start schedule and slot report handlers based on report availability, improving clarity and efficiency
refactor: enhance schedule configuration in formatters to include updated_at timestamps for scouter and predictions_batch pipelines, improving state management
refactor: update samples.json to change server_id from 'default_server' to '1' for multiple tags; add 'active' and 'updated_at' fields for better state management in orchestration
refactor: update orchestration workflow to conditionally start activity methods for pipeline timestamps based on report availability, enhancing efficiency and clarity
refactor: update orchestration workflow to replace execute_local_activity_method with start_local_activity_method and execute_activity_method with start_activity_method for improved clarity and consistency
refactor: streamline MongoDB activity methods to improve readability and maintainability; replace inline query construction with variable assignments for updated, created, and deleted pipelines
refactor: restructure services in values.yaml to separate API and OPC configurations; update mongo_db.py to import datetime correctly and enhance readability
SIENTIAPDE-1150: enhance MongoDB activities to filter successful pipelines; update orchestration workflow to include timestamp management for updated, created, and deleted pipelines; adjust tests to reflect new success criteria
chore: update image tag to 0.2.4 in values.yaml; refactor logging in slot_manager and temporal_manager for improved readability; modify MongoDB and Redis configurations for better defaults; enhance worker.py with new activities and error handling
fix: implement schedule normalization and timestamp management in MongoDB activities, enhance orchestration workflow with new formatting and normalization methods
