feat: add shutdown tests for Activities and Email classes
- Implemented unit tests for the shutdown methods in the Activities and Email classes to ensure proper resource cleanup.
- Verified that the shutdown processes for MongoDB, PostgreSQL, and Email are called as expected during the shutdown sequence.
chore: update helm chart version in values.yaml
- Updated the version number for the sientia-orchestrator-worker helm chart from 0.4.0 to 0.5.0 in values.yaml to reflect the latest release.
feat: add SDK metrics service and update telemetry configuration
- Introduced a new service for SDK metrics in values.yaml, enabling metrics collection on port 9091.
- Updated the worker.py to initialize a new runtime with Prometheus metrics configuration for the SDK metrics server.
- Adjusted environment variable for GITHUB_BRANCH to reflect the current focus on testing and performance design.
docs: update README.md to provide comprehensive overview of SIENTIA DataOps Orchestrator Temporal
- Added detailed project overview, goals, and architecture sections.
- Documented main workflows including orchestrator, alerts, and reports workflows with input parameters and functionalities.
- Included environment variable configurations for Redis, MongoDB, PostgreSQL, Couchbase, email, Temporal, application, and Kafka.
- Described deployment strategies and monitoring capabilities, including Prometheus metrics.
- Enhanced usage instructions for triggering workflows and logging operations.
refactor: enhance shutdown procedures and documentation across activities
- Added shutdown methods to MongoDB and Email classes to ensure proper resource cleanup.
- Updated docstrings for shutdown methods to clarify their purpose and functionality.
- Enhanced documentation for various methods across multiple classes, improving clarity on parameters and return values.
- Improved the main function and other utility functions with detailed docstrings for better understanding and maintainability.
docs: enhance docstrings in Formatters class
- Improved clarity and consistency of docstrings across multiple methods in the Formatters class.
- Added detailed descriptions for method arguments and return values to facilitate better understanding and usage.
- Ensured that all public methods now have comprehensive documentation, enhancing maintainability and readability.
refactor: consolidate report handling in Formatters class
- Introduced a new method, manage_and_send_report, to streamline success and error report generation.
- Improved code readability by reducing duplication in report handling logic.
- Maintained existing functionality while enhancing the structure of report messages.
SIENTIAPDE-1184
refactor: improve schedule reporting in Formatters class
- Introduced a unified reporting structure for created, updated, and deleted schedules.
- Enhanced success and error report messages for clarity, including specific schedule types in notifications.
- Updated attachment handling in success and error reports to improve context and readability.
fix: enhance error reporting in Formatters class
- Updated the parse_report_schedule method to return a dictionary for error keys, including both message and attachment.
- Improved error report formatting in send_error_report method to include attachments for better context in failure notifications.
fix: enhance error reporting in TemporalManager class
- Added typed_search_attributes to the schedule creation for improved search capabilities.
- Included traceback in error messages when schedule deletion fails, providing better context for debugging.
fix: update notification IDs for error reporting in Formatters class
- Changed notification IDs for error reports related to schedule creation, update, and deletion to include the suffix '_ERROR' for better clarity in error handling.
fix: add conditional check for data_filter in MongoDB activity
- Updated the insert_many method to only execute if data_filter is not empty, preventing unnecessary database operations and potential errors.
SIENTIAPDE-1174
chore: update sientia-dataops-library dependency version from 1.3.7 to 1.3.8 in requirements.txt
- Removed unused test group from test_formatters.py.
- Added new test case for handling scenarios with no log reports in test_alerts.py.
- Added new test case for handling scenarios with no groups in test_reports.py.
chore: update helm upgrade command in values.yaml to reflect version change
- Modified the helm upgrade command in values.yaml to use version 0.4.0 instead of 0.4.0-uat for consistency in deployment.
feat: enhance notification filtering in Formatters and SlotManager classes
- Added logic to remove receiver groups with no notifications, improving the efficiency of notification handling.
- Implemented early return in Alerts and Reports workflows when there are no notifications or receiver groups, streamlining the processing flow.
feat: integrate metrics tracking for sent emails in Email activity
- Added metrics tracking for the number of emails sent, incorporating pod ID, model name, pipeline name, and email group for better monitoring.
- Enhanced logging to provide detailed information on email sending status for each receiver group.
feat: update dependencies and enhance metrics integration
- Updated sientia-dataops-library dependency version from 1.3.5 to 1.3.7 in requirements.txt.
- Incremented image tag in values.yaml from 0.2.7 to 0.3.1.
- Added Prometheus metrics service configuration in values.yaml, enabling metrics collection.
- Implemented Prometheus client in worker.py to start a metrics server and track application health.
fix: add conditional login for SMTP server and enhance test coverage
- Updated the Email class to conditionally log in to the SMTP server only if a sender password is provided, improving security.
- Enhanced test cases for the try_send_email method to cover various scenarios, including successful email sending and handling SMTP disconnections.
fix: add timeout to SMTP connection for improved reliability
- Updated the SMTP connection in the Email class to include a timeout of 20 seconds, enhancing the robustness of email sending operations.
- Adjusted the reconnection logic to maintain consistent timeout settings during server initialization.
refactor: rename send_email method to try_send_email and update parameter type
- Renamed the send_email method to try_send_email for clarity in its purpose.
- Changed the receivers parameter type from list[str] to str to simplify the method signature.
SIENTIAPDE-1172
feat: update email configuration and enhance email sending functionality
- Changed EMAIL_SENDER in values.yaml to a new email address.
- Updated EMAIL_SENDER_PASSWORD key reference in values.yaml for improved security.
- Added PostgreSQL configuration parameters to values.yaml for database connectivity.
- Refactored Email class to include a new send_email method for better error handling during email transmission.
- Implemented reconnection logic for SMTP server disconnections to ensure reliable email delivery.
Update dependencies in requirements.txt and increment image tag in values.yaml
- Updated sientia-dataops-library dependency version from 1.3.4 to 1.3.5.
- Changed image tag in values.yaml from 0.2.5 to 0.2.7 for the latest image version.
Update values.yaml and tests to incorporate mail_type parameter for notification handling
- Changed GITHUB_BRANCH in values.yaml to reflect the new alert pipeline.
- Added 'mail_type' parameter to various test cases in SlotManager and notification workflows to enhance notification specificity and tracking.
- Ensured consistency in test data structures across multiple test functions.
feat: enhance notification handling in Formatters and SlotManager classes
- Introduced a mechanism to prevent duplicate notifications by tracking already added keys.
- Updated notification filtering logic to ensure unique notifications are processed for each receiver group.
- Improved overall notification management in orchestrator activities.
feat: enhance email notifications and orchestrator activities for reports
- Updated email HTML structure to clearly present errors, warnings, and info notifications.
- Introduced a new method in the Formatters class to filter notification reports based on group configurations.
- Enhanced the Reports workflow to integrate the new filtering functionality and manage notification packages effectively.
- Adjusted SlotManager to store timestamps with mail type specificity for better tracking.
- Improved test coverage for the new filtering functionality in the Formatters class.
refactor: streamline email generation and enhance notification handling
- Updated email HTML structure to improve clarity and presentation of notifications.
- Enhanced notification generator to better manage execution counts and output formatting.
- Refactored Email class to improve error handling and logging during email sending.
- Adjusted EmailBuilder to efficiently process error, warning, and info models.
- Improved test coverage for email and notification functionalities to ensure reliability.
refactor: update email HTML structure and enhance notification generator
- Modified email HTML to reflect updated model names and error/warning messages.
- Adjusted notification generator to improve execution counts and streamline output handling.
- Refactored activities in orchestrator to correct parameter names and improve type hints for better clarity.
- Enhanced email builder to handle error, warning, and info models more efficiently.
feat: integrate email configuration and enhance orchestrator activities
- Added email configuration parameters to values.yaml for sender details and SMTP settings.
- Refactored Email class to correct parameter names and improve logging during initialization.
- Introduced build_email_config function to streamline email configuration retrieval.
- Updated worker.py to include email and Postgres configurations in the main orchestration process.
feat: enhance email and slot manager activities with new filtering and caching functionalities
- Updated Email class to improve attachment handling and logging.
- Refactored Formatters class to process notifications more effectively by grouping them based on their configurations.
- Introduced new methods in SlotManager for filtering notification alerts and storing notification cache.
- Enhanced Alerts workflow to integrate new filtering and caching activities.
- Added tests for new functionalities in SlotManager and Formatters classes to ensure reliability.
refactor: simplify EmailBuilder parameters and remove unused templates
- Removed unused template file attributes from EmailBuilder class.
- Updated parameters method to eliminate report_data argument, streamlining the function for better clarity and efficiency.
feat: enhance orchestrator activities with new email and Postgres integrations
- Added Email and Postgres classes to the Activities class for improved functionality.
- Introduced new methods in MongoDB and SlotManager for loading and managing data.
- Updated requirements.txt to include jinja2.
- Added new formatting activity for log reports in Formatters class.
- Enhanced test coverage for MongoDB and SlotManager activities.
feat: enhance test coverage for minimal_retrain workflow
- Added a new test for the minimal_retrain function to validate its configuration output.
- Updated the test_process_schedules to include a new schedule entry for minimal_retrain, ensuring comprehensive testing of the workflow integration.
fix: update workflow_type in minimal_retrain function for consistency
- Changed workflow_type from "retrain" to "minimal_retrain" in the minimal_retrain function to align with the new schedule configuration.
SIENTIAPDE-1171 feat: add minimal retrain schedule configuration to samples.json and update formatters.py
- Added a new schedule configuration for the minimal retrain pipeline in samples.json.
- Updated the Formatters class in formatters.py to handle the new minimal retrain workflow type, integrating the minimal_retrain function for streamlined processing.
feat: add minimal_retrain function to orchestrator_functions.py for retraining workflows
- Introduced minimal_retrain function to streamline retraining configuration.
- The function integrates common configuration with specific parameters for retraining workflows.
