SIENTIAPDE-1222
Enhance tests for email and MongoDB activities
- Added a test for sending emails when no SMTP server is configured, ensuring proper handling of such cases.
- Updated MongoDB tests to include timestamp fields in the results, improving data accuracy in document retrieval.
- Modified connector configuration tests to reflect changes in SMTP server settings.
Add test for handling cases with no receiver groups in Alerts workflow
- Implemented a new test case to verify the behavior of the Alerts workflow when no receiver groups are provided, ensuring that the workflow executes correctly without unnecessary processing.
SIENTIAPDE-1205: Enhance predictions_batch function and update test cases
- Added 'predictions_storage_policy' to the predictions_batch function for improved configuration handling.
- Updated test cases to include 'predictions_storage_policy' in expected results, ensuring consistency in testing.
SIENTIAPDE-1205: Update timestamp formatting in MongoDB tests
- Modified timestamp handling in test_mongo_db.py to utilize DATETIME_FORMAT_MS_WITH_TZ for consistent formatting across test cases.
- Ensured all timestamp strings include UTC timezone information for improved accuracy in datetime comparisons.
refactor: standardize timestamp handling across MongoDB and SlotManager
- Replaced instances of `now()` with `date_now` for consistent timestamp usage in MongoDB and SlotManager activities.
- Updated test cases to mock the new `now()` function, ensuring accurate timestamp handling in unit tests.
- Adjusted timestamp formatting in various tests to align with the DATETIME_FORMAT_MS_WITH_TZ standard.
fix: add metadata handling in schedule configuration tests
- Enhanced the test cases in test_formatters.py and test_temporal_manager.py to include metadata in the schedule configuration inputs.
- This change improves the accuracy and traceability of the tests by ensuring that metadata is properly accounted for in the test scenarios.
feat: add shutdown tests for Activities and Email classes
- Implemented unit tests for the shutdown methods in the Activities and Email classes to ensure proper resource cleanup.
- Verified that the shutdown processes for MongoDB, PostgreSQL, and Email are called as expected during the shutdown sequence.
SIENTIAPDE-1184
refactor: improve schedule reporting in Formatters class
- Introduced a unified reporting structure for created, updated, and deleted schedules.
- Enhanced success and error report messages for clarity, including specific schedule types in notifications.
- Updated attachment handling in success and error reports to improve context and readability.
fix: enhance error reporting in Formatters class
- Updated the parse_report_schedule method to return a dictionary for error keys, including both message and attachment.
- Improved error report formatting in send_error_report method to include attachments for better context in failure notifications.
SIENTIAPDE-1174
chore: update sientia-dataops-library dependency version from 1.3.7 to 1.3.8 in requirements.txt
- Removed unused test group from test_formatters.py.
- Added new test case for handling scenarios with no log reports in test_alerts.py.
- Added new test case for handling scenarios with no groups in test_reports.py.
fix: add conditional login for SMTP server and enhance test coverage
- Updated the Email class to conditionally log in to the SMTP server only if a sender password is provided, improving security.
- Enhanced test cases for the try_send_email method to cover various scenarios, including successful email sending and handling SMTP disconnections.
Update values.yaml and tests to incorporate mail_type parameter for notification handling
- Changed GITHUB_BRANCH in values.yaml to reflect the new alert pipeline.
- Added 'mail_type' parameter to various test cases in SlotManager and notification workflows to enhance notification specificity and tracking.
- Ensured consistency in test data structures across multiple test functions.
feat: enhance email notifications and orchestrator activities for reports
- Updated email HTML structure to clearly present errors, warnings, and info notifications.
- Introduced a new method in the Formatters class to filter notification reports based on group configurations.
- Enhanced the Reports workflow to integrate the new filtering functionality and manage notification packages effectively.
- Adjusted SlotManager to store timestamps with mail type specificity for better tracking.
- Improved test coverage for the new filtering functionality in the Formatters class.
refactor: streamline email generation and enhance notification handling
- Updated email HTML structure to improve clarity and presentation of notifications.
- Enhanced notification generator to better manage execution counts and output formatting.
- Refactored Email class to improve error handling and logging during email sending.
- Adjusted EmailBuilder to efficiently process error, warning, and info models.
- Improved test coverage for email and notification functionalities to ensure reliability.
feat: enhance email and slot manager activities with new filtering and caching functionalities
- Updated Email class to improve attachment handling and logging.
- Refactored Formatters class to process notifications more effectively by grouping them based on their configurations.
- Introduced new methods in SlotManager for filtering notification alerts and storing notification cache.
- Enhanced Alerts workflow to integrate new filtering and caching activities.
- Added tests for new functionalities in SlotManager and Formatters classes to ensure reliability.
refactor: simplify EmailBuilder parameters and remove unused templates
- Removed unused template file attributes from EmailBuilder class.
- Updated parameters method to eliminate report_data argument, streamlining the function for better clarity and efficiency.
feat: enhance orchestrator activities with new email and Postgres integrations
- Added Email and Postgres classes to the Activities class for improved functionality.
- Introduced new methods in MongoDB and SlotManager for loading and managing data.
- Updated requirements.txt to include jinja2.
- Added new formatting activity for log reports in Formatters class.
- Enhanced test coverage for MongoDB and SlotManager activities.
feat: enhance test coverage for minimal_retrain workflow
- Added a new test for the minimal_retrain function to validate its configuration output.
- Updated the test_process_schedules to include a new schedule entry for minimal_retrain, ensuring comprehensive testing of the workflow integration.
feat: add TTL index functionality to MongoDB activities and update requirements
- Updated MongoDB class to accept TTL index duration in seconds.
- Implemented create_collection_with_ttl_index method to create collections with TTL indexes.
- Added environment variable for TTL index duration in connectors_config.
- Updated tests to cover new TTL index functionality and ensure proper behavior.
- Bumped sientia-dataops-library version to 1.3.4 in requirements.txt.
refactor: update requirements to point to local sientia-dataops-library path; modify orchestration and test files to use start_activity_method for consistency
refactor: update MongoDB notification IDs for error handling in activities; add tests for error scenarios in MongoDB and TemporalManager activities to ensure proper notification sending on failures
refactor: update notification handling in activities to include metadata in notifications, enhancing context for error reporting and success messages; update requirements to use version 1.3.0 of the sientia-dataops-library
refactor: update logging in Couchbase activity to use f-strings for improved readability; modify orchestration workflow to conditionally execute report handlers and replace method calls for consistency
SIENTIAPDE-1150: enhance MongoDB activities to filter successful pipelines; update orchestration workflow to include timestamp management for updated, created, and deleted pipelines; adjust tests to reflect new success criteria
chore: update image tag to 0.2.4 in values.yaml; refactor logging in slot_manager and temporal_manager for improved readability; modify MongoDB and Redis configurations for better defaults; enhance worker.py with new activities and error handling
fix: implement schedule normalization and timestamp management in MongoDB activities, enhance orchestration workflow with new formatting and normalization methods
fix: update schedule handling in temporal_manager and formatters to support multiple namespaces, enhance test coverage, and improve configuration handling
- Changed search attribute key from "Orchestrated" to "orchestrated" in temporal_manager.py and test cases.
- Enhanced logging for schedule creation and updates in temporal_manager.py.
- Updated schedule creation to use workflow_type directly instead of a hardcoded string.
- Modified gather_read_tags function to use server_id instead of server_name for tag identification.
- Adjusted test cases to reflect changes in server_id usage and ensure consistency across tests.
- Fixed model_name retrieval in common_config to access nested models dictionary.
- Updated test cases to align with new data structures and ensure accurate assertions.
Add comprehensive tests and utility functions for orchestrator activities
- Introduced tests for the new Formatters class, covering methods for processing schedules and slots.
- Enhanced SlotManager tests with update and delete slot functionalities.
- Added TemporalManager tests for creating, updating, and deleting schedules, including frequency parsing.
- Implemented utility functions for orchestrator operations, including frequency parsing and tag configuration building.
- Created tests for utility functions to ensure correct behavior and integration with orchestrator activities.
- Established a new converters module for parsing frequency strings into seconds.
Add unit tests for orchestrator activities and workflows
- Implement tests for Activities class, covering initialization and prepare_activity method.
- Create tests for Couchbase class, including successful and failed query loading.
- Add tests for SlotManager class, verifying OPC slot loading and active ingestor retrieval.
- Develop tests for TemporalManager class, focusing on schedule loading functionality.
- Introduce tests for Orchestrator class, ensuring proper execution of workflow activities.
- Establish a new test suite for orchestrator activities and workflows in the tests directory.
Add unit tests for connectors configuration, logger, workflows, and predictions batch
- Implement tests for MLflow, OPC, and Postgres configuration builders to validate environment variable handling and default values.
- Create tests for the logger to ensure default settings and handler configurations are correct.
- Add comprehensive tests for the FormatAndExportPrediction and PredictionProcess workflows, covering various scenarios including path flags and activity execution.
- Introduce tests for the PredictionsBatch workflow to verify the execution of local activities and child workflows.
- Include a values.yaml file for Kubernetes deployment configuration, specifying image details, service account settings, environment variables, and resource limits.
