Update GITHUB_BRANCH to feature/SIENTIAPDE-1445 and enhance formatters with a new registry for workflow types, improving configuration management. Refactor process_schedules to utilize the new registry and add error handling for unsupported workflow types. Update tests to validate new functionality and ensure proper integration.
SIENTIAPDE-1273 Add base_scouter and pi_web_api_scouter functions to orchestrator utilities for enhanced configuration management. Update tests to validate new functionalities and ensure proper integration with existing workflows.
SIENTIAPDE-1273 Add 'save_transform' and 'transform_table_name' parameters to predictions_batch function for enhanced data processing capabilities. Update corresponding test cases to reflect these changes.
Enhance orchestrator utilities and activities with improved documentation and functionality. Update README to reflect new orchestrator functions for drift and simple metrics workflows. Refactor email, formatters, mongo_db, slot_manager, and temporal_manager activities to include detailed docstrings and improve clarity on input parameters and return values. Ensure consistent metadata handling across activities for better logging and tracking.
SIENTIAPDE-1273 Refactor drift function in orchestrator utilities for improved configuration and update formatters to include drift import. Enhance tests to validate new configurations and ensure integration with existing workflows.
Implement simple metrics configuration in formatters and orchestrator utilities. Update tests to validate new simple metrics functionality and ensure proper integration with existing workflows.
Implement drift monitoring configuration and update formatters to support drift workflows. Add drift function to orchestrator utilities and enhance tests for drift functionality.
SIENTIAPDE-1312: Refactor build_tag_config to accept a list of tags and return notifications for missing servers. Update Formatters to handle notifications during slot configuration. Enhance tests to validate new behavior and error reporting.
SIENTIAPDE-1312: Update build_tag_config to ensure minimum frequency defaults to 1000 and convert frequency values to integers for accurate calculations of subscription_period_ms.
SIENTIAPDE-1312: Update build_tag_config to calculate subscription_period_ms based on minimum frequency from tags. Enhance test_build_tag_config to include frequency and validate subscription_period_ms.
Update sientia-dataops-library dependency to version 1.4.7 and enhance TemporalManager to support offset configuration for scheduling. Added offset handling in common configuration and updated related tests to validate new functionality.
Enhance README and code documentation for clarity and structure
- Added a comprehensive Table of Contents to the README for easier navigation.
- Introduced a new section on Code Quality & Validation, detailing the validation workflow and tools used.
- Improved descriptions and consistency in docstrings across various modules, including metrics, formatters, and MongoDB activities.
- Clarified function return types and arguments in the MongoDB and TemporalManager classes for better understanding.
Enhance type hinting and code structure in orchestrator activities
- Added type hints for better code clarity in `formatters.py`, `mongo_db.py`, `temporal_manager.py`, and `email_builder.py`.
- Updated `.gitignore` to include `coverage.xml` for improved coverage reporting.
- Improved readability by restructuring variable declarations and method signatures.
Refactor orchestrator activities and configuration files
- Removed unused Temporal timeout environment variables from values.yaml.
- Reorganized import statements in various activity files for better readability.
- Updated logging messages to use consistent formatting across activities.
- Enhanced test cases to ensure proper initialization and shutdown of orchestrator activities.
- Improved overall code structure and readability by applying consistent formatting and style adjustments.
SIENTIAPDE-1231: Add timeout configuration for Temporal tasks
- Introduced new timeout parameters in values.yaml for task execution.
- Updated TemporalManager to utilize execution and task timeout settings in seconds.
- Enhanced orchestrator_functions.py to include default timeout values for better orchestration control.
Update orchestration configuration and execution parameters
- Adjusted execution counts in Jupyter notebook cells for better tracking.
- Updated schedule name in the orchestration configuration to align with new requirements.
- Modified `values.yaml` to change the GitHub branch for improved versioning.
- Enhanced `Activities` and `TemporalManager` classes to include timeout parameters for task execution.
- Updated connector configuration to include new timeout settings for better orchestration control.
SIENTIAPDE-1222 Refactor server configuration handling in orchestrator_functions.py and update related tests
- Removed the security_spec field from server configuration in build_tag_config to streamline the configuration process.
- Updated test cases in test_formatters.py and test_orchestrator_functions.py to reflect the removal of security_spec, ensuring alignment with the new configuration structure and improving test accuracy.
Enhance OPC server configuration in orchestrator_functions.py
- Added optional fields for certificate paths in the build_tag_config function to improve security configuration flexibility.
Refactor orchestration and email handling
- Updated the orchestration queries to streamline model lookups and added conditions for active models.
- Changed the entry point in the local run script to use the worker module.
- Removed the deprecated samples.json file.
- Enhanced the Email activity to skip sending if the SMTP server is not configured.
- Added timestamp field handling in MongoDB activities for better data management.
- Updated connectors configuration to allow for a None SMTP server.
- Improved the common configuration function to include model configuration details.
Update environment variables in values.yaml and enhance predictions_batch function in orchestrator_functions.py
- Changed GITHUB_BRANCH to reflect the new task for adjusting laborious processes to capture timestamps from MLflow.
- Added a new filter configuration for handling empty data in the predictions_batch function to improve error management.
Remove deprecated files and configurations
- Deleted coverage.sh, Dockerfile, docker-compose.yml, email.html, input_sample.json, and Makefile as part of the cleanup process.
- Updated README.md to reflect the removal of these components and provide a clearer overview of the project structure and functionality.
- Added module-level docstrings to orchestrator, activities, and workflows for improved documentation and clarity on system architecture.
SIENTIAPDE-1205: Enhance predictions_batch function and update test cases
- Added 'predictions_storage_policy' to the predictions_batch function for improved configuration handling.
- Updated test cases to include 'predictions_storage_policy' in expected results, ensuring consistency in testing.
chore: update image tag and enhance datetime column handling in orchestrator functions
- Updated the image tag in values.yaml from 0.4.2 to 0.4.4 for the latest version.
- Added support for datetime_columns in minimal_retrain and predictions_batch functions to improve timestamp handling in orchestrator functions.
refactor: update date handling and remove deprecated constants
- Replaced the usage of datetime.now() with a new now() function from sientia_do.temporal.constants for consistent timestamp handling across multiple files.
- Updated the date format from DEFAULT_DATE_FORMAT to DATETIME_FORMAT_MS_WITH_TZ in various activities to ensure uniformity in timestamp formatting.
- Removed the obsolete patterns.py file that contained the DEFAULT_DATE_FORMAT constant.
chore: update dependencies and refactor logger imports
- Updated the sientia-dataops-library dependency version from 1.3.8 to 1.4.1 in requirements.txt.
- Changed image tag in values.yaml from 0.4.0 to 0.4.2.
- Refactored logger imports in multiple files to use the new observability module instead of the temporal module.
- Updated retry policy imports in various workflow files to reflect the new structure.
refactor: enhance shutdown procedures and documentation across activities
- Added shutdown methods to MongoDB and Email classes to ensure proper resource cleanup.
- Updated docstrings for shutdown methods to clarify their purpose and functionality.
- Enhanced documentation for various methods across multiple classes, improving clarity on parameters and return values.
- Improved the main function and other utility functions with detailed docstrings for better understanding and maintainability.
refactor: streamline email generation and enhance notification handling
- Updated email HTML structure to improve clarity and presentation of notifications.
- Enhanced notification generator to better manage execution counts and output formatting.
- Refactored Email class to improve error handling and logging during email sending.
- Adjusted EmailBuilder to efficiently process error, warning, and info models.
- Improved test coverage for email and notification functionalities to ensure reliability.
refactor: update email HTML structure and enhance notification generator
- Modified email HTML to reflect updated model names and error/warning messages.
- Adjusted notification generator to improve execution counts and streamline output handling.
- Refactored activities in orchestrator to correct parameter names and improve type hints for better clarity.
- Enhanced email builder to handle error, warning, and info models more efficiently.
feat: integrate email configuration and enhance orchestrator activities
- Added email configuration parameters to values.yaml for sender details and SMTP settings.
- Refactored Email class to correct parameter names and improve logging during initialization.
- Introduced build_email_config function to streamline email configuration retrieval.
- Updated worker.py to include email and Postgres configurations in the main orchestration process.
refactor: simplify EmailBuilder parameters and remove unused templates
- Removed unused template file attributes from EmailBuilder class.
- Updated parameters method to eliminate report_data argument, streamlining the function for better clarity and efficiency.
feat: enhance orchestrator activities with new email and Postgres integrations
- Added Email and Postgres classes to the Activities class for improved functionality.
- Introduced new methods in MongoDB and SlotManager for loading and managing data.
- Updated requirements.txt to include jinja2.
- Added new formatting activity for log reports in Formatters class.
- Enhanced test coverage for MongoDB and SlotManager activities.
fix: update workflow_type in minimal_retrain function for consistency
- Changed workflow_type from "retrain" to "minimal_retrain" in the minimal_retrain function to align with the new schedule configuration.
feat: add minimal_retrain function to orchestrator_functions.py for retraining workflows
- Introduced minimal_retrain function to streamline retraining configuration.
- The function integrates common configuration with specific parameters for retraining workflows.
feat: add TTL index functionality to MongoDB activities and update requirements
- Updated MongoDB class to accept TTL index duration in seconds.
- Implemented create_collection_with_ttl_index method to create collections with TTL indexes.
- Added environment variable for TTL index duration in connectors_config.
- Updated tests to cover new TTL index functionality and ensure proper behavior.
- Bumped sientia-dataops-library version to 1.3.4 in requirements.txt.
chore: update image tag to 0.2.4 in values.yaml; refactor logging in slot_manager and temporal_manager for improved readability; modify MongoDB and Redis configurations for better defaults; enhance worker.py with new activities and error handling
fix: implement schedule normalization and timestamp management in MongoDB activities, enhance orchestration workflow with new formatting and normalization methods
Refactor predictions_batch to use server_id instead of server_name; update values.yaml to set GITHUB_BRANCH to main; add new laborious pipeline configuration in samples.json
