Update prediction_store_policy handling in workflows
- Added 'prediction_store_policy' to the input data handling in PredictionsBatch, ensuring a default value of 'lts:1' is used when not provided.
- Modified FormatAndExportPrediction to directly use 'prediction_store_policy' from input_data, removing the default fallback.
- Updated PredictionProcess to include 'prediction_store_policy' in the output data structure, ensuring consistency across workflows.
SIENTIAPDE-1214: Enhance MLFlow and tests with datetime index handling and logging improvements
- Added a new method in MLFlow to detect and parse datetime indices in DataFrames, ensuring proper format and raising errors for invalid types.
- Updated prediction workflows to utilize the new datetime index handling, improving data integrity during transformations.
- Enhanced logging in model_repository to include detailed data outputs for better traceability.
- Adjusted timeout settings in prediction workflows for improved execution time management.
- Updated tests.ipynb to include additional checks for index types and outputs for better validation of functionality.
SIENTIAPDE-1214: Refactor MLFlow and model repository methods to use model_config dictionary
- Updated MLFlow class methods to accept model_config instead of model_retention for improved flexibility.
- Modified model_repository methods to handle model_config, extracting necessary parameters for transformation and prediction.
- Adjusted predictions_batch and prediction_process workflows to utilize model_config for better configuration management.
- Commented out the previous sientia-mlops-library dependency in requirements.txt for clarity.
Refactor connectors_config.py and conditional_filters.py for improved configuration management and data filtering
- Updated PostgreSQL and MLFlow configuration functions to enhance default values and environment variable handling.
- Simplified OPC server configuration logic and improved MongoDB connection string construction.
- Refactored conditional filters to streamline null value checks and empty data validation, removing unnecessary comments and examples for clarity.
- Removed extensive module docstrings to enhance code readability.
Remove all module docstrings and the versioning information from the Laborious package, activities, utils, and workflows. This cleanup enhances code readability and reduces unnecessary comments in the codebase.
Remove Docker configuration files and refactor project structure
- Deleted docker-compose.yml and Dockerfile as part of the project restructuring.
- Updated README.md to reflect changes in project setup and configuration.
- Introduced a new __init__.py file in the laborious package to provide an overview of the system.
- Enhanced documentation across various modules, including metrics, activities, and workflows, to improve clarity and usability.
- Added comprehensive docstrings and comments to key classes and methods for better maintainability.
Implement prediction store policy handling in Gates activity
- Added a new method `get_prediction_store_policy` to validate and parse the prediction store policy.
- Updated `format_prediction` method to utilize the new policy handling, allowing for sorting of predictions based on the specified policy.
- Enhanced test coverage for the new policy handling, including various scenarios for valid and invalid policies.
- Removed the obsolete `coverage.sh` script.
Update dependencies and enhance datetime handling
- Bump sientia-dataops-library version in requirements.txt to 1.4.4.
- Update image tag in values.yaml to 0.4.4.
- Add 'datetime_columns' to metadata in MinimalRetrain and PredictionsBatch workflows for improved data handling.
Update Gates activity and format_and_export_prediction workflow to use DATETIME_FORMAT_WITH_TZ for consistent timestamp handling
- Modified Gates activity to correctly handle the maximum timestamp without formatting it prematurely.
- Updated format_and_export_prediction workflow to utilize DATETIME_FORMAT_WITH_TZ for timestamp conversion.
- Enhanced tests to ensure timestamp conversion is applied consistently across workflows.
Update requirements and values for SIENTIAPDE-1193
- Bump sientia-dataops-library version from 1.4.1 to 1.4.2 in requirements.txt.
- Change GITHUB_BRANCH in values.yaml to reflect the new task SIENTIAPDE-1193 regarding datetime writing in temporal.
Update dependencies and refactor logging imports for observability
- Updated the sientia-dataops-library dependency version to 1.4.0 in requirements.txt.
- Changed image tag in values.yaml from 0.3.2 to 0.4.1.
- Refactored logging imports across multiple files to use the new observability module instead of the temporal.utils.logger.
- Updated retry policy imports in workflow files to reflect the new module structure.
Update dependencies, modify replica count, and implement metrics tracking
- Updated sientia-dataops-library version from 1.3.5 to 1.3.7 in requirements.txt.
- Changed replicaCount in values.yaml from 5 to 3 and incremented image tag from 0.2.7 to 0.3.1.
- Added Prometheus metrics tracking in gates.py and worker.py, including a new write_metrics method.
- Configured Prometheus service and ServiceMonitor in values.yaml for metrics collection.
Update dependencies and enhance ML model retraining functionality
- Updated sientia-dataops-library version in requirements.txt from 1.3.3 to 1.3.4.
- Incremented image tag in values.yaml from 0.2.4 to 0.2.5 and added a new environment variable MONGODB_TTL_INDEX_HOURS.
- Introduced new methods in MLFlowRepository for model retraining and production model updates, including error handling and logging.
- Added retrain_model and update_production_model activities in mlflow.py to support model management workflows.
- Modified MongoDB connection settings in connectors_config.py for improved security and configuration flexibility.
Refactor OPC connection handling to improve error notifications and update GITHUB_BRANCH in values.yaml for MongoDB integration. Change requirements.txt to point to local dataops library path.
Refactor logging in MLFlow and Gates activities; remove print statement in FormatAndExportPrediction; update data handling in PredictionProcess; delete unused redis-feeder script.
Refactor and enhance the laborious workflow and utilities
- Removed outdated test file `test_predictions_batch.py` from workflows.
- Added `input_sample.json` for standardized input configuration.
- Introduced `connectors_config.py` to manage database and service configurations.
- Implemented a logging utility in `logger.py` for consistent logging across the application.
- Created `policies.py` to define retry policies for workflows.
- Developed comprehensive tests for `MLFlowRepository` in `test_model_repository.py`.
- Added extensive tests for `OpcRepository` in `test_opc_repository.py`.
- Updated `test_predictions_batch.py` to reflect new workflow structure and testing methodology.
Implement new get_last_timestamp method in Gates class, refactor MLFlow activity methods to return only transformed data, and update PredictionsBatch and PredictionProcess workflows to utilize Activities module. Add detailed docstrings for new methods and enhance test coverage for get_last_timestamp functionality.
Refactor activity methods and update requirements.txt to enhance functionality and remove deprecated filters. Added detailed docstrings for clarity and improved error handling in data processing workflows.
