Update test_format_and_export_prediction.py to use prediction_data for workflow execution
- Replaced the mock return value with prediction_data in the test_run_none_path_flag function to ensure accurate testing of the workflow's data handling.
Enhance OPC Metrics Handling and Refactor Write Operations
- Updated the OPC class to return response times for write operations, improving metrics tracking.
- Refactored the Gates activity to incorporate OPC metrics into the metrics writing process.
- Adjusted the manage_output_tags method in OpcRepository to return response times for each tag written.
- Modified tests to validate the new metrics structure and ensure correct behavior of the updated methods.
SIENTIAPDE-1327: Add requirements file to quality gate workflow
- Included 'requirements-light.txt' in the quality gate workflow to specify additional dependencies for the CI process.
Update requirements and simplify CI workflow for quality checks
- Added new dependencies: kaleido, hyperopt, and shap to requirements.txt.
- Refactored the quality gate workflow to use a template for improved maintainability and reduced complexity.
Refactor error handling in MLFlowRepository and update tests
- Improved error message formatting in MLFlowRepository for better readability.
- Updated test assertions to ensure correct calls to MLflow methods during experiment retrieval and creation.
Update tests notebook execution count and bump image tag in values.yaml
- Incremented the execution count in the tests notebook for accurate tracking.
- Updated the image tag in values.yaml from "0.0.3" to "1.0.1" for versioning consistency.
- Enhanced error handling in model_repository.py to ensure proper retrieval of created experiments.
Refactor OPC handling by removing pod_id from initialization and updating logging format
- Removed pod_id parameter from OPC class and repository initialization to streamline connection management.
- Updated logging statements for improved readability during disconnection attempts and error handling.
Enhance OpcRepository client initialization and error handling
- Updated the Client instantiation in OpcRepository to include a timeout and watchdog interval for improved connection management.
- Added a disconnection call in the exception handling block to ensure proper resource cleanup during connection failures.
Update dependencies and enhance OpcRepository functionality
- Updated sientia-dataops-library version to 1.4.7 in requirements files.
- Modified GITHUB_BRANCH in values.yaml for improved pipeline management.
- Refactored OpcRepository class to inherit from BaseActivity, adding enhanced logging and error handling during disconnection.
- Implemented a disconnection fallback mechanism to ensure graceful handling of OPC server disconnections.
Enhance CI workflow for comprehensive code quality checks
- Added code formatting check, linting, type checking, and security analysis steps using Ruff, mypy, and Bandit.
- Integrated pytest for running tests, ensuring a robust quality gate in the CI pipeline.
Enhance validation script and CI workflow for improved code quality checks
- Added support for a --fix option in validate.sh to apply Ruff auto-fixes for formatting and linting.
- Updated GitHub Actions workflow to install development and runtime dependencies separately.
- Integrated the validation script execution into the CI pipeline to ensure code quality checks are performed automatically.
Enhance README with code quality and validation guidelines
- Added a new section on code quality and validation tools, detailing the use of Ruff, mypy, Bandit, and pytest.
- Included installation instructions for development dependencies and options for running validation scripts.
- Provided best practices for maintaining code quality and integrating CI/CD workflows.
Enhance README and repository utilities for clarity and functionality
- Updated README.md to improve descriptions and structure, adding detailed sections for features, workflows, and architecture.
- Enhanced MinioRepository with comprehensive docstrings for methods and class attributes, improving usability and documentation.
- Refined MLFlowRepository with clearer method descriptions and improved logging for better observability and maintainability.
Refactor tests and update model_repository.py for clarity and consistency
- Added a blank line in model_repository.py for improved readability.
- Adjusted formatting in test_mlflow.py to streamline assertions.
- Updated ensure_bucket_exists method tests in test_minio_repository.py to reflect the new return value of None instead of True.
Refactor MinioRepository and MLFlowRepository for improved functionality and error handling
- Updated `ensure_bucket_exists` method in MinioRepository to return None instead of a boolean, streamlining bucket existence checks.
- Replaced hardcoded error messages in MLFlowRepository with a constant for better maintainability.
- Adjusted column assignment in MLFlowRepository to use pd.Index for improved clarity.
Update .gitignore and remove values.yaml for project cleanup
- Added .ruff_cache/ and catboost_info/ to .gitignore to prevent tracking of temporary files and caches.
- Deleted values.yaml to remove outdated configuration settings, streamlining the project structure.
Enhance validation script and refactor code in various modules
- Updated the validation script to include automatic code formatting and linting fixes using Ruff.
- Removed the `clean_tmp_files` method from the Gates class to streamline functionality.
- Simplified conditional checks in the OpcRepository for better clarity and error handling.
- Added model ID to the minimal retrain workflow for improved tracking.
- Introduced new test cases for error handling in MLFlow and storage operations, ensuring robustness in repository interactions.
Update .gitignore and refactor metrics.py for improved logging and consistency
- Added coverage.xml to .gitignore to prevent tracking of coverage reports.
- Refactored metric labels in metrics.py for consistency in string formatting and improved readability.
- Enhanced logging messages in various activities to ensure uniformity in message formatting.
Refactor `force_memory_release` function in model_repository.py to improve logging
- Updated the `force_memory_release` function to remove metadata parameter and enhance logging by using `logger.info()` instead of `logger.custom_info()`.
- Adjusted calls to `force_memory_release` in the MLFlowRepository to pass the logger instance, ensuring consistent logging during memory management operations.
Refactor memory management in MLFlowRepository to use `force_memory_release` function
- Replaced direct calls to `gc.collect()` with `force_memory_release()` for improved memory optimization after model deletion.
- This change enhances memory management during model operations, ensuring more efficient resource handling.
Add memory management function to model_repository.py
- Introduced `force_memory_release` function to enhance memory management by triggering garbage collection and attempting to release unused memory.
- Utilized `ctypes` to call `malloc_trim` for further memory optimization, improving overall performance during model operations.
Enhance MLFlowRepository memory management by adding garbage collection and logging for model deletion
- Introduced garbage collection after model deletion to optimize memory usage.
- Added logging to inform when a model is deleted from memory, improving traceability during predictions.
Update values.yaml and model_repository.py for configuration and error handling improvements
- Increased replicaCount from 1 to 2 in values.yaml for enhanced scalability.
- Updated image tag from "0.0.2" to "0.0.3" in values.yaml to reflect the latest version.
- Enhanced error messaging in model_repository.py to include the actual index type when raising ValueError for index type validation.
Comment out CSV export lines in MLFlowRepository to prevent temporary file creation during model operations. This change enhances data handling by avoiding unnecessary file writes while maintaining logging functionality.
Enhance Gates and MLFlowRepository with new functionalities and improvements
- Added a new method `clean_tmp_files` in the Gates class to remove temporary files associated with model retraining.
- Updated MLFlowRepository methods to improve experiment handling, including dynamic parameter logging and model retrieval.
- Refactored model loading methods to streamline the process and enhance error handling.
- Improved logging for model operations and added support for model parameter retrieval.
- Adjusted minimal_retrain workflow to extend timeouts for activities and ensure proper model configuration handling.
Enhance MinIO integration and update environment configurations
- Added MinIO configuration parameters to .env.example and values.yaml for improved storage management.
- Updated requirements.txt to include necessary libraries for MinIO support.
- Refactored Activities class to utilize Storage for MinIO interactions.
- Enhanced MLFlow class to integrate MinIO for data retrieval during model retraining.
- Introduced build_minio_config function to streamline MinIO configuration setup.
- Updated minimal_retrain workflow to support data storage in MinIO.
Update .gitignore, values.yaml, and model_repository.py for improved data handling and logging
- Added 'catboost_info/' to .gitignore to prevent tracking of additional temporary files.
- Updated GITHUB_BRANCH in values.yaml to reflect the current branch for model retraining.
- Enhanced model_repository.py to drop duplicate timestamps in treated data and streamline attribute logging during model retraining.
Enhance MLFlow and MLFlowRepository with improved data handling and logging
- Refactored MLFlow class to sort data by 'created_at' and drop duplicates for better input preparation.
- Updated MLFlowRepository methods to include detailed logging for artifact downloads and model predictions.
- Introduced LzmaPayloadCodec for efficient payload compression in the worker, optimizing data handling for large payloads.
- Enhanced timestamp handling in treated data to ensure compatibility with model expectations.
Update model retraining and logging enhancements
- Changed the GITHUB_BRANCH value in values.yaml to 'main' for consistency.
- Refactored MLFlow class to improve timestamp handling and error messaging during model retraining.
- Enhanced MLFlowRepository methods to include metadata logging and improved model version retrieval.
- Updated minimal_retrain workflow to support extended timeout for activities and include model configuration in input data.
Update model retraining and reporting functionality
- Changed the GITHUB_BRANCH value in values.yaml to reflect the latest adjustments for retraining the courier.
- Enhanced the Gates class with a new method `format_retrain_report` to format retraining report data according to storage policies.
- Refactored the MLFlow class to improve error handling during model retraining and return structured output.
- Updated the model_repository to utilize the latest MLFlow API for retrieving model versions and improved logging.
- Modified the minimal_retrain workflow to conditionally update the production model based on retraining success.
SIENTIAPDE-1222
Enhance MLFlow and MLFlowRepository with model configuration support
- Introduced `model_config` parameter in MLFlow methods to streamline model handling and configuration management.
- Updated `retrain_model`, `transform`, and `predict` methods to accept `model_config` and `metadata` for improved flexibility and logging.
- Added `detect_and_parse_datetime_index` method to handle datetime index parsing with enhanced error handling and logging.
- Refactored model experiment creation to include transformation and prediction flavors, along with compression options.
- Improved documentation and type hints across methods for better clarity and usability.
Update .gitignore to include 'tmp/' directory and ensure '.env' is listed
- Added 'tmp/' to the .gitignore file to prevent temporary files from being tracked.
- Confirmed that '.env' is included to avoid committing sensitive environment variables.
Remove deprecated files and configurations from transformer_pyfunc module
- Deleted conda.yaml, MLmodel, python_env.yaml, requirements.txt, and various utility scripts related to data processing and model handling.
- Removed binary files including python_model.pkl and training_transformer.pkl to clean up the artifacts directory.
- This cleanup is part of the effort to streamline the transformer_pyfunc module and eliminate unused components.
Add placeholder class 'Any' in test_model_repository.py and update invalid_cases to use it
- Introduced a new placeholder class 'Any' to be used in test cases.
- Updated the 'invalid_cases' list to replace integer keys with instances of the 'Any' class, enhancing test coverage for key types.
Refactor model configuration handling in MLFlow and workflows
- Replaced 'model_retention' with 'model_config' to encapsulate retention settings and improve consistency across various components.
- Updated test cases to reflect changes in argument structure, ensuring compatibility with the new model configuration format.
- Added 'prediction_store_policy' to input data handling in workflows for enhanced configuration management.
Enhance MLFlow data handling by adding timestamp column and improving debug logging
- Added a 'timestamp' column to the input data, converting the index to a datetime format for better tracking of predictions.
- Improved debug logging to provide clearer context by including the input data preview in the log output.
