Refactor MinioRepository and MLFlowRepository for improved functionality and error handling
- Updated `ensure_bucket_exists` method in MinioRepository to return None instead of a boolean, streamlining bucket existence checks.
- Replaced hardcoded error messages in MLFlowRepository with a constant for better maintainability.
- Adjusted column assignment in MLFlowRepository to use pd.Index for improved clarity.
Enhance validation script and refactor code in various modules
- Updated the validation script to include automatic code formatting and linting fixes using Ruff.
- Removed the `clean_tmp_files` method from the Gates class to streamline functionality.
- Simplified conditional checks in the OpcRepository for better clarity and error handling.
- Added model ID to the minimal retrain workflow for improved tracking.
- Introduced new test cases for error handling in MLFlow and storage operations, ensuring robustness in repository interactions.
Update .gitignore and refactor metrics.py for improved logging and consistency
- Added coverage.xml to .gitignore to prevent tracking of coverage reports.
- Refactored metric labels in metrics.py for consistency in string formatting and improved readability.
- Enhanced logging messages in various activities to ensure uniformity in message formatting.
Refactor `force_memory_release` function in model_repository.py to improve logging
- Updated the `force_memory_release` function to remove metadata parameter and enhance logging by using `logger.info()` instead of `logger.custom_info()`.
- Adjusted calls to `force_memory_release` in the MLFlowRepository to pass the logger instance, ensuring consistent logging during memory management operations.
Refactor memory management in MLFlowRepository to use `force_memory_release` function
- Replaced direct calls to `gc.collect()` with `force_memory_release()` for improved memory optimization after model deletion.
- This change enhances memory management during model operations, ensuring more efficient resource handling.
Add memory management function to model_repository.py
- Introduced `force_memory_release` function to enhance memory management by triggering garbage collection and attempting to release unused memory.
- Utilized `ctypes` to call `malloc_trim` for further memory optimization, improving overall performance during model operations.
Enhance MLFlowRepository memory management by adding garbage collection and logging for model deletion
- Introduced garbage collection after model deletion to optimize memory usage.
- Added logging to inform when a model is deleted from memory, improving traceability during predictions.
Update values.yaml and model_repository.py for configuration and error handling improvements
- Increased replicaCount from 1 to 2 in values.yaml for enhanced scalability.
- Updated image tag from "0.0.2" to "0.0.3" in values.yaml to reflect the latest version.
- Enhanced error messaging in model_repository.py to include the actual index type when raising ValueError for index type validation.
Comment out CSV export lines in MLFlowRepository to prevent temporary file creation during model operations. This change enhances data handling by avoiding unnecessary file writes while maintaining logging functionality.
Enhance Gates and MLFlowRepository with new functionalities and improvements
- Added a new method `clean_tmp_files` in the Gates class to remove temporary files associated with model retraining.
- Updated MLFlowRepository methods to improve experiment handling, including dynamic parameter logging and model retrieval.
- Refactored model loading methods to streamline the process and enhance error handling.
- Improved logging for model operations and added support for model parameter retrieval.
- Adjusted minimal_retrain workflow to extend timeouts for activities and ensure proper model configuration handling.
Enhance MinIO integration and update environment configurations
- Added MinIO configuration parameters to .env.example and values.yaml for improved storage management.
- Updated requirements.txt to include necessary libraries for MinIO support.
- Refactored Activities class to utilize Storage for MinIO interactions.
- Enhanced MLFlow class to integrate MinIO for data retrieval during model retraining.
- Introduced build_minio_config function to streamline MinIO configuration setup.
- Updated minimal_retrain workflow to support data storage in MinIO.
Update .gitignore, values.yaml, and model_repository.py for improved data handling and logging
- Added 'catboost_info/' to .gitignore to prevent tracking of additional temporary files.
- Updated GITHUB_BRANCH in values.yaml to reflect the current branch for model retraining.
- Enhanced model_repository.py to drop duplicate timestamps in treated data and streamline attribute logging during model retraining.
Enhance MLFlow and MLFlowRepository with improved data handling and logging
- Refactored MLFlow class to sort data by 'created_at' and drop duplicates for better input preparation.
- Updated MLFlowRepository methods to include detailed logging for artifact downloads and model predictions.
- Introduced LzmaPayloadCodec for efficient payload compression in the worker, optimizing data handling for large payloads.
- Enhanced timestamp handling in treated data to ensure compatibility with model expectations.
Update model retraining and logging enhancements
- Changed the GITHUB_BRANCH value in values.yaml to 'main' for consistency.
- Refactored MLFlow class to improve timestamp handling and error messaging during model retraining.
- Enhanced MLFlowRepository methods to include metadata logging and improved model version retrieval.
- Updated minimal_retrain workflow to support extended timeout for activities and include model configuration in input data.
Update model retraining and reporting functionality
- Changed the GITHUB_BRANCH value in values.yaml to reflect the latest adjustments for retraining the courier.
- Enhanced the Gates class with a new method `format_retrain_report` to format retraining report data according to storage policies.
- Refactored the MLFlow class to improve error handling during model retraining and return structured output.
- Updated the model_repository to utilize the latest MLFlow API for retrieving model versions and improved logging.
- Modified the minimal_retrain workflow to conditionally update the production model based on retraining success.
SIENTIAPDE-1222
Enhance MLFlow and MLFlowRepository with model configuration support
- Introduced `model_config` parameter in MLFlow methods to streamline model handling and configuration management.
- Updated `retrain_model`, `transform`, and `predict` methods to accept `model_config` and `metadata` for improved flexibility and logging.
- Added `detect_and_parse_datetime_index` method to handle datetime index parsing with enhanced error handling and logging.
- Refactored model experiment creation to include transformation and prediction flavors, along with compression options.
- Improved documentation and type hints across methods for better clarity and usability.
Enhance MLFlow data handling by adding timestamp column and improving debug logging
- Added a 'timestamp' column to the input data, converting the index to a datetime format for better tracking of predictions.
- Improved debug logging to provide clearer context by including the input data preview in the log output.
Update MLFlow class to pass logger instance directly to MLFlowRepository
- Modified the initialization of MLFlowRepository in the MLFlow class to pass the logger instance directly, improving logging capabilities and consistency across the application.
SIENTIAPDE-1222 Update MLFlow methods to include metadata parameter
- Modified the transform and predict method calls in the MLFlow class to include a new 'metadata' parameter, enhancing the functionality and data handling capabilities of the model monitoring repository.
Refactor datetime index handling in MLFlow and MLFlowRepository
- Moved the detect_and_parse_datetime_index method from MLFlow to MLFlowRepository for better organization and reusability.
- Updated the method to include enhanced logging and error handling for invalid datetime formats.
- Adjusted the transform method in MLFlowRepository to utilize the new datetime index parsing logic.
- Added unit tests for both valid and invalid datetime index cases to ensure robustness.
Update model configuration keys in MLFlowRepository for consistency
- Changed 'model_retention' to 'retention_minutes' and 'is_compressed' to 'compressed' in model configuration handling, ensuring alignment with updated configuration standards.
Update prediction_store_policy handling in workflows
- Added 'prediction_store_policy' to the input data handling in PredictionsBatch, ensuring a default value of 'lts:1' is used when not provided.
- Modified FormatAndExportPrediction to directly use 'prediction_store_policy' from input_data, removing the default fallback.
- Updated PredictionProcess to include 'prediction_store_policy' in the output data structure, ensuring consistency across workflows.
Enhance MLFlow debug logging to limit output size
- Updated debug logging statements in the MLFlow activity to include a maximum of 5 items and a depth of 5 for the sample dictionary, improving readability and reducing log clutter for raw and transformed response data.
Refactor debug logging in Gates activity for improved readability
- Reformatted the debug logging statement for input data in the Gates activity to enhance clarity and maintain consistency with previous logging improvements.
Refactor MLFlow debug logging for improved readability
- Reformatted debug logging statements in the MLFlow activity to enhance clarity and consistency.
- Ensured that the output of raw and transformed response data is presented in a more readable format, maintaining the use of create_sample_dict for better visualization.
Update image tag in values.yaml and enhance debug logging in Gates and MLFlow activities
- Updated the image tag in values.yaml from '0.0.1' to '0.0.2'.
- Improved debug logging in the Gates activity to format input data and filters for better readability.
- Enhanced MLFlow activity logging to include formatted output for raw and transformed response data, ensuring consistent logging format.
Update requirements and enhance logging in Gates and MLFlow activities
- Updated the sientia-dataops-library and sientia-mlops-library dependencies in requirements.txt to the latest versions.
- Improved debug logging in the Gates activity to display a sample of input data and filters, enhancing clarity and reducing output size.
- Refactored MLFlow activity logging to utilize the create_sample_dict function for better visualization of nested data structures in logs.
SIENTIAPDE-1222
Enhance MLFlow logging with sample dictionary for response data
- Introduced a new method to create a sample dictionary for debugging, allowing for better visualization of nested data structures in logs.
- Updated debug logging to utilize the new sampling method for raw and transformed response data, improving clarity and reducing output size.
- Adjusted logging for processed input data to display only the first few rows, enhancing readability.
Refactor MLFlow logging to enhance data output clarity
- Updated debug logging to use data.to_string() for processed input data, improving readability.
- Modified raw response data logging to format the output as a string, ensuring consistent logging format.
Update values.yaml and MLFlow logging for courier integration
- Changed the image repository to 'sientia-module-courier' and updated the image tag to '0.0.1'.
- Modified environment variables for GITHUB_BRANCH and MLFLOW_PASSWORD to reflect new configurations.
- Enhanced MLFlow logging to include additional debug statements for raw response data and added a check for empty DataFrames.
Refactor MLFlow logging to improve data output clarity
- Updated the debug logging to directly capture the output of data.to_csv, enhancing traceability of processed input data.
- Removed redundant debug statements for raw response data to streamline logging and focus on essential information.
SIENTIAPDE-1214: Enhance MLFlow and tests with datetime index handling and logging improvements
- Added a new method in MLFlow to detect and parse datetime indices in DataFrames, ensuring proper format and raising errors for invalid types.
- Updated prediction workflows to utilize the new datetime index handling, improving data integrity during transformations.
- Enhanced logging in model_repository to include detailed data outputs for better traceability.
- Adjusted timeout settings in prediction workflows for improved execution time management.
- Updated tests.ipynb to include additional checks for index types and outputs for better validation of functionality.
SIENTIAPDE-1214: Refactor MLFlow and model repository methods to use model_config dictionary
- Updated MLFlow class methods to accept model_config instead of model_retention for improved flexibility.
- Modified model_repository methods to handle model_config, extracting necessary parameters for transformation and prediction.
- Adjusted predictions_batch and prediction_process workflows to utilize model_config for better configuration management.
- Commented out the previous sientia-mlops-library dependency in requirements.txt for clarity.
Refactor OPC and model repository for improved functionality and clarity
- Updated OPC server logging to handle missing prediction and confidence tags gracefully.
- Corrected documentation for OPC reconnection interval from milliseconds to seconds.
- Enhanced MLFlowRepository with new methods for model retrieval, caching, and transformation, improving model management and retraining workflows.
Update tests.ipynb and gates.py for enhanced functionality and clarity
- Adjusted execution count in tests.ipynb for reproducibility.
- Modified DataFrame creation in tests.ipynb to dynamically set size and update timestamp frequency.
- Added a new filter function for empty data in gates.py to improve data handling.
Enhance tests.ipynb and gates.py for improved logging and output clarity
- Set execution count to null in tests.ipynb for reproducibility.
- Added stdout output to tests.ipynb for better visibility of results.
- Updated DataFrame display logic in tests.ipynb to include additional print statements for length and size.
- Improved debug logging in gates.py to log prediction data as a string for better traceability.
- Changed info log in gates.py to use len() instead of size for consistency.
Update tests.ipynb and values.yaml for improved functionality and versioning
- Updated execution count in tests.ipynb for reproducibility.
- Modified DataFrame creation in tests.ipynb to include a timestamp column and save to CSV.
- Changed image tag in values.yaml from "0.4.4" to "0.4.5" for versioning.
- Updated GITHUB_BRANCH in values.yaml to reflect the latest branch adjustments.
Refactor connectors_config.py and conditional_filters.py for improved configuration management and data filtering
- Updated PostgreSQL and MLFlow configuration functions to enhance default values and environment variable handling.
- Simplified OPC server configuration logic and improved MongoDB connection string construction.
- Refactored conditional filters to streamline null value checks and empty data validation, removing unnecessary comments and examples for clarity.
- Removed extensive module docstrings to enhance code readability.
Update README.md to enhance installation instructions and refactor MLFlow filters
- Added steps for installing GitHub CLI and authenticating with GitHub.
- Updated the `api_error_filter` and `nan_values_filter` functions to improve parameter handling and streamline logic.
Remove all module docstrings and the versioning information from the Laborious package, activities, utils, and workflows. This cleanup enhances code readability and reduces unnecessary comments in the codebase.
Remove Docker configuration files and refactor project structure
- Deleted docker-compose.yml and Dockerfile as part of the project restructuring.
- Updated README.md to reflect changes in project setup and configuration.
- Introduced a new __init__.py file in the laborious package to provide an overview of the system.
- Enhanced documentation across various modules, including metrics, activities, and workflows, to improve clarity and usability.
- Added comprehensive docstrings and comments to key classes and methods for better maintainability.
Implement prediction store policy handling in Gates activity
- Added a new method `get_prediction_store_policy` to validate and parse the prediction store policy.
- Updated `format_prediction` method to utilize the new policy handling, allowing for sorting of predictions based on the specified policy.
- Enhanced test coverage for the new policy handling, including various scenarios for valid and invalid policies.
- Removed the obsolete `coverage.sh` script.
Refactor OpcRepository and update tests for async handling
- Removed the shutdown method from OpcRepository and adjusted the disconnect logic.
- Updated tests in test_activities.py and test_opc.py to support async shutdown functionality.
- Enhanced test cases in test_opc_repository.py to ensure proper async behavior and error handling in OpcRepository methods.
Revert image tag in values.yaml to 0.4.4 and rename shutdown method in OpcRepository class
- Changed image tag in values.yaml from 0.4.5 back to 0.4.4.
- Renamed __del__ method to shutdown in OpcRepository for clarity.
- Enhanced connection validation logic in OpcRepository to improve error handling and logging.
Update values.yaml and enhance logging in OPC activity
- Change replicaCount from 5 to 1 for reduced resource allocation.
- Update image tag in values.yaml to 0.4.5 for the latest version.
- Modify GITHUB_BRANCH environment variable for improved async handling.
- Add logging statements in OPC class to track server initialization and connection status.
- Expanded the .env file with configurations for MongoDB, Postgres, MlFlow, and Temporal.
- Refactored OPC class methods to be asynchronous, including init_opc, write_data, manage_output_tags, and shutdown.
- Updated the worker to initialize OPC asynchronously and adjusted shutdown handling for activities.
Update requirements and modify MLFlow activity
- Bump sientia-mlops-library version in requirements.txt from 0.38.5 to 0.38.8.
- Comment out the reset_index call in the MLFlow activity to prevent unintended data manipulation.
Update dependencies and enhance datetime handling
- Bump sientia-dataops-library version in requirements.txt to 1.4.4.
- Update image tag in values.yaml to 0.4.4.
- Add 'datetime_columns' to metadata in MinimalRetrain and PredictionsBatch workflows for improved data handling.
