Enhance validation script and refactor code in various modules
- Updated the validation script to include automatic code formatting and linting fixes using Ruff.
- Removed the `clean_tmp_files` method from the Gates class to streamline functionality.
- Simplified conditional checks in the OpcRepository for better clarity and error handling.
- Added model ID to the minimal retrain workflow for improved tracking.
- Introduced new test cases for error handling in MLFlow and storage operations, ensuring robustness in repository interactions.
Update .gitignore and refactor metrics.py for improved logging and consistency
- Added coverage.xml to .gitignore to prevent tracking of coverage reports.
- Refactored metric labels in metrics.py for consistency in string formatting and improved readability.
- Enhanced logging messages in various activities to ensure uniformity in message formatting.
Refactor test data structure in gates.py and model_repository.py for improved clarity and consistency
- Updated test cases in test_gates.py to use dictionaries for prediction and response_time, associating values with timestamps.
- Modified test_predict_success in test_model_repository.py to create a DataFrame with named indices for better readability in assertions.
Implement prediction store policy handling in Gates activity
- Added a new method `get_prediction_store_policy` to validate and parse the prediction store policy.
- Updated `format_prediction` method to utilize the new policy handling, allowing for sorting of predictions based on the specified policy.
- Enhanced test coverage for the new policy handling, including various scenarios for valid and invalid policies.
- Removed the obsolete `coverage.sh` script.
Update tests in Gates and OpcRepository to enhance logging and include metadata
- Removed unnecessary debug assertion in the Gates activity test for format prediction.
- Updated the test for last timestamp with no data to include a metadata dictionary.
- Modified OpcRepository tests to utilize custom_error logging with metadata for improved error context.
Add async test for write_metrics method in gates_activity
- Implemented a new test to validate the write_metrics functionality, ensuring metrics are correctly recorded for predictions, confidence, and response time.
- Utilized mocking to verify interactions with the metrics tracking system.
Refactor notification handling in Gates and OPC activities to use send_notification method with metadata integration, enhancing error reporting and traceability.
Refactor tests for Postgres activities and improve error handling
- Updated test_postgres.py to enhance the testing of load_custom_query method, including cases for None data and date conversion.
- Refactored repeat_last_prediction tests to use mocks for SQLAlchemy session execution.
- Added tests for export_data_to_postgres method, covering both success and error scenarios.
- Improved the initialization tests for Activities class to ensure proper instantiation of dependencies.
- Enhanced test coverage for OPC repository connection validation.
- Updated tests for prediction workflows to streamline input handling and improve clarity.
- Introduced tests for connectors configuration to validate environment variable handling for MLFlow, OPC, and Postgres.
- Added tests for logger utility to ensure default settings are correctly applied.
Refactor and enhance the laborious workflow and utilities
- Removed outdated test file `test_predictions_batch.py` from workflows.
- Added `input_sample.json` for standardized input configuration.
- Introduced `connectors_config.py` to manage database and service configurations.
- Implemented a logging utility in `logger.py` for consistent logging across the application.
- Created `policies.py` to define retry policies for workflows.
- Developed comprehensive tests for `MLFlowRepository` in `test_model_repository.py`.
- Added extensive tests for `OpcRepository` in `test_opc_repository.py`.
- Updated `test_predictions_batch.py` to reflect new workflow structure and testing methodology.
Implement new get_last_timestamp method in Gates class, refactor MLFlow activity methods to return only transformed data, and update PredictionsBatch and PredictionProcess workflows to utilize Activities module. Add detailed docstrings for new methods and enhance test coverage for get_last_timestamp functionality.
Refactor activity methods and update requirements.txt to enhance functionality and remove deprecated filters. Added detailed docstrings for clarity and improved error handling in data processing workflows.
