SIENTIAPDE-1273
Enhance security analysis and SQL injection handling
- Added skip for potential SQL injection false positives in Bandit configuration.
- Updated validate.sh to use the pyproject.toml configuration for Bandit security analysis.
- Refactored code to replace ensure_dataframe utility with direct DataFrame usage in multiple activities, improving clarity and reducing dependencies.
- Removed the deprecated dataframe_utils module to streamline the codebase.
Enhance MLFlowRepository and Activities classes with new methods and metrics
- Added `check_artifact_exists` method to MLFlowRepository for verifying artifact presence in the MLflow Model Registry.
- Implemented `get_prediction_data` method in MLFlowRepository to retrieve prediction data from models.
- Updated Activities class to integrate ModelMetrics for improved metrics handling.
- Enhanced tests for artifact existence checks and prediction data retrieval, ensuring robust coverage for new functionalities.
- Updated various workflows to include `transform_table_name` in input data for better data handling.
Update logging in MLFlowRepository and OpcRepository to use unified logging methods
- Refactored logging calls in MLFlowRepository to replace `self.logger.info` and `self.logger.debug` with `self.info` and `self.debug` for consistency.
- Updated connection logging in OpcRepository to format the connection message properly.
- Adjusted test cases to reflect changes in logging behavior and ensure proper assertions.
Enhance metrics handling in Gates and MLFlow classes
- Added checks for `None` response times before emitting OPC writing metrics in the Gates class to prevent unnecessary metric emissions.
- Updated the MLFlow class to conditionally sort and drop duplicates based on the presence of the 'created_at' column, ensuring robustness in data processing.
- Adjusted corresponding tests to validate the new behavior in both classes.
Refactor monitoring and metrics integration across various components
- Removed coverage options from `pyproject.toml`.
- Updated prediction metrics in `README.md` to replace `pipeline_name` with `workflow_name`.
- Upgraded `sientia-dataops-library` dependency version in `requirements-light.txt` and `requirements.txt`.
- Enhanced metrics handling in `laborious` activities, including `Activities`, `Gates`, `MLFlow`, and `OPC`, to utilize a new `MetricsController`.
- Refactored metric emission methods to improve clarity and consistency across the codebase.
- Updated tests to reflect changes in metrics handling and ensure proper functionality.
Refactor OPC Write Method and Comment Out Error Handling Logic
- Reformatted the `write_opc_data` method for improved readability by adjusting its signature.
- Commented out the error handling logic in `OpcRepository` related to disconnection due to multiple errors, preserving the original functionality for future reference.
- Updated tests to reflect the commented-out error handling, ensuring clarity in the testing process.
Enhance OPC Metrics Handling and Refactor Write Operations
- Updated the OPC class to return response times for write operations, improving metrics tracking.
- Refactored the Gates activity to incorporate OPC metrics into the metrics writing process.
- Adjusted the manage_output_tags method in OpcRepository to return response times for each tag written.
- Modified tests to validate the new metrics structure and ensure correct behavior of the updated methods.
Refactor error handling in MLFlowRepository and update tests
- Improved error message formatting in MLFlowRepository for better readability.
- Updated test assertions to ensure correct calls to MLflow methods during experiment retrieval and creation.
Refactor OPC handling by removing pod_id from initialization and updating logging format
- Removed pod_id parameter from OPC class and repository initialization to streamline connection management.
- Updated logging statements for improved readability during disconnection attempts and error handling.
Refactor tests and update model_repository.py for clarity and consistency
- Added a blank line in model_repository.py for improved readability.
- Adjusted formatting in test_mlflow.py to streamline assertions.
- Updated ensure_bucket_exists method tests in test_minio_repository.py to reflect the new return value of None instead of True.
Enhance validation script and refactor code in various modules
- Updated the validation script to include automatic code formatting and linting fixes using Ruff.
- Removed the `clean_tmp_files` method from the Gates class to streamline functionality.
- Simplified conditional checks in the OpcRepository for better clarity and error handling.
- Added model ID to the minimal retrain workflow for improved tracking.
- Introduced new test cases for error handling in MLFlow and storage operations, ensuring robustness in repository interactions.
Update .gitignore and refactor metrics.py for improved logging and consistency
- Added coverage.xml to .gitignore to prevent tracking of coverage reports.
- Refactored metric labels in metrics.py for consistency in string formatting and improved readability.
- Enhanced logging messages in various activities to ensure uniformity in message formatting.
Add placeholder class 'Any' in test_model_repository.py and update invalid_cases to use it
- Introduced a new placeholder class 'Any' to be used in test cases.
- Updated the 'invalid_cases' list to replace integer keys with instances of the 'Any' class, enhancing test coverage for key types.
Update test cases in model repository and prediction process
- Replaced string data with MagicMock in test_transform_success and test_transform_error to improve test isolation.
- Updated the predict method calls in test_predict_success and test_predict_error to reflect changes in argument structure.
- Added 'prediction_store_policy' to the test_run configuration in test_prediction_process for consistency with recent updates.
Refactor datetime index handling in MLFlow and MLFlowRepository
- Moved the detect_and_parse_datetime_index method from MLFlow to MLFlowRepository for better organization and reusability.
- Updated the method to include enhanced logging and error handling for invalid datetime formats.
- Adjusted the transform method in MLFlowRepository to utilize the new datetime index parsing logic.
- Added unit tests for both valid and invalid datetime index cases to ensure robustness.
Refactor test data structure in gates.py and model_repository.py for improved clarity and consistency
- Updated test cases in test_gates.py to use dictionaries for prediction and response_time, associating values with timestamps.
- Modified test_predict_success in test_model_repository.py to create a DataFrame with named indices for better readability in assertions.
Refactor OpcRepository and update tests for async handling
- Removed the shutdown method from OpcRepository and adjusted the disconnect logic.
- Updated tests in test_activities.py and test_opc.py to support async shutdown functionality.
- Enhanced test cases in test_opc_repository.py to ensure proper async behavior and error handling in OpcRepository methods.
Update tests in Gates and OpcRepository to enhance logging and include metadata
- Removed unnecessary debug assertion in the Gates activity test for format prediction.
- Updated the test for last timestamp with no data to include a metadata dictionary.
- Modified OpcRepository tests to utilize custom_error logging with metadata for improved error context.
Update test for validate_connection method in OpcRepository to include detailed response structure
- Modified the test case to assert the response format when the connection is lost, ensuring it includes notification ID, message, block, and level attributes for better clarity in error handling.
Refactor OPC data writing and repository initialization
- Simplified success tracking logic in the OPC class for writing prediction and confidence data.
- Removed unused pod_id attribute from OpcRepository initialization.
- Updated test cases to include pod_id for improved metrics tracking during data writing operations.
SIENTIAPDE-1174 Update tests to enhance MLFlow and workflow activity assertions
- Added logger initialization in MLFlowRepository for improved logging.
- Updated prediction test assertions to reflect changes in output structure.
- Increased activity method call count assertions in workflow tests for accuracy.
Refactor model_repository and enhance test coverage for MLFlow functionalities
- Updated model_repository to ensure the 'temp' directory is created if it doesn't exist using `exist_ok=True`.
- Added new tests for retraining and updating production models, including error handling scenarios.
- Improved existing tests for model management workflows to ensure robustness and reliability.
Update GITHUB_BRANCH in values.yaml and refactor MLFlowRepository for model retraining
- Changed GITHUB_BRANCH in values.yaml to reflect the new pipeline for retraining.
- Refactored MLFlowRepository by introducing create_model_experiment and perform_model_retrain methods to streamline model retraining processes.
- Updated test cases to cover new functionalities and ensure proper error handling in model management workflows.
SIENTIAPDE-1163 Enhance OPC tests to improve error handling and notifications. Added tests for MongoDB configuration and refined connection error handling in OPC repository.
Refactor notification handling in Gates and OPC activities to use send_notification method with metadata integration, enhancing error reporting and traceability.
Refactor tests for Postgres activities and improve error handling
- Updated test_postgres.py to enhance the testing of load_custom_query method, including cases for None data and date conversion.
- Refactored repeat_last_prediction tests to use mocks for SQLAlchemy session execution.
- Added tests for export_data_to_postgres method, covering both success and error scenarios.
- Improved the initialization tests for Activities class to ensure proper instantiation of dependencies.
- Enhanced test coverage for OPC repository connection validation.
- Updated tests for prediction workflows to streamline input handling and improve clarity.
- Introduced tests for connectors configuration to validate environment variable handling for MLFlow, OPC, and Postgres.
- Added tests for logger utility to ensure default settings are correctly applied.
Refactor and enhance the laborious workflow and utilities
- Removed outdated test file `test_predictions_batch.py` from workflows.
- Added `input_sample.json` for standardized input configuration.
- Introduced `connectors_config.py` to manage database and service configurations.
- Implemented a logging utility in `logger.py` for consistent logging across the application.
- Created `policies.py` to define retry policies for workflows.
- Developed comprehensive tests for `MLFlowRepository` in `test_model_repository.py`.
- Added extensive tests for `OpcRepository` in `test_opc_repository.py`.
- Updated `test_predictions_batch.py` to reflect new workflow structure and testing methodology.
Refactor activity methods and update requirements.txt to enhance functionality and remove deprecated filters. Added detailed docstrings for clarity and improved error handling in data processing workflows.
