Enhance MLFlowRepository and Activities classes with new methods and metrics
- Added `check_artifact_exists` method to MLFlowRepository for verifying artifact presence in the MLflow Model Registry.
- Implemented `get_prediction_data` method in MLFlowRepository to retrieve prediction data from models.
- Updated Activities class to integrate ModelMetrics for improved metrics handling.
- Enhanced tests for artifact existence checks and prediction data retrieval, ensuring robust coverage for new functionalities.
- Updated various workflows to include `transform_table_name` in input data for better data handling.
Update requirements and enhance metrics and data handling
- Updated the sientia-dataops-library dependency version in requirements.txt to 1.5.3.
- Added new metrics for model analysis, including lag, count, and error count in metrics.py.
- Implemented a new method for formatting transformed data in gates.py.
- Enhanced MLFlowRepository with methods to load artifact dataframes and calculate model metrics, including drift and performance metrics.
- Updated the prediction process to handle transformed data and ensure proper execution of related activities in format_and_export_prediction.py and prediction_process.py.
Refactor OPC data handling in opc_repository.py to improve code clarity. Updated commented-out code for SourceTimestamp to include NOSONAR annotations, ensuring better code quality checks.
Update logging in MLFlowRepository and OpcRepository to use unified logging methods
- Refactored logging calls in MLFlowRepository to replace `self.logger.info` and `self.logger.debug` with `self.info` and `self.debug` for consistency.
- Updated connection logging in OpcRepository to format the connection message properly.
- Adjusted test cases to reflect changes in logging behavior and ensure proper assertions.
Enhance OPC connection metrics by emitting total connections metric upon connection attempt. This change ensures accurate tracking of connection attempts in the OpcRepository class.
Refactor OPC class to remove unused opc_servers attribute and streamline initialization. Update OpcRepository to consolidate server name tags for metric emissions.
SIENTIAPDE-1325 Add OPC server name configuration and update repository handling
- Introduced a new environment variable `OPC_SERVER_NAME` in values.yaml with a default value.
- Updated the OPC class to include `server_name` when initializing OpcRepository.
- Enhanced the configuration builder to retrieve `OPC_SERVER_NAME` from the environment.
- Adjusted OpcRepository to store and utilize the `server_name` for metric emissions.
Refactor OPC metrics in metrics.py and opc_repository.py
- Replaced existing OPC connection metrics with new metrics for total connections and failed connections.
- Updated the OPC connection status metric to include server URL and adjusted corresponding metric emissions in opc_repository.py.
Update image tag and enhance OPC metrics tracking
- Bumped image tag from "1.0.1" to "1.1.0" in values.yaml.
- Added new metrics for OPC connection count and error count in metrics.py.
- Refactored OPC connection handling in opc_repository.py to emit new metrics for connection status, count, and errors.
Enhance metrics handling in Gates and MLFlow classes
- Added checks for `None` response times before emitting OPC writing metrics in the Gates class to prevent unnecessary metric emissions.
- Updated the MLFlow class to conditionally sort and drop duplicates based on the presence of the 'created_at' column, ensuring robustness in data processing.
- Adjusted corresponding tests to validate the new behavior in both classes.
Refactor monitoring and metrics integration across various components
- Removed coverage options from `pyproject.toml`.
- Updated prediction metrics in `README.md` to replace `pipeline_name` with `workflow_name`.
- Upgraded `sientia-dataops-library` dependency version in `requirements-light.txt` and `requirements.txt`.
- Enhanced metrics handling in `laborious` activities, including `Activities`, `Gates`, `MLFlow`, and `OPC`, to utilize a new `MetricsController`.
- Refactored metric emission methods to improve clarity and consistency across the codebase.
- Updated tests to reflect changes in metrics handling and ensure proper functionality.
Enhance thread safety in MLFlowRepository model caching
- Introduced a reentrant lock to synchronize access to the model cache, ensuring thread safety during cache checks and updates.
- Updated the cache management logic to acquire the lock when checking for existing models and when updating the cache after downloading a new model.
- Reduced the maximum cached workflows in the worker configuration for improved resource management.
Refactor model cache deletion in MLFlowRepository
- Updated the model cache deletion logic to remove the entire 'target' key instead of just the 'model' subkey, streamlining the cache management process.
Refactor workflow names to include 'subworkflow' prefix
- Updated the workflow definitions and calls in the predictions_batch, format_and_export_prediction, and prediction_process files to use the 'subworkflow' prefix for consistency.
- Adjusted corresponding test cases to reflect these changes in workflow execution calls.
Refactor OPC Test for Output Management
- Added spacing for improved readability in the test_manage_output_tags_failed and test_manage_output_tags_do_nothing functions.
- Reformatted the parameters in the call to manage_output_tags for better clarity and organization.
Enhance OPC Testing for Output Management
- Refactored the test_manage_output_tags_failed function to use parameterization for improved test coverage of different side effects.
- Added a new test_manage_output_tags_do_nothing to verify behavior when an invalid key is provided in the configuration, ensuring that no write operations are performed in this case.
Refactor OPC Class and Enhance Testing for Output Management
- Removed the unused 'success' parameter from the manage_output_tags method in the OPC class to streamline its signature.
- Commented out the error handling logic in the OpcRepository for clarity and future reference.
- Added new tests for manage_output_tags to validate successful and failed write operations, ensuring accurate assertions for output data and metrics.
Refactor OPC Write Method and Comment Out Error Handling Logic
- Reformatted the `write_opc_data` method for improved readability by adjusting its signature.
- Commented out the error handling logic in `OpcRepository` related to disconnection due to multiple errors, preserving the original functionality for future reference.
- Updated tests to reflect the commented-out error handling, ensuring clarity in the testing process.
Refactor OPC Write Test to Separate Output Data and Metrics
- Updated the test for the OPC write operation to separately assert output data and OPC metrics for improved clarity and accuracy in testing.
- Changed the variable names to reflect the new structure of the returned values from the write operation.
SIENTIAPDE-1314 Refactor OPC Write Operation in FormatAndExportPrediction Workflow
- Changed the variable name for the result of the OPC write operation for clarity.
- Extracted prediction and metrics from the OPC write result to improve data handling.
Update test_format_and_export_prediction.py to use prediction_data for workflow execution
- Replaced the mock return value with prediction_data in the test_run_none_path_flag function to ensure accurate testing of the workflow's data handling.
Enhance OPC Metrics Handling and Refactor Write Operations
- Updated the OPC class to return response times for write operations, improving metrics tracking.
- Refactored the Gates activity to incorporate OPC metrics into the metrics writing process.
- Adjusted the manage_output_tags method in OpcRepository to return response times for each tag written.
- Modified tests to validate the new metrics structure and ensure correct behavior of the updated methods.
SIENTIAPDE-1327: Add requirements file to quality gate workflow
- Included 'requirements-light.txt' in the quality gate workflow to specify additional dependencies for the CI process.
Update requirements and simplify CI workflow for quality checks
- Added new dependencies: kaleido, hyperopt, and shap to requirements.txt.
- Refactored the quality gate workflow to use a template for improved maintainability and reduced complexity.
Refactor error handling in MLFlowRepository and update tests
- Improved error message formatting in MLFlowRepository for better readability.
- Updated test assertions to ensure correct calls to MLflow methods during experiment retrieval and creation.
Update tests notebook execution count and bump image tag in values.yaml
- Incremented the execution count in the tests notebook for accurate tracking.
- Updated the image tag in values.yaml from "0.0.3" to "1.0.1" for versioning consistency.
- Enhanced error handling in model_repository.py to ensure proper retrieval of created experiments.
Refactor OPC handling by removing pod_id from initialization and updating logging format
- Removed pod_id parameter from OPC class and repository initialization to streamline connection management.
- Updated logging statements for improved readability during disconnection attempts and error handling.
Enhance OpcRepository client initialization and error handling
- Updated the Client instantiation in OpcRepository to include a timeout and watchdog interval for improved connection management.
- Added a disconnection call in the exception handling block to ensure proper resource cleanup during connection failures.
Update dependencies and enhance OpcRepository functionality
- Updated sientia-dataops-library version to 1.4.7 in requirements files.
- Modified GITHUB_BRANCH in values.yaml for improved pipeline management.
- Refactored OpcRepository class to inherit from BaseActivity, adding enhanced logging and error handling during disconnection.
- Implemented a disconnection fallback mechanism to ensure graceful handling of OPC server disconnections.
Enhance CI workflow for comprehensive code quality checks
- Added code formatting check, linting, type checking, and security analysis steps using Ruff, mypy, and Bandit.
- Integrated pytest for running tests, ensuring a robust quality gate in the CI pipeline.
Enhance validation script and CI workflow for improved code quality checks
- Added support for a --fix option in validate.sh to apply Ruff auto-fixes for formatting and linting.
- Updated GitHub Actions workflow to install development and runtime dependencies separately.
- Integrated the validation script execution into the CI pipeline to ensure code quality checks are performed automatically.
Enhance README with code quality and validation guidelines
- Added a new section on code quality and validation tools, detailing the use of Ruff, mypy, Bandit, and pytest.
- Included installation instructions for development dependencies and options for running validation scripts.
- Provided best practices for maintaining code quality and integrating CI/CD workflows.
Enhance README and repository utilities for clarity and functionality
- Updated README.md to improve descriptions and structure, adding detailed sections for features, workflows, and architecture.
- Enhanced MinioRepository with comprehensive docstrings for methods and class attributes, improving usability and documentation.
- Refined MLFlowRepository with clearer method descriptions and improved logging for better observability and maintainability.
Refactor tests and update model_repository.py for clarity and consistency
- Added a blank line in model_repository.py for improved readability.
- Adjusted formatting in test_mlflow.py to streamline assertions.
- Updated ensure_bucket_exists method tests in test_minio_repository.py to reflect the new return value of None instead of True.
