Refactor import statements in worker.py for improved organization
- Moved the import of the os module to the appropriate section, enhancing clarity and consistency in the import structure.
Refactor imports in opc.py to improve code clarity
- Moved the import of Hashable from collections.abc to the appropriate section, enhancing readability and organization of imports.
Update sientia-mlops-library dependency to version 0.40.6 and refactor get_prediction_data method calls for improved readability in model_repository.py and test_model_repository.py.
Enhance prediction data retrieval in MLFlowRepository by adding predict_flavor parameter
- Updated get_prediction_data method to accept a predict_flavor argument, allowing for different prediction model handling.
- Adjusted calls to get_prediction_data throughout the codebase to include the new parameter.
- Added new test cases to validate behavior for different predict_flavor values, ensuring robust functionality.
Enhance resource management and configuration in Laborious worker
- Updated `values.yaml` to define resource limits and requests for better performance tuning.
- Modified environment variables in `worker.py` to support resource-based scaling and improved task queue management.
- Introduced new functions for creating resource tuners and poller behaviors, enhancing scalability and efficiency in handling workloads.
Update sientia-mlops-library dependency to version 0.40.5 and fix SQL interval formatting in Drift and SimpleMetrics workflows for improved query accuracy.
Refactor data handling in Laborious workflows to enhance clarity and compatibility
- Improved data validation and formatting in `gates.py`.
- Streamlined model transformation management in `mlflow.py`.
- Enhanced export functionality in `format_and_export_prediction.py` for better data handling flexibility.
- Expanded test coverage for new features and improved documentation in `README.md`.
Enhance data handling and export processes in Laborious workflows
- Updated `gates.py` to improve data quality validation, filtering, and formatting operations, including enhanced metrics recording.
- Refined `mlflow.py` to better manage model transformations and reference data retrieval from MLflow Model Registry.
- Enhanced `format_and_export_prediction.py` to support separate export of transformed data, improving flexibility in data handling.
- Added comprehensive test coverage for new functionalities, including transformed data formatting and retrain report generation.
- Improved documentation in `README.md` to reflect changes in activities and workflows, ensuring clarity on data processing and export paths.
SIENTIAPDE-1273
Enhance security analysis and SQL injection handling
- Added skip for potential SQL injection false positives in Bandit configuration.
- Updated validate.sh to use the pyproject.toml configuration for Bandit security analysis.
- Refactored code to replace ensure_dataframe utility with direct DataFrame usage in multiple activities, improving clarity and reducing dependencies.
- Removed the deprecated dataframe_utils module to streamline the codebase.
Refactor data size calculation in ModelMetrics class to use shape method for improved accuracy. Changed from using len(target_data) to data.shape[0] for better clarity in data handling.
Update sientia-dataops-library dependency version to 1.6.1 and refactor return types in Gates class methods for improved type clarity. Changed return types from dict[str, Any] to dict for better compatibility with downstream systems.
Refactor data export in Drift, SimpleMetrics, and FormatAndExportPrediction workflows to improve data handling. Changed the export format from a list of dictionaries to direct objects for enhanced clarity and compatibility with downstream systems.
Refactor data export in MinimalRetrain workflow to improve structure. Changed the export format from a list of dictionaries to a direct report object for enhanced clarity and compatibility with downstream systems.
Refactor return type of get_reference_data method in MLFlow class to improve type specificity. Changed from a list of dictionaries with Hashable keys to a generic list of dictionaries for enhanced clarity in data handling.
Refactor return types in ModelMetrics class methods to enhance type clarity. Updated calculate_drift and calculate_simple_metrics methods to return lists of dictionaries instead of generic lists, improving type specificity for better data handling.
Refactor return type of calculate_drift method in ModelMetrics class to improve compatibility. Changed from a list of dictionaries to a generic list for enhanced flexibility in data handling.
Update dependencies and refactor data handling in various modules
- Updated sientia-dataops-library dependency version from 1.5.3 to 1.5.4 in requirements files.
- Updated sientia-mlops-library dependency version from 0.39.0 to 0.40.2 in requirements files.
- Refactored return types in Gates, MLFlow, and ModelMetrics classes to return dictionaries instead of DataFrames for improved compatibility with downstream systems.
- Removed the temporal_codec module as it is no longer needed for DataFrame serialization.
- Adjusted data handling in the Drift workflow to ensure proper data structure is maintained.
Refactor data handling in various modules to ensure DataFrame consistency
- Replaced direct DataFrame instantiation with `ensure_dataframe` utility in Gates, MLFlow, OPC, and ModelMetrics classes to standardize data handling.
- Updated return types in several asynchronous methods to return DataFrames instead of dictionaries for improved usability.
- Adjusted data export processes in workflows to convert DataFrames to dictionaries with `to_dict(orient='records')` for compatibility with downstream systems.
Refactor ModelMetrics return format and correct import name in worker module
- Changed the return format of the metrics data in ModelMetrics from a dictionary to a list for improved usability.
- Corrected the import statement for SimpleMetrics in the worker module to ensure consistency and clarity.
Update version and enhance metrics calculation in Laborious system
- Updated image tag in values.yaml from 1.1.0 to 1.1.1.
- Modified GITHUB_BRANCH environment variable for consistency.
- Added a new method `calculate_simple_metrics` in model_metrics.py to compute various model performance metrics including RMSE, MSE, MAE, and R2.
- Integrated the new metrics calculation into the worker setup, allowing for concurrent processing of simple metrics.
- Updated tests to cover the new metrics calculation functionality, ensuring comprehensive validation of the implementation.
Refactor Drift class to improve target name handling
- Extracted target name from model configuration in the Drift class for better clarity and maintainability.
- Updated test cases to reflect changes in input data structure, ensuring consistency in target name retrieval.
Enhance MLFlowRepository and Activities classes with new methods and metrics
- Added `check_artifact_exists` method to MLFlowRepository for verifying artifact presence in the MLflow Model Registry.
- Implemented `get_prediction_data` method in MLFlowRepository to retrieve prediction data from models.
- Updated Activities class to integrate ModelMetrics for improved metrics handling.
- Enhanced tests for artifact existence checks and prediction data retrieval, ensuring robust coverage for new functionalities.
- Updated various workflows to include `transform_table_name` in input data for better data handling.
Update requirements and enhance metrics and data handling
- Updated the sientia-dataops-library dependency version in requirements.txt to 1.5.3.
- Added new metrics for model analysis, including lag, count, and error count in metrics.py.
- Implemented a new method for formatting transformed data in gates.py.
- Enhanced MLFlowRepository with methods to load artifact dataframes and calculate model metrics, including drift and performance metrics.
- Updated the prediction process to handle transformed data and ensure proper execution of related activities in format_and_export_prediction.py and prediction_process.py.
Refactor OPC data handling in opc_repository.py to improve code clarity. Updated commented-out code for SourceTimestamp to include NOSONAR annotations, ensuring better code quality checks.
Update logging in MLFlowRepository and OpcRepository to use unified logging methods
- Refactored logging calls in MLFlowRepository to replace `self.logger.info` and `self.logger.debug` with `self.info` and `self.debug` for consistency.
- Updated connection logging in OpcRepository to format the connection message properly.
- Adjusted test cases to reflect changes in logging behavior and ensure proper assertions.
Enhance OPC connection metrics by emitting total connections metric upon connection attempt. This change ensures accurate tracking of connection attempts in the OpcRepository class.
Refactor OPC class to remove unused opc_servers attribute and streamline initialization. Update OpcRepository to consolidate server name tags for metric emissions.
SIENTIAPDE-1325 Add OPC server name configuration and update repository handling
- Introduced a new environment variable `OPC_SERVER_NAME` in values.yaml with a default value.
- Updated the OPC class to include `server_name` when initializing OpcRepository.
- Enhanced the configuration builder to retrieve `OPC_SERVER_NAME` from the environment.
- Adjusted OpcRepository to store and utilize the `server_name` for metric emissions.
Refactor OPC metrics in metrics.py and opc_repository.py
- Replaced existing OPC connection metrics with new metrics for total connections and failed connections.
- Updated the OPC connection status metric to include server URL and adjusted corresponding metric emissions in opc_repository.py.
Update image tag and enhance OPC metrics tracking
- Bumped image tag from "1.0.1" to "1.1.0" in values.yaml.
- Added new metrics for OPC connection count and error count in metrics.py.
- Refactored OPC connection handling in opc_repository.py to emit new metrics for connection status, count, and errors.
Enhance metrics handling in Gates and MLFlow classes
- Added checks for `None` response times before emitting OPC writing metrics in the Gates class to prevent unnecessary metric emissions.
- Updated the MLFlow class to conditionally sort and drop duplicates based on the presence of the 'created_at' column, ensuring robustness in data processing.
- Adjusted corresponding tests to validate the new behavior in both classes.
Refactor monitoring and metrics integration across various components
- Removed coverage options from `pyproject.toml`.
- Updated prediction metrics in `README.md` to replace `pipeline_name` with `workflow_name`.
- Upgraded `sientia-dataops-library` dependency version in `requirements-light.txt` and `requirements.txt`.
- Enhanced metrics handling in `laborious` activities, including `Activities`, `Gates`, `MLFlow`, and `OPC`, to utilize a new `MetricsController`.
- Refactored metric emission methods to improve clarity and consistency across the codebase.
- Updated tests to reflect changes in metrics handling and ensure proper functionality.
Enhance thread safety in MLFlowRepository model caching
- Introduced a reentrant lock to synchronize access to the model cache, ensuring thread safety during cache checks and updates.
- Updated the cache management logic to acquire the lock when checking for existing models and when updating the cache after downloading a new model.
- Reduced the maximum cached workflows in the worker configuration for improved resource management.
Refactor model cache deletion in MLFlowRepository
- Updated the model cache deletion logic to remove the entire 'target' key instead of just the 'model' subkey, streamlining the cache management process.
