Update .gitignore, requirements, and enhance OPC UA error handling
- Added new entries to .gitignore for openspec and cursor directories.
- Updated sientia-dataops-library dependency version in requirements-light.txt from 1.10.4 to 1.12.0.
- Enhanced OPC UA communication by refining reconnect logic and error handling in opc_repository.py, including the introduction of a reconnect flag and improved session management.
- Updated tests to cover new reconnect scenarios and ensure robust error handling for protocol states.
Enhance OPC UA communication and metrics tracking
- Updated README.md to include new OPC UA Communication section and detailed metrics for session and write diagnostics.
- Added new metrics in laborious/metrics.py for tracking OPC UA session states and write attempts.
- Refactored OPC activity in laborious/activities/opc.py to handle session errors and improve error reporting.
- Updated e2e tests to cover new scenarios for OPC session/channel errors and reconnect handling.
- Modified .gitignore to include relatorio files and mlruns directory.
- Added ipykernel to requirements-dev.txt for Jupyter notebook support.
SIENTIAPDE-1712 Enhance logging across various classes by adding logger parameters and improving debug statements. This update includes adjustments in Gates, MLFlow, ModelMetrics, and MLFlowRepository classes for better traceability and observability during operations.
SIENTIAPDE-1712 Enhance logging in MLFlowRepository class by introducing a dedicated _debug_dataframe method for conditional logging of DataFrame content. This update improves observability during model transformation, prediction, and retraining processes while managing log output effectively.
SIENTIAPDE-1712 Refactor logging in MLFlowRepository class by removing the debug_dataframe method and replacing it with direct debug statements for improved clarity. This change enhances the logging of DataFrame content during model transformation, prediction, and retraining processes, ensuring better observability without excessive log output.
SIENTIAPDE-1712 Refactor debug logging for DataFrames across multiple classes. Introduced a new method to log DataFrame content conditionally based on row count in Gates, MLFlow, ModelMetrics, and MLFlowRepository classes, improving debugging capabilities while managing log output effectively.
SIENTIAPDE-1712 Implement debug logging for DataFrames in MLFlow and MLFlowRepository classes. Added a method to log DataFrame content conditionally based on row count, enhancing debugging capabilities while preventing excessive log output.
Remove code validation script and refactor imports in activities and workflows
- Deleted the `validate.sh` script, which was responsible for running code quality checks.
- Cleaned up import statements in `activities.py`, `gates.py`, `mlflow.py`, and `storage.py` by removing unused imports and organizing them.
- Refactored initialization methods in `MinioManager` and `MLFlow` classes for improved readability.
- Updated various workflows to ensure compatibility with the new structure and removed unnecessary comments.
- Enhanced test cases to accommodate changes in the activities and workflows, ensuring proper mocking of dependencies.
Implement MinIO Offload and Retention Features
- Added configuration options for MinIO retention hours and offload threshold in README.
- Introduced MinIO payload offloading for large DataFrame-derived payloads, storing them as parquet files.
- Updated activities to utilize MinIO for data loading and cleanup, including new methods for offloading and retention management.
- Refactored existing activities to integrate MinIO functionality, ensuring compatibility with previous workflows.
- Removed the legacy MinioRepository class, consolidating MinIO operations under a new manager structure.
- Updated requirements to use the latest version of the sientia-dataops-library.
Refactor validation script and improve logging in API and model repository
- Updated validation script to include 'e2e/' directory in code formatting and linting checks.
- Enhanced error logging in API class to improve readability of error messages.
- Refactored debug logging in model repository for better structured output.
- Cleaned up import statements in various files for improved organization.
Enhance MLFlow logging and add skip_transform option in MLFlowRepository
- Updated logging in mlflow.py to output processed input data as CSV.
- Introduced skip_transform parameter in MLFlowRepository to conditionally bypass data transformation.
- Improved logging in model_repository.py to display data in a more structured format (to_dict) for predictions and transformations.
Update sientia-mlops-library dependency to version 0.40.6 and refactor get_prediction_data method calls for improved readability in model_repository.py and test_model_repository.py.
Enhance prediction data retrieval in MLFlowRepository by adding predict_flavor parameter
- Updated get_prediction_data method to accept a predict_flavor argument, allowing for different prediction model handling.
- Adjusted calls to get_prediction_data throughout the codebase to include the new parameter.
- Added new test cases to validate behavior for different predict_flavor values, ensuring robust functionality.
SIENTIAPDE-1273
Enhance security analysis and SQL injection handling
- Added skip for potential SQL injection false positives in Bandit configuration.
- Updated validate.sh to use the pyproject.toml configuration for Bandit security analysis.
- Refactored code to replace ensure_dataframe utility with direct DataFrame usage in multiple activities, improving clarity and reducing dependencies.
- Removed the deprecated dataframe_utils module to streamline the codebase.
Enhance MLFlowRepository and Activities classes with new methods and metrics
- Added `check_artifact_exists` method to MLFlowRepository for verifying artifact presence in the MLflow Model Registry.
- Implemented `get_prediction_data` method in MLFlowRepository to retrieve prediction data from models.
- Updated Activities class to integrate ModelMetrics for improved metrics handling.
- Enhanced tests for artifact existence checks and prediction data retrieval, ensuring robust coverage for new functionalities.
- Updated various workflows to include `transform_table_name` in input data for better data handling.
Update requirements and enhance metrics and data handling
- Updated the sientia-dataops-library dependency version in requirements.txt to 1.5.3.
- Added new metrics for model analysis, including lag, count, and error count in metrics.py.
- Implemented a new method for formatting transformed data in gates.py.
- Enhanced MLFlowRepository with methods to load artifact dataframes and calculate model metrics, including drift and performance metrics.
- Updated the prediction process to handle transformed data and ensure proper execution of related activities in format_and_export_prediction.py and prediction_process.py.
Refactor OPC data handling in opc_repository.py to improve code clarity. Updated commented-out code for SourceTimestamp to include NOSONAR annotations, ensuring better code quality checks.
Update logging in MLFlowRepository and OpcRepository to use unified logging methods
- Refactored logging calls in MLFlowRepository to replace `self.logger.info` and `self.logger.debug` with `self.info` and `self.debug` for consistency.
- Updated connection logging in OpcRepository to format the connection message properly.
- Adjusted test cases to reflect changes in logging behavior and ensure proper assertions.
Enhance OPC connection metrics by emitting total connections metric upon connection attempt. This change ensures accurate tracking of connection attempts in the OpcRepository class.
Refactor OPC class to remove unused opc_servers attribute and streamline initialization. Update OpcRepository to consolidate server name tags for metric emissions.
SIENTIAPDE-1325 Add OPC server name configuration and update repository handling
- Introduced a new environment variable `OPC_SERVER_NAME` in values.yaml with a default value.
- Updated the OPC class to include `server_name` when initializing OpcRepository.
- Enhanced the configuration builder to retrieve `OPC_SERVER_NAME` from the environment.
- Adjusted OpcRepository to store and utilize the `server_name` for metric emissions.
Refactor OPC metrics in metrics.py and opc_repository.py
- Replaced existing OPC connection metrics with new metrics for total connections and failed connections.
- Updated the OPC connection status metric to include server URL and adjusted corresponding metric emissions in opc_repository.py.
Update image tag and enhance OPC metrics tracking
- Bumped image tag from "1.0.1" to "1.1.0" in values.yaml.
- Added new metrics for OPC connection count and error count in metrics.py.
- Refactored OPC connection handling in opc_repository.py to emit new metrics for connection status, count, and errors.
Refactor monitoring and metrics integration across various components
- Removed coverage options from `pyproject.toml`.
- Updated prediction metrics in `README.md` to replace `pipeline_name` with `workflow_name`.
- Upgraded `sientia-dataops-library` dependency version in `requirements-light.txt` and `requirements.txt`.
- Enhanced metrics handling in `laborious` activities, including `Activities`, `Gates`, `MLFlow`, and `OPC`, to utilize a new `MetricsController`.
- Refactored metric emission methods to improve clarity and consistency across the codebase.
- Updated tests to reflect changes in metrics handling and ensure proper functionality.
Enhance thread safety in MLFlowRepository model caching
- Introduced a reentrant lock to synchronize access to the model cache, ensuring thread safety during cache checks and updates.
- Updated the cache management logic to acquire the lock when checking for existing models and when updating the cache after downloading a new model.
- Reduced the maximum cached workflows in the worker configuration for improved resource management.
Refactor model cache deletion in MLFlowRepository
- Updated the model cache deletion logic to remove the entire 'target' key instead of just the 'model' subkey, streamlining the cache management process.
Refactor OPC Class and Enhance Testing for Output Management
- Removed the unused 'success' parameter from the manage_output_tags method in the OPC class to streamline its signature.
- Commented out the error handling logic in the OpcRepository for clarity and future reference.
- Added new tests for manage_output_tags to validate successful and failed write operations, ensuring accurate assertions for output data and metrics.
Refactor OPC Write Method and Comment Out Error Handling Logic
- Reformatted the `write_opc_data` method for improved readability by adjusting its signature.
- Commented out the error handling logic in `OpcRepository` related to disconnection due to multiple errors, preserving the original functionality for future reference.
- Updated tests to reflect the commented-out error handling, ensuring clarity in the testing process.
Enhance OPC Metrics Handling and Refactor Write Operations
- Updated the OPC class to return response times for write operations, improving metrics tracking.
- Refactored the Gates activity to incorporate OPC metrics into the metrics writing process.
- Adjusted the manage_output_tags method in OpcRepository to return response times for each tag written.
- Modified tests to validate the new metrics structure and ensure correct behavior of the updated methods.
Refactor error handling in MLFlowRepository and update tests
- Improved error message formatting in MLFlowRepository for better readability.
- Updated test assertions to ensure correct calls to MLflow methods during experiment retrieval and creation.
Update tests notebook execution count and bump image tag in values.yaml
- Incremented the execution count in the tests notebook for accurate tracking.
- Updated the image tag in values.yaml from "0.0.3" to "1.0.1" for versioning consistency.
- Enhanced error handling in model_repository.py to ensure proper retrieval of created experiments.
Refactor OPC handling by removing pod_id from initialization and updating logging format
- Removed pod_id parameter from OPC class and repository initialization to streamline connection management.
- Updated logging statements for improved readability during disconnection attempts and error handling.
Enhance OpcRepository client initialization and error handling
- Updated the Client instantiation in OpcRepository to include a timeout and watchdog interval for improved connection management.
- Added a disconnection call in the exception handling block to ensure proper resource cleanup during connection failures.
Update dependencies and enhance OpcRepository functionality
- Updated sientia-dataops-library version to 1.4.7 in requirements files.
- Modified GITHUB_BRANCH in values.yaml for improved pipeline management.
- Refactored OpcRepository class to inherit from BaseActivity, adding enhanced logging and error handling during disconnection.
- Implemented a disconnection fallback mechanism to ensure graceful handling of OPC server disconnections.
Enhance README and repository utilities for clarity and functionality
- Updated README.md to improve descriptions and structure, adding detailed sections for features, workflows, and architecture.
- Enhanced MinioRepository with comprehensive docstrings for methods and class attributes, improving usability and documentation.
- Refined MLFlowRepository with clearer method descriptions and improved logging for better observability and maintainability.
Refactor tests and update model_repository.py for clarity and consistency
- Added a blank line in model_repository.py for improved readability.
- Adjusted formatting in test_mlflow.py to streamline assertions.
- Updated ensure_bucket_exists method tests in test_minio_repository.py to reflect the new return value of None instead of True.
Refactor MinioRepository and MLFlowRepository for improved functionality and error handling
- Updated `ensure_bucket_exists` method in MinioRepository to return None instead of a boolean, streamlining bucket existence checks.
- Replaced hardcoded error messages in MLFlowRepository with a constant for better maintainability.
- Adjusted column assignment in MLFlowRepository to use pd.Index for improved clarity.
Enhance validation script and refactor code in various modules
- Updated the validation script to include automatic code formatting and linting fixes using Ruff.
- Removed the `clean_tmp_files` method from the Gates class to streamline functionality.
- Simplified conditional checks in the OpcRepository for better clarity and error handling.
- Added model ID to the minimal retrain workflow for improved tracking.
- Introduced new test cases for error handling in MLFlow and storage operations, ensuring robustness in repository interactions.
Update .gitignore and refactor metrics.py for improved logging and consistency
- Added coverage.xml to .gitignore to prevent tracking of coverage reports.
- Refactored metric labels in metrics.py for consistency in string formatting and improved readability.
- Enhanced logging messages in various activities to ensure uniformity in message formatting.
Refactor `force_memory_release` function in model_repository.py to improve logging
- Updated the `force_memory_release` function to remove metadata parameter and enhance logging by using `logger.info()` instead of `logger.custom_info()`.
- Adjusted calls to `force_memory_release` in the MLFlowRepository to pass the logger instance, ensuring consistent logging during memory management operations.
Refactor memory management in MLFlowRepository to use `force_memory_release` function
- Replaced direct calls to `gc.collect()` with `force_memory_release()` for improved memory optimization after model deletion.
- This change enhances memory management during model operations, ensuring more efficient resource handling.
Add memory management function to model_repository.py
- Introduced `force_memory_release` function to enhance memory management by triggering garbage collection and attempting to release unused memory.
- Utilized `ctypes` to call `malloc_trim` for further memory optimization, improving overall performance during model operations.
Enhance MLFlowRepository memory management by adding garbage collection and logging for model deletion
- Introduced garbage collection after model deletion to optimize memory usage.
- Added logging to inform when a model is deleted from memory, improving traceability during predictions.
