Enhance MLFlow reference data handling and testing
- Updated the artifact handling in MLFlow to prioritize 'retrain_input.csv' over 'train_data.csv' when resolving reference data.
- Introduced new methods for resolving artifact names and locating downloaded CSV files.
- Modified the `get_reference_data` method to improve artifact resolution and error handling.
- Expanded unit tests to cover scenarios for missing artifacts and preference logic between retrain and train data CSVs.
Update values.yaml and connectors_config.py for runtime and environment variable changes
- Renamed the runtime entry from "single" to "legacy" in values.yaml for clarity.
- Updated the helm upgrade command comment in values.yaml to reflect the correct path for deployment.
- Changed the environment variable name from "MLFLOW_HOST" and "MLFLOW_PORT" to "MLFLOW_URL" in connectors_config.py for consistency.
Update requirements and enhance OPC communication handling
- Updated `requirements-local.txt` to use the latest versions of `sientia-dataops-library` (1.12.1) and `sientia-model-library` (0.10.0).
- Improved documentation in `opc-communication.md` to clarify the transition from async to sync implementation and added details on error classification and reconnection behavior.
- Refactored `init_opc` method in `opc.py` to enhance connection handling and logging, ensuring independent server initialization.
- Enhanced validation and writing methods in `opc.py` to provide better feedback and error handling for OPC server operations.
- Updated `opc_repository.py` to improve error payload construction and session management metrics.
Enhance OPC UA communication and metrics tracking
- Updated README.md to include new OPC UA Communication section and detailed metrics for session and write diagnostics.
- Added new metrics in laborious/metrics.py for tracking OPC UA session states and write attempts.
- Refactored OPC activity in laborious/activities/opc.py to handle session errors and improve error reporting.
- Updated e2e tests to cover new scenarios for OPC session/channel errors and reconnect handling.
- Modified .gitignore to include relatorio files and mlruns directory.
- Added ipykernel to requirements-dev.txt for Jupyter notebook support.
SIENTIAPDE-1646 Enhance OPC integration and E2E testing framework
- Updated .gitignore to exclude '.cursor/*' for better file management.
- Added new marker in pyproject.toml for E2E tests using OPC.
- Introduced async OPC server fixtures and improved connection handling in conftest.py.
- Enhanced error handling in OPC activities and added metrics for session management in metrics.py.
- Updated E2E tests to cover new OPC scenarios, including session errors and reconnect handling.
- Refactored helper functions to improve prediction assertion logic in helpers.py.
- Documented new OPC scenarios in scenarios.md for clarity on expected outcomes.
Refactor OPC and OpcRepository classes to remove logger dependency and streamline logging calls
- Removed logger attribute from OPC and OpcRepository classes, replacing direct logger calls with class methods for logging.
- Updated write_data method signatures to eliminate logger parameter, simplifying the interface.
- Adjusted related tests to reflect changes in method signatures and logging behavior.
Refactor MLflow configuration to use environment variable for tracking URL
- Removed the `_build_mlflow_tracking_url` function and replaced its usage with `getenv` to directly retrieve the `MLFLOW_URL` environment variable.
- This change simplifies the configuration process by allowing users to specify the tracking URL directly through an environment variable.
SIENTIAPDE-1646 Update dependencies and refactor OPC integration
- Replaced `opcua` with `asyncua` in `requirements-local.txt` and `requirements.txt` to utilize the async capabilities.
- Updated `values.yaml` to change the GitHub branch from `feature/SIENTIAPDE-1646` to `release/SIENTIAPDE-1646`.
- Refactored `opc.py` and `opc_repository.py` to accommodate the new `asyncua` library, ensuring compatibility with the synchronous API.
- Adjusted tests in `test_opc_repository.py` to reflect changes in the client implementation and maintain functionality.
Update README, requirements, and E2E tests for improved configuration and functionality
- Enhanced the README with updated model configuration examples, including the addition of an alias for production.
- Removed the `requirements-light.txt` file and updated `requirements-local.txt` and `requirements.txt` to replace `asyncua` with `opcua`.
- Refactored E2E test scenarios to utilize scenario input files for better maintainability and clarity.
- Improved test coverage for MinIO offload functionality and added new helper functions for loading scenario inputs.
- Updated `values.yaml` to reflect new global configurations and environment variables for the laborious worker.
Enhance environment configuration and update dependencies
- Added new environment variables for PluginStore and MLflow configuration in `.env.example`, including `RUNTIME`, `STORE_BASE_URL`, `STORE_OWNER`, `STORE_REPO`, `STORE_BRANCH`, `STORE_USERNAME`, `STORE_PASSWORD`, `STORE_CACHE_TTL_SECONDS`, `PYPI_SERVER`, `PYPI_USERNAME`, and `PYPI_PASSWORD`.
- Updated `git-requirements-mapping.txt` to reflect changes in repository names.
- Modified `requirements-light.txt` and `requirements.txt` to upgrade `sientia-dataops-library` to version 1.12.0 and `sientia-mlops-library` to version 0.8.1.
- Updated `values.yaml` to include new environment variables for worker runtime and PluginStore configuration.
- Refactored E2E tests to utilize new MLflow repository stubs and PluginStore mocks for improved testing accuracy.
SIENTIAPDE-1712 Enhance logging across various classes by adding logger parameters and improving debug statements. This update includes adjustments in Gates, MLFlow, ModelMetrics, and MLFlowRepository classes for better traceability and observability during operations.
SIENTIAPDE-1712 Enhance logging in MLFlowRepository class by introducing a dedicated _debug_dataframe method for conditional logging of DataFrame content. This update improves observability during model transformation, prediction, and retraining processes while managing log output effectively.
SIENTIAPDE-1712 Refactor logging in MLFlowRepository class by removing the debug_dataframe method and replacing it with direct debug statements for improved clarity. This change enhances the logging of DataFrame content during model transformation, prediction, and retraining processes, ensuring better observability without excessive log output.
SIENTIAPDE-1712 Refactor logging in MinioDataFramePayload class to utilize custom_debug method for improved clarity and consistency. Enhanced DataFrame size logging by integrating a dedicated debug method, streamlining the logging process.
SIENTIAPDE-1712 Enhance logging in Gates, MLFlow, and Storage classes by integrating logger parameter for improved traceability. This update allows for better monitoring of operations and data handling across these components.
SIENTIAPDE-1712 Implement debug logging in MinioDataFramePayload class for enhanced traceability. Added a static method for conditional logging and integrated debug statements throughout methods to capture DataFrame size estimates, upload actions, and retrieval processes, improving overall observability.
SIENTIAPDE-1712 Refactor debug logging for DataFrames across multiple classes. Introduced a new method to log DataFrame content conditionally based on row count in Gates, MLFlow, ModelMetrics, and MLFlowRepository classes, improving debugging capabilities while managing log output effectively.
SIENTIAPDE-1712 Implement debug logging for DataFrames in MLFlow and MLFlowRepository classes. Added a method to log DataFrame content conditionally based on row count, enhancing debugging capabilities while preventing excessive log output.
Update dependencies and refactor input filter handling for consistency
- Updated sientia-dataops-library dependency version from 1.10.3 to 1.10.4 in requirements.txt.
- Refactored input filter handling in the Gates class to read policy and config keys in a case-insensitive manner.
- Updated test cases to ensure consistency in filter key naming conventions across various scenarios.
Update MinioDataFramePayload to reflect changes from training to prediction datasets
- Renamed TRAINING_DATASETS_PREFIX to PREDICTION_DATASETS_PREFIX for clarity.
- Updated object key naming convention to use prediction datasets directory.
SIENTIAPDE-1712 Add last_timestamp parameter to MLFlow and Gates activities for enhanced tracking
- Introduced last_timestamp parameter in the MLFlow and Gates classes to improve tracking of data processing times.
- Updated MinioDataFramePayload to handle last_timestamp, ensuring it defaults to the maximum timestamp from the dataframe if not provided.
SIENTIAPDE-1712 Add timestamp column to processed data in MLFlow and improve dataframe validation in MinioDataFramePayload
- Added a 'timestamp' column to the processed data in the MLFlow class for better tracking of data entries.
- Updated the validation check in MinioDataFramePayload to handle None values for the dataframe more explicitly.
Refactor MinioDataFramePayload usage across activities
- Updated instances of MinioDataFramePayload initialization in Gates, MLFlow, and Storage classes to use the new from_dict method for better data reconstruction from dictionaries.
- Enhanced the PredictionProcess workflow to utilize the updated payload handling.
- Added passthrough fixtures in tests to accommodate the new from_dict method for consistent testing behavior.
Enhance README and Implement Drift Detection and Metrics Workflows
- Added new sections in README for Drift Workflow and Simple Metrics Workflow, detailing their execution flows and functionalities.
- Introduced `drift.py` for data drift detection, comparing current data against reference datasets.
- Added `simple_metrics.py` for calculating regression metrics (RMSE, MSE, MAE, R²).
- Updated `values.yaml` to include configuration for MinIO retention hours and offload threshold.
- Refactored `minio_dataframe_payload.py` to use the new offload threshold environment variable.
- Adjusted tests to reflect changes in environment variable handling for MinIO offload threshold.
Remove code validation script and refactor imports in activities and workflows
- Deleted the `validate.sh` script, which was responsible for running code quality checks.
- Cleaned up import statements in `activities.py`, `gates.py`, `mlflow.py`, and `storage.py` by removing unused imports and organizing them.
- Refactored initialization methods in `MinioManager` and `MLFlow` classes for improved readability.
- Updated various workflows to ensure compatibility with the new structure and removed unnecessary comments.
- Enhanced test cases to accommodate changes in the activities and workflows, ensuring proper mocking of dependencies.
Implement MinIO Offload and Retention Features
- Added configuration options for MinIO retention hours and offload threshold in README.
- Introduced MinIO payload offloading for large DataFrame-derived payloads, storing them as parquet files.
- Updated activities to utilize MinIO for data loading and cleanup, including new methods for offloading and retention management.
- Refactored existing activities to integrate MinIO functionality, ensuring compatibility with previous workflows.
- Removed the legacy MinioRepository class, consolidating MinIO operations under a new manager structure.
- Updated requirements to use the latest version of the sientia-dataops-library.
Refactor validation script and improve logging in API and model repository
- Updated validation script to include 'e2e/' directory in code formatting and linting checks.
- Enhanced error logging in API class to improve readability of error messages.
- Refactored debug logging in model repository for better structured output.
- Cleaned up import statements in various files for improved organization.
Enhance MLFlow logging and add skip_transform option in MLFlowRepository
- Updated logging in mlflow.py to output processed input data as CSV.
- Introduced skip_transform parameter in MLFlowRepository to conditionally bypass data transformation.
- Improved logging in model_repository.py to display data in a more structured format (to_dict) for predictions and transformations.
Refactor Activities and API Integration for PI Web API
- Reintroduced the API import in the Activities class for proper integration.
- Cleaned up whitespace and formatting in the API class and related tests for improved readability.
- Updated test cases to ensure consistent formatting in error messages and configuration structures for PI Web API.
- Enhanced connectors_config.py with additional whitespace for better organization.
Enhance Activities and Prediction Workflows with PI Web API Integration
- Updated the Activities class to include API integration, allowing for configuration of PI Web API parameters.
- Modified prediction workflows to support output configuration for PI Web API, enabling data writing to the API.
- Refactored connectors_config.py by removing unused PostgreSQL and MongoDB configuration functions.
- Added tests to validate the new PI Web API functionality in activities and workflows, ensuring robust integration and functionality.
Update sientia-mlops-library dependency to version 0.40.6 and refactor get_prediction_data method calls for improved readability in model_repository.py and test_model_repository.py.
Enhance prediction data retrieval in MLFlowRepository by adding predict_flavor parameter
- Updated get_prediction_data method to accept a predict_flavor argument, allowing for different prediction model handling.
- Adjusted calls to get_prediction_data throughout the codebase to include the new parameter.
- Added new test cases to validate behavior for different predict_flavor values, ensuring robust functionality.
SIENTIAPDE-1273
Enhance security analysis and SQL injection handling
- Added skip for potential SQL injection false positives in Bandit configuration.
- Updated validate.sh to use the pyproject.toml configuration for Bandit security analysis.
- Refactored code to replace ensure_dataframe utility with direct DataFrame usage in multiple activities, improving clarity and reducing dependencies.
- Removed the deprecated dataframe_utils module to streamline the codebase.
Update dependencies and refactor data handling in various modules
- Updated sientia-dataops-library dependency version from 1.5.3 to 1.5.4 in requirements files.
- Updated sientia-mlops-library dependency version from 0.39.0 to 0.40.2 in requirements files.
- Refactored return types in Gates, MLFlow, and ModelMetrics classes to return dictionaries instead of DataFrames for improved compatibility with downstream systems.
- Removed the temporal_codec module as it is no longer needed for DataFrame serialization.
- Adjusted data handling in the Drift workflow to ensure proper data structure is maintained.
Refactor data handling in various modules to ensure DataFrame consistency
- Replaced direct DataFrame instantiation with `ensure_dataframe` utility in Gates, MLFlow, OPC, and ModelMetrics classes to standardize data handling.
- Updated return types in several asynchronous methods to return DataFrames instead of dictionaries for improved usability.
- Adjusted data export processes in workflows to convert DataFrames to dictionaries with `to_dict(orient='records')` for compatibility with downstream systems.
Enhance MLFlowRepository and Activities classes with new methods and metrics
- Added `check_artifact_exists` method to MLFlowRepository for verifying artifact presence in the MLflow Model Registry.
- Implemented `get_prediction_data` method in MLFlowRepository to retrieve prediction data from models.
- Updated Activities class to integrate ModelMetrics for improved metrics handling.
- Enhanced tests for artifact existence checks and prediction data retrieval, ensuring robust coverage for new functionalities.
- Updated various workflows to include `transform_table_name` in input data for better data handling.
Update requirements and enhance metrics and data handling
- Updated the sientia-dataops-library dependency version in requirements.txt to 1.5.3.
- Added new metrics for model analysis, including lag, count, and error count in metrics.py.
- Implemented a new method for formatting transformed data in gates.py.
- Enhanced MLFlowRepository with methods to load artifact dataframes and calculate model metrics, including drift and performance metrics.
- Updated the prediction process to handle transformed data and ensure proper execution of related activities in format_and_export_prediction.py and prediction_process.py.
Refactor OPC data handling in opc_repository.py to improve code clarity. Updated commented-out code for SourceTimestamp to include NOSONAR annotations, ensuring better code quality checks.
Update logging in MLFlowRepository and OpcRepository to use unified logging methods
- Refactored logging calls in MLFlowRepository to replace `self.logger.info` and `self.logger.debug` with `self.info` and `self.debug` for consistency.
- Updated connection logging in OpcRepository to format the connection message properly.
- Adjusted test cases to reflect changes in logging behavior and ensure proper assertions.
Enhance OPC connection metrics by emitting total connections metric upon connection attempt. This change ensures accurate tracking of connection attempts in the OpcRepository class.
Refactor OPC class to remove unused opc_servers attribute and streamline initialization. Update OpcRepository to consolidate server name tags for metric emissions.
