Enhance E2E testing with MinIO support and update documentation
- Updated `requirements-dev.txt` to include MinIO support in testcontainers.
- Added a new fixture for MinIO container setup in `conftest.py` to facilitate E2E tests involving S3-compatible storage.
- Introduced a new test fixture for activities using a real MinIO container in `conftest.py`.
- Updated E2E test scenarios and documentation to reflect the integration of MinIO for offload uploads and clarified error handling in workflows.
- Refactored existing tests to improve clarity and maintainability.
Update dependencies and refactor input filter handling for consistency
- Updated sientia-dataops-library dependency version from 1.10.3 to 1.10.4 in requirements.txt.
- Refactored input filter handling in the Gates class to read policy and config keys in a case-insensitive manner.
- Updated test cases to ensure consistency in filter key naming conventions across various scenarios.
Refactor metrics and API handling for improved consistency and clarity
- Removed the SIENTIA_CORE_LABELS constant and replaced it with CORE_LABELS for uniformity across metrics.
- Updated the API class to ensure operation_type is always included in core labels for PI Web API metrics.
- Simplified metric tag handling in the Gates class by consolidating common tags into a single core_tags dictionary.
- Enhanced the PredictionProcess class to improve error handling and variable naming for clarity.
SIENTIAPDE-1712 Add last_timestamp parameter to MLFlow and Gates activities for enhanced tracking
- Introduced last_timestamp parameter in the MLFlow and Gates classes to improve tracking of data processing times.
- Updated MinioDataFramePayload to handle last_timestamp, ensuring it defaults to the maximum timestamp from the dataframe if not provided.
SIENTIAPDE-1712 Add timestamp column to processed data in MLFlow and improve dataframe validation in MinioDataFramePayload
- Added a 'timestamp' column to the processed data in the MLFlow class for better tracking of data entries.
- Updated the validation check in MinioDataFramePayload to handle None values for the dataframe more explicitly.
Refactor imports in gates.py and mlflow.py for improved organization
- Removed unnecessary import statement in mlflow.py and re-added it in a more appropriate location.
- Cleaned up the workflow metadata assignment in gates.py for better readability.
Refactor MinioDataFramePayload usage across activities
- Updated instances of MinioDataFramePayload initialization in Gates, MLFlow, and Storage classes to use the new from_dict method for better data reconstruction from dictionaries.
- Enhanced the PredictionProcess workflow to utilize the updated payload handling.
- Added passthrough fixtures in tests to accommodate the new from_dict method for consistent testing behavior.
Remove `query_to_minio` method from Storage class and update worker activities to eliminate its usage. This change streamlines the codebase by removing unused functionality related to MinIO queries.
Update liveness and readiness probe initial delays in values.yaml; change GITHUB_BRANCH to feature/SIENTIAPDE-1712; refactor MinioRepository initialization in activities.py
Remove code validation script and refactor imports in activities and workflows
- Deleted the `validate.sh` script, which was responsible for running code quality checks.
- Cleaned up import statements in `activities.py`, `gates.py`, `mlflow.py`, and `storage.py` by removing unused imports and organizing them.
- Refactored initialization methods in `MinioManager` and `MLFlow` classes for improved readability.
- Updated various workflows to ensure compatibility with the new structure and removed unnecessary comments.
- Enhanced test cases to accommodate changes in the activities and workflows, ensuring proper mocking of dependencies.
Implement MinIO Offload and Retention Features
- Added configuration options for MinIO retention hours and offload threshold in README.
- Introduced MinIO payload offloading for large DataFrame-derived payloads, storing them as parquet files.
- Updated activities to utilize MinIO for data loading and cleanup, including new methods for offloading and retention management.
- Refactored existing activities to integrate MinIO functionality, ensuring compatibility with previous workflows.
- Removed the legacy MinioRepository class, consolidating MinIO operations under a new manager structure.
- Updated requirements to use the latest version of the sientia-dataops-library.
Enhance API class with optional operation_type in get_core_labels method
- Added a new method get_core_labels to the API class that generates core labels for metrics.
- Introduced an optional operation_type parameter to control the inclusion of the operation_type key in the returned labels.
- Maintained compatibility with the base implementation while providing flexibility for metrics without the operation_type label.
Enhance API class with default headers for JSON requests
- Added a headers configuration to the API class to specify default Content-Type, Accept, x-requested-with, and User-Agent for JSON requests.
Update sientia-dataops-library dependency version from 1.8.1 to 1.8.2 in requirements.txt and modify API class to accept a list of response data for PI Web API integration.
SIENTIAPDE-1478 Update tests.ipynb execution counts, modify timestamps, and enhance API class for PI Web API integration
- Adjusted execution counts in tests.ipynb for consistency.
- Updated timestamps in test outputs to reflect new data.
- Refactored API class to streamline data writing to PI Web API by removing redundant endpoint handling.
Refactor validation script and improve logging in API and model repository
- Updated validation script to include 'e2e/' directory in code formatting and linting checks.
- Enhanced error logging in API class to improve readability of error messages.
- Refactored debug logging in model repository for better structured output.
- Cleaned up import statements in various files for improved organization.
Enhance end-to-end tests for PredictionsBatch workflow scenarios
- Introduced mock repositories for PI Web API and OPC operations to improve test coverage.
- Updated test scenarios to handle partial write errors for PI Web API and OPC.
- Refactored existing tests to assert correct behavior under various error conditions.
- Enhanced logging and error handling in API and OPC activities to provide clearer feedback on failures.
- Removed outdated integration test file to streamline test suite.
Enhance MLFlow logging and add skip_transform option in MLFlowRepository
- Updated logging in mlflow.py to output processed input data as CSV.
- Introduced skip_transform parameter in MLFlowRepository to conditionally bypass data transformation.
- Improved logging in model_repository.py to display data in a more structured format (to_dict) for predictions and transformations.
Update coverage source in pyproject.toml, add testcontainers for PostgreSQL in requirements-dev.txt, increment image tag and adjust probe delays in values.yaml, and refine condition checks in format_and_export_prediction.py and mlflow.py. Additionally, enhance test coverage in test_gates.py.
SIENTIAPDE-1478
Implement PI Web API response processing and metrics tracking
- Added a new method in the API class to process responses from the PI Web API, validating tag writes and emitting metrics for success and errors.
- Enhanced error handling for missing WebIds and tag names in responses, with appropriate logging and notifications.
- Updated tests to cover various scenarios for processing PI Web API responses, ensuring robust functionality and metrics emission.
- Refactored existing methods to integrate the new response processing logic, improving overall code clarity and maintainability.
Enhance README and Codebase with PI Web API Integration
- Updated README.md to include details about PI Web API integration, including configuration and export capabilities.
- Modified Activities class to incorporate PI Web API export operations and error handling.
- Added new API class for handling PI Web API interactions, including writing prediction and confidence data.
- Updated prediction workflows to support PI Web API output configuration.
- Enhanced worker and sub-workflows to include PI Web API in task queues and export processes.
- Improved documentation and error handling for PI Web API connections and configurations.
Refactor Activities and API Integration for PI Web API
- Reintroduced the API import in the Activities class for proper integration.
- Cleaned up whitespace and formatting in the API class and related tests for improved readability.
- Updated test cases to ensure consistent formatting in error messages and configuration structures for PI Web API.
- Enhanced connectors_config.py with additional whitespace for better organization.
Enhance Activities and Prediction Workflows with PI Web API Integration
- Updated the Activities class to include API integration, allowing for configuration of PI Web API parameters.
- Modified prediction workflows to support output configuration for PI Web API, enabling data writing to the API.
- Refactored connectors_config.py by removing unused PostgreSQL and MongoDB configuration functions.
- Added tests to validate the new PI Web API functionality in activities and workflows, ensuring robust integration and functionality.
Refactor imports in opc.py to improve code clarity
- Moved the import of Hashable from collections.abc to the appropriate section, enhancing readability and organization of imports.
Enhance data handling and export processes in Laborious workflows
- Updated `gates.py` to improve data quality validation, filtering, and formatting operations, including enhanced metrics recording.
- Refined `mlflow.py` to better manage model transformations and reference data retrieval from MLflow Model Registry.
- Enhanced `format_and_export_prediction.py` to support separate export of transformed data, improving flexibility in data handling.
- Added comprehensive test coverage for new functionalities, including transformed data formatting and retrain report generation.
- Improved documentation in `README.md` to reflect changes in activities and workflows, ensuring clarity on data processing and export paths.
SIENTIAPDE-1273
Enhance security analysis and SQL injection handling
- Added skip for potential SQL injection false positives in Bandit configuration.
- Updated validate.sh to use the pyproject.toml configuration for Bandit security analysis.
- Refactored code to replace ensure_dataframe utility with direct DataFrame usage in multiple activities, improving clarity and reducing dependencies.
- Removed the deprecated dataframe_utils module to streamline the codebase.
Refactor data size calculation in ModelMetrics class to use shape method for improved accuracy. Changed from using len(target_data) to data.shape[0] for better clarity in data handling.
Update sientia-dataops-library dependency version to 1.6.1 and refactor return types in Gates class methods for improved type clarity. Changed return types from dict[str, Any] to dict for better compatibility with downstream systems.
Refactor return type of get_reference_data method in MLFlow class to improve type specificity. Changed from a list of dictionaries with Hashable keys to a generic list of dictionaries for enhanced clarity in data handling.
Refactor return types in ModelMetrics class methods to enhance type clarity. Updated calculate_drift and calculate_simple_metrics methods to return lists of dictionaries instead of generic lists, improving type specificity for better data handling.
Refactor return type of calculate_drift method in ModelMetrics class to improve compatibility. Changed from a list of dictionaries to a generic list for enhanced flexibility in data handling.
Update dependencies and refactor data handling in various modules
- Updated sientia-dataops-library dependency version from 1.5.3 to 1.5.4 in requirements files.
- Updated sientia-mlops-library dependency version from 0.39.0 to 0.40.2 in requirements files.
- Refactored return types in Gates, MLFlow, and ModelMetrics classes to return dictionaries instead of DataFrames for improved compatibility with downstream systems.
- Removed the temporal_codec module as it is no longer needed for DataFrame serialization.
- Adjusted data handling in the Drift workflow to ensure proper data structure is maintained.
Refactor data handling in various modules to ensure DataFrame consistency
- Replaced direct DataFrame instantiation with `ensure_dataframe` utility in Gates, MLFlow, OPC, and ModelMetrics classes to standardize data handling.
- Updated return types in several asynchronous methods to return DataFrames instead of dictionaries for improved usability.
- Adjusted data export processes in workflows to convert DataFrames to dictionaries with `to_dict(orient='records')` for compatibility with downstream systems.
Update version and enhance metrics calculation in Laborious system
- Updated image tag in values.yaml from 1.1.0 to 1.1.1.
- Modified GITHUB_BRANCH environment variable for consistency.
- Added a new method `calculate_simple_metrics` in model_metrics.py to compute various model performance metrics including RMSE, MSE, MAE, and R2.
- Integrated the new metrics calculation into the worker setup, allowing for concurrent processing of simple metrics.
- Updated tests to cover the new metrics calculation functionality, ensuring comprehensive validation of the implementation.
Enhance MLFlowRepository and Activities classes with new methods and metrics
- Added `check_artifact_exists` method to MLFlowRepository for verifying artifact presence in the MLflow Model Registry.
- Implemented `get_prediction_data` method in MLFlowRepository to retrieve prediction data from models.
- Updated Activities class to integrate ModelMetrics for improved metrics handling.
- Enhanced tests for artifact existence checks and prediction data retrieval, ensuring robust coverage for new functionalities.
- Updated various workflows to include `transform_table_name` in input data for better data handling.
Update requirements and enhance metrics and data handling
- Updated the sientia-dataops-library dependency version in requirements.txt to 1.5.3.
- Added new metrics for model analysis, including lag, count, and error count in metrics.py.
- Implemented a new method for formatting transformed data in gates.py.
- Enhanced MLFlowRepository with methods to load artifact dataframes and calculate model metrics, including drift and performance metrics.
- Updated the prediction process to handle transformed data and ensure proper execution of related activities in format_and_export_prediction.py and prediction_process.py.
Update logging in MLFlowRepository and OpcRepository to use unified logging methods
- Refactored logging calls in MLFlowRepository to replace `self.logger.info` and `self.logger.debug` with `self.info` and `self.debug` for consistency.
- Updated connection logging in OpcRepository to format the connection message properly.
- Adjusted test cases to reflect changes in logging behavior and ensure proper assertions.
Refactor OPC class to remove unused opc_servers attribute and streamline initialization. Update OpcRepository to consolidate server name tags for metric emissions.
SIENTIAPDE-1325 Add OPC server name configuration and update repository handling
- Introduced a new environment variable `OPC_SERVER_NAME` in values.yaml with a default value.
- Updated the OPC class to include `server_name` when initializing OpcRepository.
- Enhanced the configuration builder to retrieve `OPC_SERVER_NAME` from the environment.
- Adjusted OpcRepository to store and utilize the `server_name` for metric emissions.
Enhance metrics handling in Gates and MLFlow classes
- Added checks for `None` response times before emitting OPC writing metrics in the Gates class to prevent unnecessary metric emissions.
- Updated the MLFlow class to conditionally sort and drop duplicates based on the presence of the 'created_at' column, ensuring robustness in data processing.
- Adjusted corresponding tests to validate the new behavior in both classes.
Refactor monitoring and metrics integration across various components
- Removed coverage options from `pyproject.toml`.
- Updated prediction metrics in `README.md` to replace `pipeline_name` with `workflow_name`.
- Upgraded `sientia-dataops-library` dependency version in `requirements-light.txt` and `requirements.txt`.
- Enhanced metrics handling in `laborious` activities, including `Activities`, `Gates`, `MLFlow`, and `OPC`, to utilize a new `MetricsController`.
- Refactored metric emission methods to improve clarity and consistency across the codebase.
- Updated tests to reflect changes in metrics handling and ensure proper functionality.
