SIENTIAPDE-1712 Enhance logging in MLFlowRepository class by introducing a dedicated _debug_dataframe method for conditional logging of DataFrame content. This update improves observability during model transformation, prediction, and retraining processes while managing log output effectively.
SIENTIAPDE-1712 Refactor logging in MLFlowRepository class by removing the debug_dataframe method and replacing it with direct debug statements for improved clarity. This change enhances the logging of DataFrame content during model transformation, prediction, and retraining processes, ensuring better observability without excessive log output.
SIENTIAPDE-1712 Refactor logging in MinioDataFramePayload class to utilize custom_debug method for improved clarity and consistency. Enhanced DataFrame size logging by integrating a dedicated debug method, streamlining the logging process.
SIENTIAPDE-1712 Enhance logging in Gates, MLFlow, and Storage classes by integrating logger parameter for improved traceability. This update allows for better monitoring of operations and data handling across these components.
SIENTIAPDE-1712 Implement debug logging in MinioDataFramePayload class for enhanced traceability. Added a static method for conditional logging and integrated debug statements throughout methods to capture DataFrame size estimates, upload actions, and retrieval processes, improving overall observability.
SIENTIAPDE-1712 Refactor debug logging for DataFrames across multiple classes. Introduced a new method to log DataFrame content conditionally based on row count in Gates, MLFlow, ModelMetrics, and MLFlowRepository classes, improving debugging capabilities while managing log output effectively.
SIENTIAPDE-1712 Implement debug logging for DataFrames in MLFlow and MLFlowRepository classes. Added a method to log DataFrame content conditionally based on row count, enhancing debugging capabilities while preventing excessive log output.
Refactor prediction process to use execute_activity_method for MLFlow model transformation and prediction requests, enhancing consistency in workflow execution.
Update Gates and FormatAndExportPrediction classes to standardize timestamp usage
- Changed 'last_timestamp' to 'timestamp' in the Gates class for consistency in input data handling.
- Updated the FormatAndExportPrediction class to reflect the same change in the output data structure.
Update Gates and FormatAndExportPrediction classes to use 'last_timestamp' for improved data handling
- Modified the Gates class to utilize 'last_timestamp' when only one row is present, ensuring accurate timestamp assignment.
- Updated the FormatAndExportPrediction class to replace 'timestamp' with 'last_timestamp' in the output data structure.
Enhance prediction export functionality by adding 'on_conflict' and 'unique_columns' parameters to the payload initialization in FormatAndExportPrediction class.
Enhance E2E testing with MinIO support and update documentation
- Updated `requirements-dev.txt` to include MinIO support in testcontainers.
- Added a new fixture for MinIO container setup in `conftest.py` to facilitate E2E tests involving S3-compatible storage.
- Introduced a new test fixture for activities using a real MinIO container in `conftest.py`.
- Updated E2E test scenarios and documentation to reflect the integration of MinIO for offload uploads and clarified error handling in workflows.
- Refactored existing tests to improve clarity and maintainability.
Update dependencies and refactor input filter handling for consistency
- Updated sientia-dataops-library dependency version from 1.10.3 to 1.10.4 in requirements.txt.
- Refactored input filter handling in the Gates class to read policy and config keys in a case-insensitive manner.
- Updated test cases to ensure consistency in filter key naming conventions across various scenarios.
Update MinioDataFramePayload to reflect changes from training to prediction datasets
- Renamed TRAINING_DATASETS_PREFIX to PREDICTION_DATASETS_PREFIX for clarity.
- Updated object key naming convention to use prediction datasets directory.
Refactor metrics and API handling for improved consistency and clarity
- Removed the SIENTIA_CORE_LABELS constant and replaced it with CORE_LABELS for uniformity across metrics.
- Updated the API class to ensure operation_type is always included in core labels for PI Web API metrics.
- Simplified metric tag handling in the Gates class by consolidating common tags into a single core_tags dictionary.
- Enhanced the PredictionProcess class to improve error handling and variable naming for clarity.
SIENTIAPDE-1712 Add last_timestamp parameter to MLFlow and Gates activities for enhanced tracking
- Introduced last_timestamp parameter in the MLFlow and Gates classes to improve tracking of data processing times.
- Updated MinioDataFramePayload to handle last_timestamp, ensuring it defaults to the maximum timestamp from the dataframe if not provided.
SIENTIAPDE-1712 Add timestamp column to processed data in MLFlow and improve dataframe validation in MinioDataFramePayload
- Added a 'timestamp' column to the processed data in the MLFlow class for better tracking of data entries.
- Updated the validation check in MinioDataFramePayload to handle None values for the dataframe more explicitly.
Refactor imports in gates.py and mlflow.py for improved organization
- Removed unnecessary import statement in mlflow.py and re-added it in a more appropriate location.
- Cleaned up the workflow metadata assignment in gates.py for better readability.
Update environment variables in values.yaml and enhance metric labels in metrics.py
- Changed POSTGRES_USER and POSTGRES_PASSWORD values in values.yaml for improved security.
- Added 'runtime' label to metrics in metrics.py for better environment identification.
- Updated CORE_LABELS to include 'runtime' for consistency across metrics.
- Modified type hint for data parameter in PredictionProcess to use a dictionary for better clarity.
- Adjusted tests to reflect changes in core labels and MinIO configuration.
SIENTIAPDE-1712 Update type hints in PredictionProcess to improve clarity and enforce data structure consistency. Changed `data` parameter to a dictionary type and updated the way `last_timestamp` is accessed.
Refactor MinioDataFramePayload usage across activities
- Updated instances of MinioDataFramePayload initialization in Gates, MLFlow, and Storage classes to use the new from_dict method for better data reconstruction from dictionaries.
- Enhanced the PredictionProcess workflow to utilize the updated payload handling.
- Added passthrough fixtures in tests to accommodate the new from_dict method for consistent testing behavior.
Remove `query_to_minio` method from Storage class and update worker activities to eliminate its usage. This change streamlines the codebase by removing unused functionality related to MinIO queries.
Update liveness and readiness probe initial delays in values.yaml; change GITHUB_BRANCH to feature/SIENTIAPDE-1712; refactor MinioRepository initialization in activities.py
Enhance README and Implement Drift Detection and Metrics Workflows
- Added new sections in README for Drift Workflow and Simple Metrics Workflow, detailing their execution flows and functionalities.
- Introduced `drift.py` for data drift detection, comparing current data against reference datasets.
- Added `simple_metrics.py` for calculating regression metrics (RMSE, MSE, MAE, R²).
- Updated `values.yaml` to include configuration for MinIO retention hours and offload threshold.
- Refactored `minio_dataframe_payload.py` to use the new offload threshold environment variable.
- Adjusted tests to reflect changes in environment variable handling for MinIO offload threshold.
Remove code validation script and refactor imports in activities and workflows
- Deleted the `validate.sh` script, which was responsible for running code quality checks.
- Cleaned up import statements in `activities.py`, `gates.py`, `mlflow.py`, and `storage.py` by removing unused imports and organizing them.
- Refactored initialization methods in `MinioManager` and `MLFlow` classes for improved readability.
- Updated various workflows to ensure compatibility with the new structure and removed unnecessary comments.
- Enhanced test cases to accommodate changes in the activities and workflows, ensuring proper mocking of dependencies.
Implement MinIO Offload and Retention Features
- Added configuration options for MinIO retention hours and offload threshold in README.
- Introduced MinIO payload offloading for large DataFrame-derived payloads, storing them as parquet files.
- Updated activities to utilize MinIO for data loading and cleanup, including new methods for offloading and retention management.
- Refactored existing activities to integrate MinIO functionality, ensuring compatibility with previous workflows.
- Removed the legacy MinioRepository class, consolidating MinIO operations under a new manager structure.
- Updated requirements to use the latest version of the sientia-dataops-library.
Enhance API class with optional operation_type in get_core_labels method
- Added a new method get_core_labels to the API class that generates core labels for metrics.
- Introduced an optional operation_type parameter to control the inclusion of the operation_type key in the returned labels.
- Maintained compatibility with the base implementation while providing flexibility for metrics without the operation_type label.
Enhance API class with default headers for JSON requests
- Added a headers configuration to the API class to specify default Content-Type, Accept, x-requested-with, and User-Agent for JSON requests.
Update sientia-dataops-library dependency version from 1.8.1 to 1.8.2 in requirements.txt and modify API class to accept a list of response data for PI Web API integration.
SIENTIAPDE-1478 Update tests.ipynb execution counts, modify timestamps, and enhance API class for PI Web API integration
- Adjusted execution counts in tests.ipynb for consistency.
- Updated timestamps in test outputs to reflect new data.
- Refactored API class to streamline data writing to PI Web API by removing redundant endpoint handling.
Refactor validation script and improve logging in API and model repository
- Updated validation script to include 'e2e/' directory in code formatting and linting checks.
- Enhanced error logging in API class to improve readability of error messages.
- Refactored debug logging in model repository for better structured output.
- Cleaned up import statements in various files for improved organization.
Enhance end-to-end tests for PredictionsBatch workflow scenarios
- Introduced mock repositories for PI Web API and OPC operations to improve test coverage.
- Updated test scenarios to handle partial write errors for PI Web API and OPC.
- Refactored existing tests to assert correct behavior under various error conditions.
- Enhanced logging and error handling in API and OPC activities to provide clearer feedback on failures.
- Removed outdated integration test file to streamline test suite.
Enhance MLFlow logging and add skip_transform option in MLFlowRepository
- Updated logging in mlflow.py to output processed input data as CSV.
- Introduced skip_transform parameter in MLFlowRepository to conditionally bypass data transformation.
- Improved logging in model_repository.py to display data in a more structured format (to_dict) for predictions and transformations.
Update coverage source in pyproject.toml, add testcontainers for PostgreSQL in requirements-dev.txt, increment image tag and adjust probe delays in values.yaml, and refine condition checks in format_and_export_prediction.py and mlflow.py. Additionally, enhance test coverage in test_gates.py.
