SIENTIAPDE-1712 Add last_timestamp parameter to MLFlow and Gates activities for enhanced tracking
- Introduced last_timestamp parameter in the MLFlow and Gates classes to improve tracking of data processing times.
- Updated MinioDataFramePayload to handle last_timestamp, ensuring it defaults to the maximum timestamp from the dataframe if not provided.
SIENTIAPDE-1712 Add timestamp column to processed data in MLFlow and improve dataframe validation in MinioDataFramePayload
- Added a 'timestamp' column to the processed data in the MLFlow class for better tracking of data entries.
- Updated the validation check in MinioDataFramePayload to handle None values for the dataframe more explicitly.
Refactor imports in gates.py and mlflow.py for improved organization
- Removed unnecessary import statement in mlflow.py and re-added it in a more appropriate location.
- Cleaned up the workflow metadata assignment in gates.py for better readability.
Update environment variables in values.yaml and enhance metric labels in metrics.py
- Changed POSTGRES_USER and POSTGRES_PASSWORD values in values.yaml for improved security.
- Added 'runtime' label to metrics in metrics.py for better environment identification.
- Updated CORE_LABELS to include 'runtime' for consistency across metrics.
- Modified type hint for data parameter in PredictionProcess to use a dictionary for better clarity.
- Adjusted tests to reflect changes in core labels and MinIO configuration.
SIENTIAPDE-1712 Update type hints in PredictionProcess to improve clarity and enforce data structure consistency. Changed `data` parameter to a dictionary type and updated the way `last_timestamp` is accessed.
Refactor MinioDataFramePayload usage across activities
- Updated instances of MinioDataFramePayload initialization in Gates, MLFlow, and Storage classes to use the new from_dict method for better data reconstruction from dictionaries.
- Enhanced the PredictionProcess workflow to utilize the updated payload handling.
- Added passthrough fixtures in tests to accommodate the new from_dict method for consistent testing behavior.
Remove `query_to_minio` method from Storage class and update worker activities to eliminate its usage. This change streamlines the codebase by removing unused functionality related to MinIO queries.
Update liveness and readiness probe initial delays in values.yaml; change GITHUB_BRANCH to feature/SIENTIAPDE-1712; refactor MinioRepository initialization in activities.py
Enhance README and Implement Drift Detection and Metrics Workflows
- Added new sections in README for Drift Workflow and Simple Metrics Workflow, detailing their execution flows and functionalities.
- Introduced `drift.py` for data drift detection, comparing current data against reference datasets.
- Added `simple_metrics.py` for calculating regression metrics (RMSE, MSE, MAE, R²).
- Updated `values.yaml` to include configuration for MinIO retention hours and offload threshold.
- Refactored `minio_dataframe_payload.py` to use the new offload threshold environment variable.
- Adjusted tests to reflect changes in environment variable handling for MinIO offload threshold.
Remove code validation script and refactor imports in activities and workflows
- Deleted the `validate.sh` script, which was responsible for running code quality checks.
- Cleaned up import statements in `activities.py`, `gates.py`, `mlflow.py`, and `storage.py` by removing unused imports and organizing them.
- Refactored initialization methods in `MinioManager` and `MLFlow` classes for improved readability.
- Updated various workflows to ensure compatibility with the new structure and removed unnecessary comments.
- Enhanced test cases to accommodate changes in the activities and workflows, ensuring proper mocking of dependencies.
Implement MinIO Offload and Retention Features
- Added configuration options for MinIO retention hours and offload threshold in README.
- Introduced MinIO payload offloading for large DataFrame-derived payloads, storing them as parquet files.
- Updated activities to utilize MinIO for data loading and cleanup, including new methods for offloading and retention management.
- Refactored existing activities to integrate MinIO functionality, ensuring compatibility with previous workflows.
- Removed the legacy MinioRepository class, consolidating MinIO operations under a new manager structure.
- Updated requirements to use the latest version of the sientia-dataops-library.
Enhance API class with optional operation_type in get_core_labels method
- Added a new method get_core_labels to the API class that generates core labels for metrics.
- Introduced an optional operation_type parameter to control the inclusion of the operation_type key in the returned labels.
- Maintained compatibility with the base implementation while providing flexibility for metrics without the operation_type label.
SIENTIAPDE-1478 Update tests.ipynb to adjust execution counts and enhance plotting limits
- Modified execution counts for code cells to ensure consistency.
- Updated plotting limits to account for both actual values and predictions, improving data visualization accuracy.
Enhance API class with default headers for JSON requests
- Added a headers configuration to the API class to specify default Content-Type, Accept, x-requested-with, and User-Agent for JSON requests.
SIENTIAPDE-1478 Update tests.ipynb to enhance data loading and visualization
- Refactored data loading functions to improve clarity and efficiency.
- Updated plotting functionality to include additional data series and improved axis handling.
- Adjusted execution counts and output formatting for consistency with new data structures.
SIENTIAPDE-1478 Update tests.ipynb with new execution counts, modify output values and timestamps, and enhance plotting functionality
- Adjusted execution counts for code cells to maintain consistency.
- Updated output values and timestamps in test results to reflect new data.
- Enhanced plotting functionality by adding a second subplot for additional data visualization.
Refactor PI Web API response handling in tests
- Updated test cases in test_api.py to handle response data as lists instead of dictionaries for consistency with the API's expected output format.
- Adjusted mock responses to reflect the new structure, ensuring tests accurately simulate API behavior.
- Enhanced clarity in test descriptions and improved overall test coverage for response processing scenarios.
Update sientia-dataops-library dependency version from 1.8.1 to 1.8.2 in requirements.txt and modify API class to accept a list of response data for PI Web API integration.
SIENTIAPDE-1478 Update tests.ipynb execution counts, modify timestamps, and enhance API class for PI Web API integration
- Adjusted execution counts in tests.ipynb for consistency.
- Updated timestamps in test outputs to reflect new data.
- Refactored API class to streamline data writing to PI Web API by removing redundant endpoint handling.
Update sonar-project.properties to exclude all worker files from coverage and modify execution counts and timestamps in tests.ipynb. Add a new test for empty DataFrame handling in test_model_repository.py.
Refactor validation script and improve logging in API and model repository
- Updated validation script to include 'e2e/' directory in code formatting and linting checks.
- Enhanced error logging in API class to improve readability of error messages.
- Refactored debug logging in model repository for better structured output.
- Cleaned up import statements in various files for improved organization.
Enhance end-to-end tests for PredictionsBatch workflow scenarios
- Introduced mock repositories for PI Web API and OPC operations to improve test coverage.
- Updated test scenarios to handle partial write errors for PI Web API and OPC.
- Refactored existing tests to assert correct behavior under various error conditions.
- Enhanced logging and error handling in API and OPC activities to provide clearer feedback on failures.
- Removed outdated integration test file to streamline test suite.
Enhance end-to-end tests for PredictionsBatch workflow scenarios
- Updated scenario descriptions and assertions for error handling in prediction exports.
- Introduced new test cases for handling exports with only OPC or PI Web API.
- Refactored existing tests to improve clarity and maintainability, including dynamic data insertion.
- Adjusted workflow input configurations to better reflect expected behaviors for various error scenarios.
Refactor end-to-end tests for PredictionsBatch workflow
- Updated the `assert_continue` function to accept dynamic prediction confidence and comments.
- Simplified test scenarios by introducing helper functions for data insertion.
- Enhanced test cases to verify behavior for CONTINUE, STOP, and REPEAT policies at transform gates.
- Improved clarity and maintainability of test structure.
SIENTIAPDE-1478
Enhance end-to-end tests for PredictionsBatch workflow
- Added new test scenarios for input and transform gates handling CONTINUE, STOP, and REPEAT policies.
- Implemented sample data insertion functions for testing various prediction outcomes.
- Updated existing tests to verify behavior under different input conditions and response validations.
- Refactored test structure for clarity and maintainability.
Enhance MLFlow logging and add skip_transform option in MLFlowRepository
- Updated logging in mlflow.py to output processed input data as CSV.
- Introduced skip_transform parameter in MLFlowRepository to conditionally bypass data transformation.
- Improved logging in model_repository.py to display data in a more structured format (to_dict) for predictions and transformations.
Update coverage source in pyproject.toml, add testcontainers for PostgreSQL in requirements-dev.txt, increment image tag and adjust probe delays in values.yaml, and refine condition checks in format_and_export_prediction.py and mlflow.py. Additionally, enhance test coverage in test_gates.py.
