Update sientia-dataops-library dependency version from 1.8.1 to 1.8.2 in requirements.txt and modify API class to accept a list of response data for PI Web API integration.
SIENTIAPDE-1478 Update tests.ipynb execution counts, modify timestamps, and enhance API class for PI Web API integration
- Adjusted execution counts in tests.ipynb for consistency.
- Updated timestamps in test outputs to reflect new data.
- Refactored API class to streamline data writing to PI Web API by removing redundant endpoint handling.
Update sonar-project.properties to exclude all worker files from coverage and modify execution counts and timestamps in tests.ipynb. Add a new test for empty DataFrame handling in test_model_repository.py.
Refactor validation script and improve logging in API and model repository
- Updated validation script to include 'e2e/' directory in code formatting and linting checks.
- Enhanced error logging in API class to improve readability of error messages.
- Refactored debug logging in model repository for better structured output.
- Cleaned up import statements in various files for improved organization.
Enhance end-to-end tests for PredictionsBatch workflow scenarios
- Introduced mock repositories for PI Web API and OPC operations to improve test coverage.
- Updated test scenarios to handle partial write errors for PI Web API and OPC.
- Refactored existing tests to assert correct behavior under various error conditions.
- Enhanced logging and error handling in API and OPC activities to provide clearer feedback on failures.
- Removed outdated integration test file to streamline test suite.
Enhance end-to-end tests for PredictionsBatch workflow scenarios
- Updated scenario descriptions and assertions for error handling in prediction exports.
- Introduced new test cases for handling exports with only OPC or PI Web API.
- Refactored existing tests to improve clarity and maintainability, including dynamic data insertion.
- Adjusted workflow input configurations to better reflect expected behaviors for various error scenarios.
Refactor end-to-end tests for PredictionsBatch workflow
- Updated the `assert_continue` function to accept dynamic prediction confidence and comments.
- Simplified test scenarios by introducing helper functions for data insertion.
- Enhanced test cases to verify behavior for CONTINUE, STOP, and REPEAT policies at transform gates.
- Improved clarity and maintainability of test structure.
SIENTIAPDE-1478
Enhance end-to-end tests for PredictionsBatch workflow
- Added new test scenarios for input and transform gates handling CONTINUE, STOP, and REPEAT policies.
- Implemented sample data insertion functions for testing various prediction outcomes.
- Updated existing tests to verify behavior under different input conditions and response validations.
- Refactored test structure for clarity and maintainability.
Enhance MLFlow logging and add skip_transform option in MLFlowRepository
- Updated logging in mlflow.py to output processed input data as CSV.
- Introduced skip_transform parameter in MLFlowRepository to conditionally bypass data transformation.
- Improved logging in model_repository.py to display data in a more structured format (to_dict) for predictions and transformations.
Update coverage source in pyproject.toml, add testcontainers for PostgreSQL in requirements-dev.txt, increment image tag and adjust probe delays in values.yaml, and refine condition checks in format_and_export_prediction.py and mlflow.py. Additionally, enhance test coverage in test_gates.py.
Update requirements.txt and values.yaml for dependency and image tag adjustments
- Replaced local path with the correct GitHub URL for sientia-dataops-library in requirements.txt.
- Downgraded the image tag from "1.1.2" to "1.0.1" in values.yaml for consistency with deployment requirements.
Update environment variables in values.yaml and refactor worker.py for improved worker preparation
- Removed KAFKA_BOOTSTRAP_SERVERS from environment variables in values.yaml.
- Added PYPI_SERVER environment variable for library distribution.
- Refactored worker.py to replace resource tuner and poller behavior with a new prepare_worker function, streamlining worker initialization and enhancing code clarity.
SIENTIAPDE-1478
Implement PI Web API response processing and metrics tracking
- Added a new method in the API class to process responses from the PI Web API, validating tag writes and emitting metrics for success and errors.
- Enhanced error handling for missing WebIds and tag names in responses, with appropriate logging and notifications.
- Updated tests to cover various scenarios for processing PI Web API responses, ensuring robust functionality and metrics emission.
- Refactored existing methods to integrate the new response processing logic, improving overall code clarity and maintainability.
Enhance README and Codebase with PI Web API Integration
- Updated README.md to include details about PI Web API integration, including configuration and export capabilities.
- Modified Activities class to incorporate PI Web API export operations and error handling.
- Added new API class for handling PI Web API interactions, including writing prediction and confidence data.
- Updated prediction workflows to support PI Web API output configuration.
- Enhanced worker and sub-workflows to include PI Web API in task queues and export processes.
- Improved documentation and error handling for PI Web API connections and configurations.
Refactor Activities and API Integration for PI Web API
- Reintroduced the API import in the Activities class for proper integration.
- Cleaned up whitespace and formatting in the API class and related tests for improved readability.
- Updated test cases to ensure consistent formatting in error messages and configuration structures for PI Web API.
- Enhanced connectors_config.py with additional whitespace for better organization.
Enhance Activities and Prediction Workflows with PI Web API Integration
- Updated the Activities class to include API integration, allowing for configuration of PI Web API parameters.
- Modified prediction workflows to support output configuration for PI Web API, enabling data writing to the API.
- Refactored connectors_config.py by removing unused PostgreSQL and MongoDB configuration functions.
- Added tests to validate the new PI Web API functionality in activities and workflows, ensuring robust integration and functionality.
Refactor import statements in worker.py for improved organization
- Moved the import of the os module to the appropriate section, enhancing clarity and consistency in the import structure.
Refactor imports in opc.py to improve code clarity
- Moved the import of Hashable from collections.abc to the appropriate section, enhancing readability and organization of imports.
Update sientia-mlops-library dependency to version 0.40.6 and refactor get_prediction_data method calls for improved readability in model_repository.py and test_model_repository.py.
Enhance prediction data retrieval in MLFlowRepository by adding predict_flavor parameter
- Updated get_prediction_data method to accept a predict_flavor argument, allowing for different prediction model handling.
- Adjusted calls to get_prediction_data throughout the codebase to include the new parameter.
- Added new test cases to validate behavior for different predict_flavor values, ensuring robust functionality.
Enhance resource management and configuration in Laborious worker
- Updated `values.yaml` to define resource limits and requests for better performance tuning.
- Modified environment variables in `worker.py` to support resource-based scaling and improved task queue management.
- Introduced new functions for creating resource tuners and poller behaviors, enhancing scalability and efficiency in handling workloads.
Update sientia-mlops-library dependency to version 0.40.5 and fix SQL interval formatting in Drift and SimpleMetrics workflows for improved query accuracy.
Refactor data handling in Laborious workflows to enhance clarity and compatibility
- Improved data validation and formatting in `gates.py`.
- Streamlined model transformation management in `mlflow.py`.
- Enhanced export functionality in `format_and_export_prediction.py` for better data handling flexibility.
- Expanded test coverage for new features and improved documentation in `README.md`.
Enhance data handling and export processes in Laborious workflows
- Updated `gates.py` to improve data quality validation, filtering, and formatting operations, including enhanced metrics recording.
- Refined `mlflow.py` to better manage model transformations and reference data retrieval from MLflow Model Registry.
- Enhanced `format_and_export_prediction.py` to support separate export of transformed data, improving flexibility in data handling.
- Added comprehensive test coverage for new functionalities, including transformed data formatting and retrain report generation.
- Improved documentation in `README.md` to reflect changes in activities and workflows, ensuring clarity on data processing and export paths.
SIENTIAPDE-1273
Enhance security analysis and SQL injection handling
- Added skip for potential SQL injection false positives in Bandit configuration.
- Updated validate.sh to use the pyproject.toml configuration for Bandit security analysis.
- Refactored code to replace ensure_dataframe utility with direct DataFrame usage in multiple activities, improving clarity and reducing dependencies.
- Removed the deprecated dataframe_utils module to streamline the codebase.
