Refactor validation script and improve logging in API and model repository
- Updated validation script to include 'e2e/' directory in code formatting and linting checks.
- Enhanced error logging in API class to improve readability of error messages.
- Refactored debug logging in model repository for better structured output.
- Cleaned up import statements in various files for improved organization.
Enhance end-to-end tests for PredictionsBatch workflow scenarios
- Introduced mock repositories for PI Web API and OPC operations to improve test coverage.
- Updated test scenarios to handle partial write errors for PI Web API and OPC.
- Refactored existing tests to assert correct behavior under various error conditions.
- Enhanced logging and error handling in API and OPC activities to provide clearer feedback on failures.
- Removed outdated integration test file to streamline test suite.
SIENTIAPDE-1478
Enhance end-to-end tests for PredictionsBatch workflow
- Added new test scenarios for input and transform gates handling CONTINUE, STOP, and REPEAT policies.
- Implemented sample data insertion functions for testing various prediction outcomes.
- Updated existing tests to verify behavior under different input conditions and response validations.
- Refactored test structure for clarity and maintainability.
Update coverage source in pyproject.toml, add testcontainers for PostgreSQL in requirements-dev.txt, increment image tag and adjust probe delays in values.yaml, and refine condition checks in format_and_export_prediction.py and mlflow.py. Additionally, enhance test coverage in test_gates.py.
SIENTIAPDE-1478
Implement PI Web API response processing and metrics tracking
- Added a new method in the API class to process responses from the PI Web API, validating tag writes and emitting metrics for success and errors.
- Enhanced error handling for missing WebIds and tag names in responses, with appropriate logging and notifications.
- Updated tests to cover various scenarios for processing PI Web API responses, ensuring robust functionality and metrics emission.
- Refactored existing methods to integrate the new response processing logic, improving overall code clarity and maintainability.
Refactor Activities and API Integration for PI Web API
- Reintroduced the API import in the Activities class for proper integration.
- Cleaned up whitespace and formatting in the API class and related tests for improved readability.
- Updated test cases to ensure consistent formatting in error messages and configuration structures for PI Web API.
- Enhanced connectors_config.py with additional whitespace for better organization.
Enhance Activities and Prediction Workflows with PI Web API Integration
- Updated the Activities class to include API integration, allowing for configuration of PI Web API parameters.
- Modified prediction workflows to support output configuration for PI Web API, enabling data writing to the API.
- Refactored connectors_config.py by removing unused PostgreSQL and MongoDB configuration functions.
- Added tests to validate the new PI Web API functionality in activities and workflows, ensuring robust integration and functionality.
Enhance data handling and export processes in Laborious workflows
- Updated `gates.py` to improve data quality validation, filtering, and formatting operations, including enhanced metrics recording.
- Refined `mlflow.py` to better manage model transformations and reference data retrieval from MLflow Model Registry.
- Enhanced `format_and_export_prediction.py` to support separate export of transformed data, improving flexibility in data handling.
- Added comprehensive test coverage for new functionalities, including transformed data formatting and retrain report generation.
- Improved documentation in `README.md` to reflect changes in activities and workflows, ensuring clarity on data processing and export paths.
SIENTIAPDE-1273
Enhance security analysis and SQL injection handling
- Added skip for potential SQL injection false positives in Bandit configuration.
- Updated validate.sh to use the pyproject.toml configuration for Bandit security analysis.
- Refactored code to replace ensure_dataframe utility with direct DataFrame usage in multiple activities, improving clarity and reducing dependencies.
- Removed the deprecated dataframe_utils module to streamline the codebase.
Update version and enhance metrics calculation in Laborious system
- Updated image tag in values.yaml from 1.1.0 to 1.1.1.
- Modified GITHUB_BRANCH environment variable for consistency.
- Added a new method `calculate_simple_metrics` in model_metrics.py to compute various model performance metrics including RMSE, MSE, MAE, and R2.
- Integrated the new metrics calculation into the worker setup, allowing for concurrent processing of simple metrics.
- Updated tests to cover the new metrics calculation functionality, ensuring comprehensive validation of the implementation.
Enhance MLFlowRepository and Activities classes with new methods and metrics
- Added `check_artifact_exists` method to MLFlowRepository for verifying artifact presence in the MLflow Model Registry.
- Implemented `get_prediction_data` method in MLFlowRepository to retrieve prediction data from models.
- Updated Activities class to integrate ModelMetrics for improved metrics handling.
- Enhanced tests for artifact existence checks and prediction data retrieval, ensuring robust coverage for new functionalities.
- Updated various workflows to include `transform_table_name` in input data for better data handling.
Update logging in MLFlowRepository and OpcRepository to use unified logging methods
- Refactored logging calls in MLFlowRepository to replace `self.logger.info` and `self.logger.debug` with `self.info` and `self.debug` for consistency.
- Updated connection logging in OpcRepository to format the connection message properly.
- Adjusted test cases to reflect changes in logging behavior and ensure proper assertions.
Enhance metrics handling in Gates and MLFlow classes
- Added checks for `None` response times before emitting OPC writing metrics in the Gates class to prevent unnecessary metric emissions.
- Updated the MLFlow class to conditionally sort and drop duplicates based on the presence of the 'created_at' column, ensuring robustness in data processing.
- Adjusted corresponding tests to validate the new behavior in both classes.
Refactor monitoring and metrics integration across various components
- Removed coverage options from `pyproject.toml`.
- Updated prediction metrics in `README.md` to replace `pipeline_name` with `workflow_name`.
- Upgraded `sientia-dataops-library` dependency version in `requirements-light.txt` and `requirements.txt`.
- Enhanced metrics handling in `laborious` activities, including `Activities`, `Gates`, `MLFlow`, and `OPC`, to utilize a new `MetricsController`.
- Refactored metric emission methods to improve clarity and consistency across the codebase.
- Updated tests to reflect changes in metrics handling and ensure proper functionality.
Refactor OPC Test for Output Management
- Added spacing for improved readability in the test_manage_output_tags_failed and test_manage_output_tags_do_nothing functions.
- Reformatted the parameters in the call to manage_output_tags for better clarity and organization.
Enhance OPC Testing for Output Management
- Refactored the test_manage_output_tags_failed function to use parameterization for improved test coverage of different side effects.
- Added a new test_manage_output_tags_do_nothing to verify behavior when an invalid key is provided in the configuration, ensuring that no write operations are performed in this case.
Refactor OPC Class and Enhance Testing for Output Management
- Removed the unused 'success' parameter from the manage_output_tags method in the OPC class to streamline its signature.
- Commented out the error handling logic in the OpcRepository for clarity and future reference.
- Added new tests for manage_output_tags to validate successful and failed write operations, ensuring accurate assertions for output data and metrics.
Refactor OPC Write Test to Separate Output Data and Metrics
- Updated the test for the OPC write operation to separately assert output data and OPC metrics for improved clarity and accuracy in testing.
- Changed the variable names to reflect the new structure of the returned values from the write operation.
Enhance OPC Metrics Handling and Refactor Write Operations
- Updated the OPC class to return response times for write operations, improving metrics tracking.
- Refactored the Gates activity to incorporate OPC metrics into the metrics writing process.
- Adjusted the manage_output_tags method in OpcRepository to return response times for each tag written.
- Modified tests to validate the new metrics structure and ensure correct behavior of the updated methods.
Refactor OPC handling by removing pod_id from initialization and updating logging format
- Removed pod_id parameter from OPC class and repository initialization to streamline connection management.
- Updated logging statements for improved readability during disconnection attempts and error handling.
Refactor tests and update model_repository.py for clarity and consistency
- Added a blank line in model_repository.py for improved readability.
- Adjusted formatting in test_mlflow.py to streamline assertions.
- Updated ensure_bucket_exists method tests in test_minio_repository.py to reflect the new return value of None instead of True.
Enhance validation script and refactor code in various modules
- Updated the validation script to include automatic code formatting and linting fixes using Ruff.
- Removed the `clean_tmp_files` method from the Gates class to streamline functionality.
- Simplified conditional checks in the OpcRepository for better clarity and error handling.
- Added model ID to the minimal retrain workflow for improved tracking.
- Introduced new test cases for error handling in MLFlow and storage operations, ensuring robustness in repository interactions.
Update .gitignore and refactor metrics.py for improved logging and consistency
- Added coverage.xml to .gitignore to prevent tracking of coverage reports.
- Refactored metric labels in metrics.py for consistency in string formatting and improved readability.
- Enhanced logging messages in various activities to ensure uniformity in message formatting.
Refactor model configuration handling in MLFlow and workflows
- Replaced 'model_retention' with 'model_config' to encapsulate retention settings and improve consistency across various components.
- Updated test cases to reflect changes in argument structure, ensuring compatibility with the new model configuration format.
- Added 'prediction_store_policy' to input data handling in workflows for enhanced configuration management.
Refactor datetime index handling in MLFlow and MLFlowRepository
- Moved the detect_and_parse_datetime_index method from MLFlow to MLFlowRepository for better organization and reusability.
- Updated the method to include enhanced logging and error handling for invalid datetime formats.
- Adjusted the transform method in MLFlowRepository to utilize the new datetime index parsing logic.
- Added unit tests for both valid and invalid datetime index cases to ensure robustness.
Refactor model configuration handling in MLFlow and workflows
- Replaced 'model_retention' with 'model_config' to encapsulate retention settings and improve consistency across various components.
- Updated test cases to reflect changes in argument structure, ensuring compatibility with the new model configuration format.
- Added 'prediction_store_policy' to input data handling in workflows for enhanced configuration management.
Refactor datetime index handling in MLFlow and MLFlowRepository
- Moved the detect_and_parse_datetime_index method from MLFlow to MLFlowRepository for better organization and reusability.
- Updated the method to include enhanced logging and error handling for invalid datetime formats.
- Adjusted the transform method in MLFlowRepository to utilize the new datetime index parsing logic.
- Added unit tests for both valid and invalid datetime index cases to ensure robustness.
Refactor test data structure in gates.py and model_repository.py for improved clarity and consistency
- Updated test cases in test_gates.py to use dictionaries for prediction and response_time, associating values with timestamps.
- Modified test_predict_success in test_model_repository.py to create a DataFrame with named indices for better readability in assertions.
Implement prediction store policy handling in Gates activity
- Added a new method `get_prediction_store_policy` to validate and parse the prediction store policy.
- Updated `format_prediction` method to utilize the new policy handling, allowing for sorting of predictions based on the specified policy.
- Enhanced test coverage for the new policy handling, including various scenarios for valid and invalid policies.
- Removed the obsolete `coverage.sh` script.
Refactor OpcRepository and update tests for async handling
- Removed the shutdown method from OpcRepository and adjusted the disconnect logic.
- Updated tests in test_activities.py and test_opc.py to support async shutdown functionality.
- Enhanced test cases in test_opc_repository.py to ensure proper async behavior and error handling in OpcRepository methods.
Update tests in Gates and OpcRepository to enhance logging and include metadata
- Removed unnecessary debug assertion in the Gates activity test for format prediction.
- Updated the test for last timestamp with no data to include a metadata dictionary.
- Modified OpcRepository tests to utilize custom_error logging with metadata for improved error context.
Refactor OPC data writing and repository initialization
- Simplified success tracking logic in the OPC class for writing prediction and confidence data.
- Removed unused pod_id attribute from OpcRepository initialization.
- Updated test cases to include pod_id for improved metrics tracking during data writing operations.
Add async test for write_metrics method in gates_activity
- Implemented a new test to validate the write_metrics functionality, ensuring metrics are correctly recorded for predictions, confidence, and response time.
- Utilized mocking to verify interactions with the metrics tracking system.
SIENTIAPDE-1174 Update tests to enhance MLFlow and workflow activity assertions
- Added logger initialization in MLFlowRepository for improved logging.
- Updated prediction test assertions to reflect changes in output structure.
- Increased activity method call count assertions in workflow tests for accuracy.
Update GITHUB_BRANCH in values.yaml and enhance error notification handling in MLFlow
- Changed GITHUB_BRANCH in values.yaml to reflect the new pipeline for alerts orchestration.
- Added NotificationLevel.ERROR to error notifications in MLFlow for retraining and production model updates.
- Updated tests to verify error notification levels for model management workflows.
Refactor model_repository and enhance test coverage for MLFlow functionalities
- Updated model_repository to ensure the 'temp' directory is created if it doesn't exist using `exist_ok=True`.
- Added new tests for retraining and updating production models, including error handling scenarios.
- Improved existing tests for model management workflows to ensure robustness and reliability.
SIENTIAPDE-1163 Enhance OPC tests to improve error handling and notifications. Added tests for MongoDB configuration and refined connection error handling in OPC repository.
Refactor notification handling in Gates and OPC activities to use send_notification method with metadata integration, enhancing error reporting and traceability.
Refactor tests for Postgres activities and improve error handling
- Updated test_postgres.py to enhance the testing of load_custom_query method, including cases for None data and date conversion.
- Refactored repeat_last_prediction tests to use mocks for SQLAlchemy session execution.
- Added tests for export_data_to_postgres method, covering both success and error scenarios.
- Improved the initialization tests for Activities class to ensure proper instantiation of dependencies.
- Enhanced test coverage for OPC repository connection validation.
- Updated tests for prediction workflows to streamline input handling and improve clarity.
- Introduced tests for connectors configuration to validate environment variable handling for MLFlow, OPC, and Postgres.
- Added tests for logger utility to ensure default settings are correctly applied.
Refactor and enhance the laborious workflow and utilities
- Removed outdated test file `test_predictions_batch.py` from workflows.
- Added `input_sample.json` for standardized input configuration.
- Introduced `connectors_config.py` to manage database and service configurations.
- Implemented a logging utility in `logger.py` for consistent logging across the application.
- Created `policies.py` to define retry policies for workflows.
- Developed comprehensive tests for `MLFlowRepository` in `test_model_repository.py`.
- Added extensive tests for `OpcRepository` in `test_opc_repository.py`.
- Updated `test_predictions_batch.py` to reflect new workflow structure and testing methodology.
