Enhance BaseActivity initialization across multiple activities to include error counter
- Updated the initialization of the BaseActivity in Gates, MLFlow, and OPC classes to set the error counter to True, improving error tracking and handling capabilities.
Update dependencies and configuration for improved functionality
- Updated sientia-dataops-library version from 1.3.7 to 1.3.8 in requirements.txt.
- Changed GITHUB_BRANCH in values.yaml to reflect new testing focus: SIENTIAPDE-1169.
- Modified notification level in gates.py from WARNING to ERROR for better error handling.
pdate metrics tracking to include response time histogram
- Changed PREDICTION_RESPONSE_TIME_MONITOR from Gauge to Histogram for better response time analysis.
- Updated response time observation method in gates.py to utilize the new Histogram functionality.
Update dependencies, modify replica count, and implement metrics tracking
- Updated sientia-dataops-library version from 1.3.5 to 1.3.7 in requirements.txt.
- Changed replicaCount in values.yaml from 5 to 3 and incremented image tag from 0.2.7 to 0.3.1.
- Added Prometheus metrics tracking in gates.py and worker.py, including a new write_metrics method.
- Configured Prometheus service and ServiceMonitor in values.yaml for metrics collection.
Update notification handler references to use CoreNotificationHandler and add MongoDB configuration to connectors. Update GITHUB_BRANCH in values.yaml for optimization tasks.
Refactor notification handling in Gates and OPC activities to use send_notification method with metadata integration, enhancing error reporting and traceability.
Refactor Gates and OpcRepository for improved logging and error handling. Removed debug statement in Gates and enhanced disconnect method in OpcRepository to log disconnection status and handle exceptions.
Refactor logging in MLFlow and Gates activities; remove print statement in FormatAndExportPrediction; update data handling in PredictionProcess; delete unused redis-feeder script.
Refactor tests for Postgres activities and improve error handling
- Updated test_postgres.py to enhance the testing of load_custom_query method, including cases for None data and date conversion.
- Refactored repeat_last_prediction tests to use mocks for SQLAlchemy session execution.
- Added tests for export_data_to_postgres method, covering both success and error scenarios.
- Improved the initialization tests for Activities class to ensure proper instantiation of dependencies.
- Enhanced test coverage for OPC repository connection validation.
- Updated tests for prediction workflows to streamline input handling and improve clarity.
- Introduced tests for connectors configuration to validate environment variable handling for MLFlow, OPC, and Postgres.
- Added tests for logger utility to ensure default settings are correctly applied.
Refactor and enhance the laborious workflow and utilities
- Removed outdated test file `test_predictions_batch.py` from workflows.
- Added `input_sample.json` for standardized input configuration.
- Introduced `connectors_config.py` to manage database and service configurations.
- Implemented a logging utility in `logger.py` for consistent logging across the application.
- Created `policies.py` to define retry policies for workflows.
- Developed comprehensive tests for `MLFlowRepository` in `test_model_repository.py`.
- Added extensive tests for `OpcRepository` in `test_opc_repository.py`.
- Updated `test_predictions_batch.py` to reflect new workflow structure and testing methodology.
Implement new get_last_timestamp method in Gates class, refactor MLFlow activity methods to return only transformed data, and update PredictionsBatch and PredictionProcess workflows to utilize Activities module. Add detailed docstrings for new methods and enhance test coverage for get_last_timestamp functionality.
Refactor activity methods and update requirements.txt to enhance functionality and remove deprecated filters. Added detailed docstrings for clarity and improved error handling in data processing workflows.
