Refactor logging in Gates and MLFlow activities to use info level for key operations
- Updated logging statements in the Gates class to replace debug logs with info logs for input and output gate operations, enhancing visibility.
- Modified MLFlow class to use info logs for data transformation and prediction processes, improving clarity in the logging output.
- Adjusted OPC class to return the count of successfully written tags, providing better insight into data writing operations.
Enhance logging in MLFlow activity to include formatted JSON response data
- Added JSON import for better formatting of response data in debug logs.
- Updated logging statements to output transformed and prediction response data as pretty-printed JSON, improving readability and debugging context.
Update dependencies and refactor logging imports for observability
- Updated the sientia-dataops-library dependency version to 1.4.0 in requirements.txt.
- Changed image tag in values.yaml from 0.3.2 to 0.4.1.
- Refactored logging imports across multiple files to use the new observability module instead of the temporal.utils.logger.
- Updated retry policy imports in workflow files to reflect the new module structure.
Implement server validation and output tag management in OPC class
- Added a new method to validate the existence of OPC servers before writing data, improving error handling.
- Introduced a method to manage writing of prediction and confidence tags, streamlining the data writing process.
- Refactored the write_opc_data method to utilize the new validation and management methods for better code organization and clarity.
Refactor OPC data writing and repository initialization
- Simplified success tracking logic in the OPC class for writing prediction and confidence data.
- Removed unused pod_id attribute from OpcRepository initialization.
- Updated test cases to include pod_id for improved metrics tracking during data writing operations.
Refactor OPC data writing to improve success tracking
- Updated the OPC class to store the success status of data writing operations for both prediction and confidence tags.
- Added logging for successful and failed writes to the OPC server, enhancing traceability of data operations.
Implement OPC writing metrics and enhance OPC class initialization
- Added metrics for counting predictions written to the OPC server and monitoring their response times.
- Enhanced the OPC class initialization to include the pod ID for better tracking.
- Updated the write method to increment the prediction count and observe response times.
Enhance BaseActivity initialization across multiple activities to include error counter
- Updated the initialization of the BaseActivity in Gates, MLFlow, and OPC classes to set the error counter to True, improving error tracking and handling capabilities.
Update dependencies and configuration for improved functionality
- Updated sientia-dataops-library version from 1.3.7 to 1.3.8 in requirements.txt.
- Changed GITHUB_BRANCH in values.yaml to reflect new testing focus: SIENTIAPDE-1169.
- Modified notification level in gates.py from WARNING to ERROR for better error handling.
pdate metrics tracking to include response time histogram
- Changed PREDICTION_RESPONSE_TIME_MONITOR from Gauge to Histogram for better response time analysis.
- Updated response time observation method in gates.py to utilize the new Histogram functionality.
SIENTIAPDE-1174 Update replica count and enhance logging in MLFlow
- Changed replicaCount in values.yaml from 3 to 1 for reduced resource usage.
- Added debug logging for prediction response data in MLFlow to improve traceability.
Update dependencies, modify replica count, and implement metrics tracking
- Updated sientia-dataops-library version from 1.3.5 to 1.3.7 in requirements.txt.
- Changed replicaCount in values.yaml from 5 to 3 and incremented image tag from 0.2.7 to 0.3.1.
- Added Prometheus metrics tracking in gates.py and worker.py, including a new write_metrics method.
- Configured Prometheus service and ServiceMonitor in values.yaml for metrics collection.
Update GITHUB_BRANCH in values.yaml and enhance error notification handling in MLFlow
- Changed GITHUB_BRANCH in values.yaml to reflect the new pipeline for alerts orchestration.
- Added NotificationLevel.ERROR to error notifications in MLFlow for retraining and production model updates.
- Updated tests to verify error notification levels for model management workflows.
Update dependencies and enhance ML model retraining functionality
- Updated sientia-dataops-library version in requirements.txt from 1.3.3 to 1.3.4.
- Incremented image tag in values.yaml from 0.2.4 to 0.2.5 and added a new environment variable MONGODB_TTL_INDEX_HOURS.
- Introduced new methods in MLFlowRepository for model retraining and production model updates, including error handling and logging.
- Added retrain_model and update_production_model activities in mlflow.py to support model management workflows.
- Modified MongoDB connection settings in connectors_config.py for improved security and configuration flexibility.
Update GITHUB_BRANCH in values.yaml to reflect new stress testing branch and modify OPC error handling to set success flag instead of continuing on error.
Enhance OPC error handling by adding detailed notifications for missing servers during write operations. The notification includes the list of available OPC servers for better debugging.
Refactor OPC connection handling to improve error notifications and update GITHUB_BRANCH in values.yaml for MongoDB integration. Change requirements.txt to point to local dataops library path.
Update notification handler references to use CoreNotificationHandler and add MongoDB configuration to connectors. Update GITHUB_BRANCH in values.yaml for optimization tasks.
Refactor notification handling in Gates and OPC activities to use send_notification method with metadata integration, enhancing error reporting and traceability.
Refactor Gates and OpcRepository for improved logging and error handling. Removed debug statement in Gates and enhanced disconnect method in OpcRepository to log disconnection status and handle exceptions.
Refactor OPC activity and OpcRepository to include metadata parameter in write_data methods, improving error handling and logging capabilities for better traceability.
SIENTIAPDE-1110 Refactor OPC activity to use server IDs instead of names, update values.yaml for OPC_ID, and enhance OpcRepository initialization for improved clarity and consistency.
Refactor logging in MLFlow and Gates activities; remove print statement in FormatAndExportPrediction; update data handling in PredictionProcess; delete unused redis-feeder script.
Refactor tests for Postgres activities and improve error handling
- Updated test_postgres.py to enhance the testing of load_custom_query method, including cases for None data and date conversion.
- Refactored repeat_last_prediction tests to use mocks for SQLAlchemy session execution.
- Added tests for export_data_to_postgres method, covering both success and error scenarios.
- Improved the initialization tests for Activities class to ensure proper instantiation of dependencies.
- Enhanced test coverage for OPC repository connection validation.
- Updated tests for prediction workflows to streamline input handling and improve clarity.
- Introduced tests for connectors configuration to validate environment variable handling for MLFlow, OPC, and Postgres.
- Added tests for logger utility to ensure default settings are correctly applied.
Refactor and enhance the laborious workflow and utilities
- Removed outdated test file `test_predictions_batch.py` from workflows.
- Added `input_sample.json` for standardized input configuration.
- Introduced `connectors_config.py` to manage database and service configurations.
- Implemented a logging utility in `logger.py` for consistent logging across the application.
- Created `policies.py` to define retry policies for workflows.
- Developed comprehensive tests for `MLFlowRepository` in `test_model_repository.py`.
- Added extensive tests for `OpcRepository` in `test_opc_repository.py`.
- Updated `test_predictions_batch.py` to reflect new workflow structure and testing methodology.
