Refactor OPC Class and Enhance Testing for Output Management
- Removed the unused 'success' parameter from the manage_output_tags method in the OPC class to streamline its signature.
- Commented out the error handling logic in the OpcRepository for clarity and future reference.
- Added new tests for manage_output_tags to validate successful and failed write operations, ensuring accurate assertions for output data and metrics.
Refactor OPC Write Method and Comment Out Error Handling Logic
- Reformatted the `write_opc_data` method for improved readability by adjusting its signature.
- Commented out the error handling logic in `OpcRepository` related to disconnection due to multiple errors, preserving the original functionality for future reference.
- Updated tests to reflect the commented-out error handling, ensuring clarity in the testing process.
SIENTIAPDE-1314 Refactor OPC Write Operation in FormatAndExportPrediction Workflow
- Changed the variable name for the result of the OPC write operation for clarity.
- Extracted prediction and metrics from the OPC write result to improve data handling.
Enhance OPC Metrics Handling and Refactor Write Operations
- Updated the OPC class to return response times for write operations, improving metrics tracking.
- Refactored the Gates activity to incorporate OPC metrics into the metrics writing process.
- Adjusted the manage_output_tags method in OpcRepository to return response times for each tag written.
- Modified tests to validate the new metrics structure and ensure correct behavior of the updated methods.
Refactor OPC handling by removing pod_id from initialization and updating logging format
- Removed pod_id parameter from OPC class and repository initialization to streamline connection management.
- Updated logging statements for improved readability during disconnection attempts and error handling.
Update .gitignore and refactor metrics.py for improved logging and consistency
- Added coverage.xml to .gitignore to prevent tracking of coverage reports.
- Refactored metric labels in metrics.py for consistency in string formatting and improved readability.
- Enhanced logging messages in various activities to ensure uniformity in message formatting.
Refactor OPC and model repository for improved functionality and clarity
- Updated OPC server logging to handle missing prediction and confidence tags gracefully.
- Corrected documentation for OPC reconnection interval from milliseconds to seconds.
- Enhanced MLFlowRepository with new methods for model retrieval, caching, and transformation, improving model management and retraining workflows.
Remove Docker configuration files and refactor project structure
- Deleted docker-compose.yml and Dockerfile as part of the project restructuring.
- Updated README.md to reflect changes in project setup and configuration.
- Introduced a new __init__.py file in the laborious package to provide an overview of the system.
- Enhanced documentation across various modules, including metrics, activities, and workflows, to improve clarity and usability.
- Added comprehensive docstrings and comments to key classes and methods for better maintainability.
Update values.yaml and enhance logging in OPC activity
- Change replicaCount from 5 to 1 for reduced resource allocation.
- Update image tag in values.yaml to 0.4.5 for the latest version.
- Modify GITHUB_BRANCH environment variable for improved async handling.
- Add logging statements in OPC class to track server initialization and connection status.
- Expanded the .env file with configurations for MongoDB, Postgres, MlFlow, and Temporal.
- Refactored OPC class methods to be asynchronous, including init_opc, write_data, manage_output_tags, and shutdown.
- Updated the worker to initialize OPC asynchronously and adjusted shutdown handling for activities.
Refactor logging in Gates and MLFlow activities to use info level for key operations
- Updated logging statements in the Gates class to replace debug logs with info logs for input and output gate operations, enhancing visibility.
- Modified MLFlow class to use info logs for data transformation and prediction processes, improving clarity in the logging output.
- Adjusted OPC class to return the count of successfully written tags, providing better insight into data writing operations.
Update dependencies and refactor logging imports for observability
- Updated the sientia-dataops-library dependency version to 1.4.0 in requirements.txt.
- Changed image tag in values.yaml from 0.3.2 to 0.4.1.
- Refactored logging imports across multiple files to use the new observability module instead of the temporal.utils.logger.
- Updated retry policy imports in workflow files to reflect the new module structure.
Implement server validation and output tag management in OPC class
- Added a new method to validate the existence of OPC servers before writing data, improving error handling.
- Introduced a method to manage writing of prediction and confidence tags, streamlining the data writing process.
- Refactored the write_opc_data method to utilize the new validation and management methods for better code organization and clarity.
Refactor OPC data writing and repository initialization
- Simplified success tracking logic in the OPC class for writing prediction and confidence data.
- Removed unused pod_id attribute from OpcRepository initialization.
- Updated test cases to include pod_id for improved metrics tracking during data writing operations.
Refactor OPC data writing to improve success tracking
- Updated the OPC class to store the success status of data writing operations for both prediction and confidence tags.
- Added logging for successful and failed writes to the OPC server, enhancing traceability of data operations.
Implement OPC writing metrics and enhance OPC class initialization
- Added metrics for counting predictions written to the OPC server and monitoring their response times.
- Enhanced the OPC class initialization to include the pod ID for better tracking.
- Updated the write method to increment the prediction count and observe response times.
Enhance BaseActivity initialization across multiple activities to include error counter
- Updated the initialization of the BaseActivity in Gates, MLFlow, and OPC classes to set the error counter to True, improving error tracking and handling capabilities.
Update GITHUB_BRANCH in values.yaml to reflect new stress testing branch and modify OPC error handling to set success flag instead of continuing on error.
Enhance OPC error handling by adding detailed notifications for missing servers during write operations. The notification includes the list of available OPC servers for better debugging.
Refactor OPC connection handling to improve error notifications and update GITHUB_BRANCH in values.yaml for MongoDB integration. Change requirements.txt to point to local dataops library path.
Update notification handler references to use CoreNotificationHandler and add MongoDB configuration to connectors. Update GITHUB_BRANCH in values.yaml for optimization tasks.
Refactor notification handling in Gates and OPC activities to use send_notification method with metadata integration, enhancing error reporting and traceability.
Refactor OPC activity and OpcRepository to include metadata parameter in write_data methods, improving error handling and logging capabilities for better traceability.
SIENTIAPDE-1110 Refactor OPC activity to use server IDs instead of names, update values.yaml for OPC_ID, and enhance OpcRepository initialization for improved clarity and consistency.
Refactor and enhance the laborious workflow and utilities
- Removed outdated test file `test_predictions_batch.py` from workflows.
- Added `input_sample.json` for standardized input configuration.
- Introduced `connectors_config.py` to manage database and service configurations.
- Implemented a logging utility in `logger.py` for consistent logging across the application.
- Created `policies.py` to define retry policies for workflows.
- Developed comprehensive tests for `MLFlowRepository` in `test_model_repository.py`.
- Added extensive tests for `OpcRepository` in `test_opc_repository.py`.
- Updated `test_predictions_batch.py` to reflect new workflow structure and testing methodology.
Refactor activity methods and update requirements.txt to enhance functionality and remove deprecated filters. Added detailed docstrings for clarity and improved error handling in data processing workflows.
