SIENTIAPDE-1222
Enhance MLFlow logging with sample dictionary for response data
- Introduced a new method to create a sample dictionary for debugging, allowing for better visualization of nested data structures in logs.
- Updated debug logging to utilize the new sampling method for raw and transformed response data, improving clarity and reducing output size.
- Adjusted logging for processed input data to display only the first few rows, enhancing readability.
Refactor MLFlow logging to enhance data output clarity
- Updated debug logging to use data.to_string() for processed input data, improving readability.
- Modified raw response data logging to format the output as a string, ensuring consistent logging format.
Update values.yaml and MLFlow logging for courier integration
- Changed the image repository to 'sientia-module-courier' and updated the image tag to '0.0.1'.
- Modified environment variables for GITHUB_BRANCH and MLFLOW_PASSWORD to reflect new configurations.
- Enhanced MLFlow logging to include additional debug statements for raw response data and added a check for empty DataFrames.
Refactor MLFlow logging to improve data output clarity
- Updated the debug logging to directly capture the output of data.to_csv, enhancing traceability of processed input data.
- Removed redundant debug statements for raw response data to streamline logging and focus on essential information.
SIENTIAPDE-1214: Enhance MLFlow and tests with datetime index handling and logging improvements
- Added a new method in MLFlow to detect and parse datetime indices in DataFrames, ensuring proper format and raising errors for invalid types.
- Updated prediction workflows to utilize the new datetime index handling, improving data integrity during transformations.
- Enhanced logging in model_repository to include detailed data outputs for better traceability.
- Adjusted timeout settings in prediction workflows for improved execution time management.
- Updated tests.ipynb to include additional checks for index types and outputs for better validation of functionality.
SIENTIAPDE-1214: Refactor MLFlow and model repository methods to use model_config dictionary
- Updated MLFlow class methods to accept model_config instead of model_retention for improved flexibility.
- Modified model_repository methods to handle model_config, extracting necessary parameters for transformation and prediction.
- Adjusted predictions_batch and prediction_process workflows to utilize model_config for better configuration management.
- Commented out the previous sientia-mlops-library dependency in requirements.txt for clarity.
Refactor OPC and model repository for improved functionality and clarity
- Updated OPC server logging to handle missing prediction and confidence tags gracefully.
- Corrected documentation for OPC reconnection interval from milliseconds to seconds.
- Enhanced MLFlowRepository with new methods for model retrieval, caching, and transformation, improving model management and retraining workflows.
Update tests.ipynb and gates.py for enhanced functionality and clarity
- Adjusted execution count in tests.ipynb for reproducibility.
- Modified DataFrame creation in tests.ipynb to dynamically set size and update timestamp frequency.
- Added a new filter function for empty data in gates.py to improve data handling.
Enhance tests.ipynb and gates.py for improved logging and output clarity
- Set execution count to null in tests.ipynb for reproducibility.
- Added stdout output to tests.ipynb for better visibility of results.
- Updated DataFrame display logic in tests.ipynb to include additional print statements for length and size.
- Improved debug logging in gates.py to log prediction data as a string for better traceability.
- Changed info log in gates.py to use len() instead of size for consistency.
Update tests.ipynb and values.yaml for improved functionality and versioning
- Updated execution count in tests.ipynb for reproducibility.
- Modified DataFrame creation in tests.ipynb to include a timestamp column and save to CSV.
- Changed image tag in values.yaml from "0.4.4" to "0.4.5" for versioning.
- Updated GITHUB_BRANCH in values.yaml to reflect the latest branch adjustments.
Remove all module docstrings and the versioning information from the Laborious package, activities, utils, and workflows. This cleanup enhances code readability and reduces unnecessary comments in the codebase.
Remove Docker configuration files and refactor project structure
- Deleted docker-compose.yml and Dockerfile as part of the project restructuring.
- Updated README.md to reflect changes in project setup and configuration.
- Introduced a new __init__.py file in the laborious package to provide an overview of the system.
- Enhanced documentation across various modules, including metrics, activities, and workflows, to improve clarity and usability.
- Added comprehensive docstrings and comments to key classes and methods for better maintainability.
Implement prediction store policy handling in Gates activity
- Added a new method `get_prediction_store_policy` to validate and parse the prediction store policy.
- Updated `format_prediction` method to utilize the new policy handling, allowing for sorting of predictions based on the specified policy.
- Enhanced test coverage for the new policy handling, including various scenarios for valid and invalid policies.
- Removed the obsolete `coverage.sh` script.
Update values.yaml and enhance logging in OPC activity
- Change replicaCount from 5 to 1 for reduced resource allocation.
- Update image tag in values.yaml to 0.4.5 for the latest version.
- Modify GITHUB_BRANCH environment variable for improved async handling.
- Add logging statements in OPC class to track server initialization and connection status.
- Expanded the .env file with configurations for MongoDB, Postgres, MlFlow, and Temporal.
- Refactored OPC class methods to be asynchronous, including init_opc, write_data, manage_output_tags, and shutdown.
- Updated the worker to initialize OPC asynchronously and adjusted shutdown handling for activities.
Update requirements and modify MLFlow activity
- Bump sientia-mlops-library version in requirements.txt from 0.38.5 to 0.38.8.
- Comment out the reset_index call in the MLFlow activity to prevent unintended data manipulation.
Update Gates activity and format_and_export_prediction workflow to use DATETIME_FORMAT_WITH_TZ for consistent timestamp handling
- Modified Gates activity to correctly handle the maximum timestamp without formatting it prematurely.
- Updated format_and_export_prediction workflow to utilize DATETIME_FORMAT_WITH_TZ for timestamp conversion.
- Enhanced tests to ensure timestamp conversion is applied consistently across workflows.
Update requirements and refactor Gates activity for improved datetime handling
- Update sientia-dataops-library reference in requirements.txt to version 1.4.3.
- Refactor datetime handling in Gates activity to use a consistent format with timezone support, replacing direct datetime calls with a centralized now() function and DATETIME_FORMAT_WITH_TZ constant.
Refactor logging in Gates and MLFlow activities to use info level for key operations
- Updated logging statements in the Gates class to replace debug logs with info logs for input and output gate operations, enhancing visibility.
- Modified MLFlow class to use info logs for data transformation and prediction processes, improving clarity in the logging output.
- Adjusted OPC class to return the count of successfully written tags, providing better insight into data writing operations.
Enhance logging in MLFlow activity to include formatted JSON response data
- Added JSON import for better formatting of response data in debug logs.
- Updated logging statements to output transformed and prediction response data as pretty-printed JSON, improving readability and debugging context.
Update dependencies and refactor logging imports for observability
- Updated the sientia-dataops-library dependency version to 1.4.0 in requirements.txt.
- Changed image tag in values.yaml from 0.3.2 to 0.4.1.
- Refactored logging imports across multiple files to use the new observability module instead of the temporal.utils.logger.
- Updated retry policy imports in workflow files to reflect the new module structure.
Implement server validation and output tag management in OPC class
- Added a new method to validate the existence of OPC servers before writing data, improving error handling.
- Introduced a method to manage writing of prediction and confidence tags, streamlining the data writing process.
- Refactored the write_opc_data method to utilize the new validation and management methods for better code organization and clarity.
Refactor OPC data writing and repository initialization
- Simplified success tracking logic in the OPC class for writing prediction and confidence data.
- Removed unused pod_id attribute from OpcRepository initialization.
- Updated test cases to include pod_id for improved metrics tracking during data writing operations.
Refactor OPC data writing to improve success tracking
- Updated the OPC class to store the success status of data writing operations for both prediction and confidence tags.
- Added logging for successful and failed writes to the OPC server, enhancing traceability of data operations.
Implement OPC writing metrics and enhance OPC class initialization
- Added metrics for counting predictions written to the OPC server and monitoring their response times.
- Enhanced the OPC class initialization to include the pod ID for better tracking.
- Updated the write method to increment the prediction count and observe response times.
Enhance BaseActivity initialization across multiple activities to include error counter
- Updated the initialization of the BaseActivity in Gates, MLFlow, and OPC classes to set the error counter to True, improving error tracking and handling capabilities.
Update dependencies and configuration for improved functionality
- Updated sientia-dataops-library version from 1.3.7 to 1.3.8 in requirements.txt.
- Changed GITHUB_BRANCH in values.yaml to reflect new testing focus: SIENTIAPDE-1169.
- Modified notification level in gates.py from WARNING to ERROR for better error handling.
pdate metrics tracking to include response time histogram
- Changed PREDICTION_RESPONSE_TIME_MONITOR from Gauge to Histogram for better response time analysis.
- Updated response time observation method in gates.py to utilize the new Histogram functionality.
SIENTIAPDE-1174 Update replica count and enhance logging in MLFlow
- Changed replicaCount in values.yaml from 3 to 1 for reduced resource usage.
- Added debug logging for prediction response data in MLFlow to improve traceability.
Update dependencies, modify replica count, and implement metrics tracking
- Updated sientia-dataops-library version from 1.3.5 to 1.3.7 in requirements.txt.
- Changed replicaCount in values.yaml from 5 to 3 and incremented image tag from 0.2.7 to 0.3.1.
- Added Prometheus metrics tracking in gates.py and worker.py, including a new write_metrics method.
- Configured Prometheus service and ServiceMonitor in values.yaml for metrics collection.
Update GITHUB_BRANCH in values.yaml and enhance error notification handling in MLFlow
- Changed GITHUB_BRANCH in values.yaml to reflect the new pipeline for alerts orchestration.
- Added NotificationLevel.ERROR to error notifications in MLFlow for retraining and production model updates.
- Updated tests to verify error notification levels for model management workflows.
Update dependencies and enhance ML model retraining functionality
- Updated sientia-dataops-library version in requirements.txt from 1.3.3 to 1.3.4.
- Incremented image tag in values.yaml from 0.2.4 to 0.2.5 and added a new environment variable MONGODB_TTL_INDEX_HOURS.
- Introduced new methods in MLFlowRepository for model retraining and production model updates, including error handling and logging.
- Added retrain_model and update_production_model activities in mlflow.py to support model management workflows.
- Modified MongoDB connection settings in connectors_config.py for improved security and configuration flexibility.
Update GITHUB_BRANCH in values.yaml to reflect new stress testing branch and modify OPC error handling to set success flag instead of continuing on error.
Enhance OPC error handling by adding detailed notifications for missing servers during write operations. The notification includes the list of available OPC servers for better debugging.
Refactor OPC connection handling to improve error notifications and update GITHUB_BRANCH in values.yaml for MongoDB integration. Change requirements.txt to point to local dataops library path.
Update notification handler references to use CoreNotificationHandler and add MongoDB configuration to connectors. Update GITHUB_BRANCH in values.yaml for optimization tasks.
Refactor notification handling in Gates and OPC activities to use send_notification method with metadata integration, enhancing error reporting and traceability.
Refactor Gates and OpcRepository for improved logging and error handling. Removed debug statement in Gates and enhanced disconnect method in OpcRepository to log disconnection status and handle exceptions.
Refactor OPC activity and OpcRepository to include metadata parameter in write_data methods, improving error handling and logging capabilities for better traceability.
