Refactor MLFlow debug logging for improved readability
- Reformatted debug logging statements in the MLFlow activity to enhance clarity and consistency.
- Ensured that the output of raw and transformed response data is presented in a more readable format, maintaining the use of create_sample_dict for better visualization.
Update image tag in values.yaml and enhance debug logging in Gates and MLFlow activities
- Updated the image tag in values.yaml from '0.0.1' to '0.0.2'.
- Improved debug logging in the Gates activity to format input data and filters for better readability.
- Enhanced MLFlow activity logging to include formatted output for raw and transformed response data, ensuring consistent logging format.
Update requirements and enhance logging in Gates and MLFlow activities
- Updated the sientia-dataops-library and sientia-mlops-library dependencies in requirements.txt to the latest versions.
- Improved debug logging in the Gates activity to display a sample of input data and filters, enhancing clarity and reducing output size.
- Refactored MLFlow activity logging to utilize the create_sample_dict function for better visualization of nested data structures in logs.
SIENTIAPDE-1222
Enhance MLFlow logging with sample dictionary for response data
- Introduced a new method to create a sample dictionary for debugging, allowing for better visualization of nested data structures in logs.
- Updated debug logging to utilize the new sampling method for raw and transformed response data, improving clarity and reducing output size.
- Adjusted logging for processed input data to display only the first few rows, enhancing readability.
Refactor MLFlow logging to enhance data output clarity
- Updated debug logging to use data.to_string() for processed input data, improving readability.
- Modified raw response data logging to format the output as a string, ensuring consistent logging format.
Update values.yaml and MLFlow logging for courier integration
- Changed the image repository to 'sientia-module-courier' and updated the image tag to '0.0.1'.
- Modified environment variables for GITHUB_BRANCH and MLFLOW_PASSWORD to reflect new configurations.
- Enhanced MLFlow logging to include additional debug statements for raw response data and added a check for empty DataFrames.
Refactor MLFlow logging to improve data output clarity
- Updated the debug logging to directly capture the output of data.to_csv, enhancing traceability of processed input data.
- Removed redundant debug statements for raw response data to streamline logging and focus on essential information.
SIENTIAPDE-1214: Enhance MLFlow and tests with datetime index handling and logging improvements
- Added a new method in MLFlow to detect and parse datetime indices in DataFrames, ensuring proper format and raising errors for invalid types.
- Updated prediction workflows to utilize the new datetime index handling, improving data integrity during transformations.
- Enhanced logging in model_repository to include detailed data outputs for better traceability.
- Adjusted timeout settings in prediction workflows for improved execution time management.
- Updated tests.ipynb to include additional checks for index types and outputs for better validation of functionality.
SIENTIAPDE-1214: Refactor MLFlow and model repository methods to use model_config dictionary
- Updated MLFlow class methods to accept model_config instead of model_retention for improved flexibility.
- Modified model_repository methods to handle model_config, extracting necessary parameters for transformation and prediction.
- Adjusted predictions_batch and prediction_process workflows to utilize model_config for better configuration management.
- Commented out the previous sientia-mlops-library dependency in requirements.txt for clarity.
Remove Docker configuration files and refactor project structure
- Deleted docker-compose.yml and Dockerfile as part of the project restructuring.
- Updated README.md to reflect changes in project setup and configuration.
- Introduced a new __init__.py file in the laborious package to provide an overview of the system.
- Enhanced documentation across various modules, including metrics, activities, and workflows, to improve clarity and usability.
- Added comprehensive docstrings and comments to key classes and methods for better maintainability.
Update requirements and modify MLFlow activity
- Bump sientia-mlops-library version in requirements.txt from 0.38.5 to 0.38.8.
- Comment out the reset_index call in the MLFlow activity to prevent unintended data manipulation.
Refactor logging in Gates and MLFlow activities to use info level for key operations
- Updated logging statements in the Gates class to replace debug logs with info logs for input and output gate operations, enhancing visibility.
- Modified MLFlow class to use info logs for data transformation and prediction processes, improving clarity in the logging output.
- Adjusted OPC class to return the count of successfully written tags, providing better insight into data writing operations.
Enhance logging in MLFlow activity to include formatted JSON response data
- Added JSON import for better formatting of response data in debug logs.
- Updated logging statements to output transformed and prediction response data as pretty-printed JSON, improving readability and debugging context.
Update dependencies and refactor logging imports for observability
- Updated the sientia-dataops-library dependency version to 1.4.0 in requirements.txt.
- Changed image tag in values.yaml from 0.3.2 to 0.4.1.
- Refactored logging imports across multiple files to use the new observability module instead of the temporal.utils.logger.
- Updated retry policy imports in workflow files to reflect the new module structure.
Enhance BaseActivity initialization across multiple activities to include error counter
- Updated the initialization of the BaseActivity in Gates, MLFlow, and OPC classes to set the error counter to True, improving error tracking and handling capabilities.
SIENTIAPDE-1174 Update replica count and enhance logging in MLFlow
- Changed replicaCount in values.yaml from 3 to 1 for reduced resource usage.
- Added debug logging for prediction response data in MLFlow to improve traceability.
Update GITHUB_BRANCH in values.yaml and enhance error notification handling in MLFlow
- Changed GITHUB_BRANCH in values.yaml to reflect the new pipeline for alerts orchestration.
- Added NotificationLevel.ERROR to error notifications in MLFlow for retraining and production model updates.
- Updated tests to verify error notification levels for model management workflows.
Update dependencies and enhance ML model retraining functionality
- Updated sientia-dataops-library version in requirements.txt from 1.3.3 to 1.3.4.
- Incremented image tag in values.yaml from 0.2.4 to 0.2.5 and added a new environment variable MONGODB_TTL_INDEX_HOURS.
- Introduced new methods in MLFlowRepository for model retraining and production model updates, including error handling and logging.
- Added retrain_model and update_production_model activities in mlflow.py to support model management workflows.
- Modified MongoDB connection settings in connectors_config.py for improved security and configuration flexibility.
Update notification handler references to use CoreNotificationHandler and add MongoDB configuration to connectors. Update GITHUB_BRANCH in values.yaml for optimization tasks.
Refactor logging in MLFlow and Gates activities; remove print statement in FormatAndExportPrediction; update data handling in PredictionProcess; delete unused redis-feeder script.
Refactor and enhance the laborious workflow and utilities
- Removed outdated test file `test_predictions_batch.py` from workflows.
- Added `input_sample.json` for standardized input configuration.
- Introduced `connectors_config.py` to manage database and service configurations.
- Implemented a logging utility in `logger.py` for consistent logging across the application.
- Created `policies.py` to define retry policies for workflows.
- Developed comprehensive tests for `MLFlowRepository` in `test_model_repository.py`.
- Added extensive tests for `OpcRepository` in `test_opc_repository.py`.
- Updated `test_predictions_batch.py` to reflect new workflow structure and testing methodology.
Implement new get_last_timestamp method in Gates class, refactor MLFlow activity methods to return only transformed data, and update PredictionsBatch and PredictionProcess workflows to utilize Activities module. Add detailed docstrings for new methods and enhance test coverage for get_last_timestamp functionality.
Refactor activity methods and update requirements.txt to enhance functionality and remove deprecated filters. Added detailed docstrings for clarity and improved error handling in data processing workflows.
