Refactor MLFlow logging to enhance data output clarity
- Updated debug logging to use data.to_string() for processed input data, improving readability.
- Modified raw response data logging to format the output as a string, ensuring consistent logging format.
Update values.yaml and MLFlow logging for courier integration
- Changed the image repository to 'sientia-module-courier' and updated the image tag to '0.0.1'.
- Modified environment variables for GITHUB_BRANCH and MLFLOW_PASSWORD to reflect new configurations.
- Enhanced MLFlow logging to include additional debug statements for raw response data and added a check for empty DataFrames.
Refactor MLFlow logging to improve data output clarity
- Updated the debug logging to directly capture the output of data.to_csv, enhancing traceability of processed input data.
- Removed redundant debug statements for raw response data to streamline logging and focus on essential information.
SIENTIAPDE-1214: Enhance MLFlow and tests with datetime index handling and logging improvements
- Added a new method in MLFlow to detect and parse datetime indices in DataFrames, ensuring proper format and raising errors for invalid types.
- Updated prediction workflows to utilize the new datetime index handling, improving data integrity during transformations.
- Enhanced logging in model_repository to include detailed data outputs for better traceability.
- Adjusted timeout settings in prediction workflows for improved execution time management.
- Updated tests.ipynb to include additional checks for index types and outputs for better validation of functionality.
SIENTIAPDE-1214: Update requirements.txt to clarify dependencies and improve project setup
- Commented out the previous sientia-mlops-library dependency for better clarity.
- Ensured that the requirements.txt reflects the current state of dependencies for easier management.
SIENTIAPDE-1214: Refactor MLFlow and model repository methods to use model_config dictionary
- Updated MLFlow class methods to accept model_config instead of model_retention for improved flexibility.
- Modified model_repository methods to handle model_config, extracting necessary parameters for transformation and prediction.
- Adjusted predictions_batch and prediction_process workflows to utilize model_config for better configuration management.
- Commented out the previous sientia-mlops-library dependency in requirements.txt for clarity.
Update .gitignore to include 'tmp/' directory and ensure '.env' is listed
- Added 'tmp/' to the .gitignore file to prevent temporary files from being tracked.
- Confirmed that '.env' is included to avoid committing sensitive environment variables.
Remove deprecated files and configurations from transformer_pyfunc module
- Deleted conda.yaml, MLmodel, python_env.yaml, requirements.txt, and various utility scripts related to data processing and model handling.
- Removed binary files including python_model.pkl and training_transformer.pkl to clean up the artifacts directory.
- This cleanup is part of the effort to streamline the transformer_pyfunc module and eliminate unused components.
Add placeholder class 'Any' in test_model_repository.py and update invalid_cases to use it
- Introduced a new placeholder class 'Any' to be used in test cases.
- Updated the 'invalid_cases' list to replace integer keys with instances of the 'Any' class, enhancing test coverage for key types.
Refactor OPC and model repository for improved functionality and clarity
- Updated OPC server logging to handle missing prediction and confidence tags gracefully.
- Corrected documentation for OPC reconnection interval from milliseconds to seconds.
- Enhanced MLFlowRepository with new methods for model retrieval, caching, and transformation, improving model management and retraining workflows.
Refactor model configuration handling in MLFlow and workflows
- Replaced 'model_retention' with 'model_config' to encapsulate retention settings and improve consistency across various components.
- Updated test cases to reflect changes in argument structure, ensuring compatibility with the new model configuration format.
- Added 'prediction_store_policy' to input data handling in workflows for enhanced configuration management.
Enhance MLFlow data handling by adding timestamp column and improving debug logging
- Added a 'timestamp' column to the input data, converting the index to a datetime format for better tracking of predictions.
- Improved debug logging to provide clearer context by including the input data preview in the log output.
Update MLFlow class to pass logger instance directly to MLFlowRepository
- Modified the initialization of MLFlowRepository in the MLFlow class to pass the logger instance directly, improving logging capabilities and consistency across the application.
SIENTIAPDE-1222 Update MLFlow methods to include metadata parameter
- Modified the transform and predict method calls in the MLFlow class to include a new 'metadata' parameter, enhancing the functionality and data handling capabilities of the model monitoring repository.
Update test cases in model repository and prediction process
- Replaced string data with MagicMock in test_transform_success and test_transform_error to improve test isolation.
- Updated the predict method calls in test_predict_success and test_predict_error to reflect changes in argument structure.
- Added 'prediction_store_policy' to the test_run configuration in test_prediction_process for consistency with recent updates.
Refactor datetime index handling in MLFlow and MLFlowRepository
- Moved the detect_and_parse_datetime_index method from MLFlow to MLFlowRepository for better organization and reusability.
- Updated the method to include enhanced logging and error handling for invalid datetime formats.
- Adjusted the transform method in MLFlowRepository to utilize the new datetime index parsing logic.
- Added unit tests for both valid and invalid datetime index cases to ensure robustness.
Update README.md to reflect new features and configuration changes
- Added details about two dedicated task queues: `predictions_batch-queue` and `minimal_retrain-queue`.
- Enhanced descriptions of activities and workflows, including multiple inheritance patterns and configurable MLFlow model serving.
- Updated monitoring metrics section to include new labels and metrics for prediction and OPC export operations.
- Revised configuration section with updated default values and added new environment variables for Kubernetes pod identification.
- Improved clarity in the Predictions Batch Workflow configuration example, including structured input filters and updated retention policies.
Update model configuration key for compression in MLFlowRepository
- Changed the key for compression from 'compressed' to 'is_compressed' in model configuration handling to ensure consistency with updated standards.
Update model configuration keys in MLFlowRepository for consistency
- Changed 'model_retention' to 'retention_minutes' and 'is_compressed' to 'compressed' in model configuration handling, ensuring alignment with updated configuration standards.
Enhance debug logging in MLFlowRepository to improve data traceability
- Added a debug logging statement to capture received data for model prediction, improving visibility into input data.
- Changed an existing info logging statement to debug level for consistency, ensuring all relevant data is logged at the appropriate level.
Update prediction_store_policy handling in workflows
- Added 'prediction_store_policy' to the input data handling in PredictionsBatch, ensuring a default value of 'lts:1' is used when not provided.
- Modified FormatAndExportPrediction to directly use 'prediction_store_policy' from input_data, removing the default fallback.
- Updated PredictionProcess to include 'prediction_store_policy' in the output data structure, ensuring consistency across workflows.
Enhance MLFlow debug logging to limit output size
- Updated debug logging statements in the MLFlow activity to include a maximum of 5 items and a depth of 5 for the sample dictionary, improving readability and reducing log clutter for raw and transformed response data.
Refactor debug logging in Gates activity for improved readability
- Reformatted the debug logging statement for input data in the Gates activity to enhance clarity and maintain consistency with previous logging improvements.
Refactor MLFlow debug logging for improved readability
- Reformatted debug logging statements in the MLFlow activity to enhance clarity and consistency.
- Ensured that the output of raw and transformed response data is presented in a more readable format, maintaining the use of create_sample_dict for better visualization.
Update image tag in values.yaml and enhance debug logging in Gates and MLFlow activities
- Updated the image tag in values.yaml from '0.0.1' to '0.0.2'.
- Improved debug logging in the Gates activity to format input data and filters for better readability.
- Enhanced MLFlow activity logging to include formatted output for raw and transformed response data, ensuring consistent logging format.
Update requirements and enhance logging in Gates and MLFlow activities
- Updated the sientia-dataops-library and sientia-mlops-library dependencies in requirements.txt to the latest versions.
- Improved debug logging in the Gates activity to display a sample of input data and filters, enhancing clarity and reducing output size.
- Refactored MLFlow activity logging to utilize the create_sample_dict function for better visualization of nested data structures in logs.
SIENTIAPDE-1222
Enhance MLFlow logging with sample dictionary for response data
- Introduced a new method to create a sample dictionary for debugging, allowing for better visualization of nested data structures in logs.
- Updated debug logging to utilize the new sampling method for raw and transformed response data, improving clarity and reducing output size.
- Adjusted logging for processed input data to display only the first few rows, enhancing readability.
Refactor MLFlow logging to enhance data output clarity
- Updated debug logging to use data.to_string() for processed input data, improving readability.
- Modified raw response data logging to format the output as a string, ensuring consistent logging format.
Update values.yaml and MLFlow logging for courier integration
- Changed the image repository to 'sientia-module-courier' and updated the image tag to '0.0.1'.
- Modified environment variables for GITHUB_BRANCH and MLFLOW_PASSWORD to reflect new configurations.
- Enhanced MLFlow logging to include additional debug statements for raw response data and added a check for empty DataFrames.
Refactor MLFlow logging to improve data output clarity
- Updated the debug logging to directly capture the output of data.to_csv, enhancing traceability of processed input data.
- Removed redundant debug statements for raw response data to streamline logging and focus on essential information.
SIENTIAPDE-1214: Enhance MLFlow and tests with datetime index handling and logging improvements
- Added a new method in MLFlow to detect and parse datetime indices in DataFrames, ensuring proper format and raising errors for invalid types.
- Updated prediction workflows to utilize the new datetime index handling, improving data integrity during transformations.
- Enhanced logging in model_repository to include detailed data outputs for better traceability.
- Adjusted timeout settings in prediction workflows for improved execution time management.
- Updated tests.ipynb to include additional checks for index types and outputs for better validation of functionality.
SIENTIAPDE-1214: Update requirements.txt to clarify dependencies and improve project setup
- Commented out the previous sientia-mlops-library dependency for better clarity.
- Ensured that the requirements.txt reflects the current state of dependencies for easier management.
SIENTIAPDE-1214: Refactor MLFlow and model repository methods to use model_config dictionary
- Updated MLFlow class methods to accept model_config instead of model_retention for improved flexibility.
- Modified model_repository methods to handle model_config, extracting necessary parameters for transformation and prediction.
- Adjusted predictions_batch and prediction_process workflows to utilize model_config for better configuration management.
- Commented out the previous sientia-mlops-library dependency in requirements.txt for clarity.
Update tests.ipynb and gates.py for enhanced functionality and clarity
- Adjusted execution count in tests.ipynb for reproducibility.
- Modified DataFrame creation in tests.ipynb to dynamically set size and update timestamp frequency.
- Added a new filter function for empty data in gates.py to improve data handling.
Refactor test data structure in gates.py and model_repository.py for improved clarity and consistency
- Updated test cases in test_gates.py to use dictionaries for prediction and response_time, associating values with timestamps.
- Modified test_predict_success in test_model_repository.py to create a DataFrame with named indices for better readability in assertions.
Enhance tests.ipynb and gates.py for improved logging and output clarity
- Set execution count to null in tests.ipynb for reproducibility.
- Added stdout output to tests.ipynb for better visibility of results.
- Updated DataFrame display logic in tests.ipynb to include additional print statements for length and size.
- Improved debug logging in gates.py to log prediction data as a string for better traceability.
- Changed info log in gates.py to use len() instead of size for consistency.
Update tests.ipynb and values.yaml for improved functionality and versioning
- Updated execution count in tests.ipynb for reproducibility.
- Modified DataFrame creation in tests.ipynb to include a timestamp column and save to CSV.
- Changed image tag in values.yaml from "0.4.4" to "0.4.5" for versioning.
- Updated GITHUB_BRANCH in values.yaml to reflect the latest branch adjustments.
Refactor connectors_config.py and conditional_filters.py for improved configuration management and data filtering
- Updated PostgreSQL and MLFlow configuration functions to enhance default values and environment variable handling.
- Simplified OPC server configuration logic and improved MongoDB connection string construction.
- Refactored conditional filters to streamline null value checks and empty data validation, removing unnecessary comments and examples for clarity.
- Removed extensive module docstrings to enhance code readability.
Update README.md to enhance installation instructions and refactor MLFlow filters
- Added steps for installing GitHub CLI and authenticating with GitHub.
- Updated the `api_error_filter` and `nan_values_filter` functions to improve parameter handling and streamline logic.
