QTZPOC-13: Centralize MLflow joblib fallback logic, ensuring consistent model version resolution by reusing the initial run ID, improving logging, and updating dependencies.
QTZPOC-13: Introduce joblib fallback for MLflow model loading
Enhances model loading resilience by attempting to deserialize models directly with joblib if MLflow's native flavor loaders fail. This mechanism handles cases where .pkl files are present in the artifact.
Adds joblib as a core dependency and includes new end-to-end and unit tests to validate fallback behavior, covering success, no-fallback, and double-failure scenarios. Also updates .gitignore for .claude/ files and incorporates minor refactorings and linting adjustments across E2E tests and utility functions.
Update dependencies and refactor input filter handling for consistency
- Updated sientia-dataops-library dependency version from 1.10.3 to 1.10.4 in requirements.txt.
- Refactored input filter handling in the Gates class to read policy and config keys in a case-insensitive manner.
- Updated test cases to ensure consistency in filter key naming conventions across various scenarios.
Implement MinIO Offload and Retention Features
- Added configuration options for MinIO retention hours and offload threshold in README.
- Introduced MinIO payload offloading for large DataFrame-derived payloads, storing them as parquet files.
- Updated activities to utilize MinIO for data loading and cleanup, including new methods for offloading and retention management.
- Refactored existing activities to integrate MinIO functionality, ensuring compatibility with previous workflows.
- Removed the legacy MinioRepository class, consolidating MinIO operations under a new manager structure.
- Updated requirements to use the latest version of the sientia-dataops-library.
Update sientia-dataops-library dependency version from 1.8.1 to 1.8.2 in requirements.txt and modify API class to accept a list of response data for PI Web API integration.
Update requirements.txt and values.yaml for dependency and image tag adjustments
- Replaced local path with the correct GitHub URL for sientia-dataops-library in requirements.txt.
- Downgraded the image tag from "1.1.2" to "1.0.1" in values.yaml for consistency with deployment requirements.
SIENTIAPDE-1478
Implement PI Web API response processing and metrics tracking
- Added a new method in the API class to process responses from the PI Web API, validating tag writes and emitting metrics for success and errors.
- Enhanced error handling for missing WebIds and tag names in responses, with appropriate logging and notifications.
- Updated tests to cover various scenarios for processing PI Web API responses, ensuring robust functionality and metrics emission.
- Refactored existing methods to integrate the new response processing logic, improving overall code clarity and maintainability.
Enhance Activities and Prediction Workflows with PI Web API Integration
- Updated the Activities class to include API integration, allowing for configuration of PI Web API parameters.
- Modified prediction workflows to support output configuration for PI Web API, enabling data writing to the API.
- Refactored connectors_config.py by removing unused PostgreSQL and MongoDB configuration functions.
- Added tests to validate the new PI Web API functionality in activities and workflows, ensuring robust integration and functionality.
Update sientia-mlops-library dependency to version 0.40.6 and refactor get_prediction_data method calls for improved readability in model_repository.py and test_model_repository.py.
Update sientia-mlops-library dependency to version 0.40.5 and fix SQL interval formatting in Drift and SimpleMetrics workflows for improved query accuracy.
Update sientia-dataops-library dependency version to 1.6.1 and refactor return types in Gates class methods for improved type clarity. Changed return types from dict[str, Any] to dict for better compatibility with downstream systems.
Update dependencies and refactor data handling in various modules
- Updated sientia-dataops-library dependency version from 1.5.3 to 1.5.4 in requirements files.
- Updated sientia-mlops-library dependency version from 0.39.0 to 0.40.2 in requirements files.
- Refactored return types in Gates, MLFlow, and ModelMetrics classes to return dictionaries instead of DataFrames for improved compatibility with downstream systems.
- Removed the temporal_codec module as it is no longer needed for DataFrame serialization.
- Adjusted data handling in the Drift workflow to ensure proper data structure is maintained.
Update requirements and enhance metrics and data handling
- Updated the sientia-dataops-library dependency version in requirements.txt to 1.5.3.
- Added new metrics for model analysis, including lag, count, and error count in metrics.py.
- Implemented a new method for formatting transformed data in gates.py.
- Enhanced MLFlowRepository with methods to load artifact dataframes and calculate model metrics, including drift and performance metrics.
- Updated the prediction process to handle transformed data and ensure proper execution of related activities in format_and_export_prediction.py and prediction_process.py.
Refactor monitoring and metrics integration across various components
- Removed coverage options from `pyproject.toml`.
- Updated prediction metrics in `README.md` to replace `pipeline_name` with `workflow_name`.
- Upgraded `sientia-dataops-library` dependency version in `requirements-light.txt` and `requirements.txt`.
- Enhanced metrics handling in `laborious` activities, including `Activities`, `Gates`, `MLFlow`, and `OPC`, to utilize a new `MetricsController`.
- Refactored metric emission methods to improve clarity and consistency across the codebase.
- Updated tests to reflect changes in metrics handling and ensure proper functionality.
Update requirements and simplify CI workflow for quality checks
- Added new dependencies: kaleido, hyperopt, and shap to requirements.txt.
- Refactored the quality gate workflow to use a template for improved maintainability and reduced complexity.
Update dependencies and enhance OpcRepository functionality
- Updated sientia-dataops-library version to 1.4.7 in requirements files.
- Modified GITHUB_BRANCH in values.yaml for improved pipeline management.
- Refactored OpcRepository class to inherit from BaseActivity, adding enhanced logging and error handling during disconnection.
- Implemented a disconnection fallback mechanism to ensure graceful handling of OPC server disconnections.
Enhance MinIO integration and update environment configurations
- Added MinIO configuration parameters to .env.example and values.yaml for improved storage management.
- Updated requirements.txt to include necessary libraries for MinIO support.
- Refactored Activities class to utilize Storage for MinIO interactions.
- Enhanced MLFlow class to integrate MinIO for data retrieval during model retraining.
- Introduced build_minio_config function to streamline MinIO configuration setup.
- Updated minimal_retrain workflow to support data storage in MinIO.
Update requirements and enhance logging in Gates and MLFlow activities
- Updated the sientia-dataops-library and sientia-mlops-library dependencies in requirements.txt to the latest versions.
- Improved debug logging in the Gates activity to display a sample of input data and filters, enhancing clarity and reducing output size.
- Refactored MLFlow activity logging to utilize the create_sample_dict function for better visualization of nested data structures in logs.
SIENTIAPDE-1214: Refactor MLFlow and model repository methods to use model_config dictionary
- Updated MLFlow class methods to accept model_config instead of model_retention for improved flexibility.
- Modified model_repository methods to handle model_config, extracting necessary parameters for transformation and prediction.
- Adjusted predictions_batch and prediction_process workflows to utilize model_config for better configuration management.
- Commented out the previous sientia-mlops-library dependency in requirements.txt for clarity.
Update requirements and modify MLFlow activity
- Bump sientia-mlops-library version in requirements.txt from 0.38.5 to 0.38.8.
- Comment out the reset_index call in the MLFlow activity to prevent unintended data manipulation.
Update dependencies and enhance datetime handling
- Bump sientia-dataops-library version in requirements.txt to 1.4.4.
- Update image tag in values.yaml to 0.4.4.
- Add 'datetime_columns' to metadata in MinimalRetrain and PredictionsBatch workflows for improved data handling.
Update requirements and refactor Gates activity for improved datetime handling
- Update sientia-dataops-library reference in requirements.txt to version 1.4.3.
- Refactor datetime handling in Gates activity to use a consistent format with timezone support, replacing direct datetime calls with a centralized now() function and DATETIME_FORMAT_WITH_TZ constant.
Update requirements and values for SIENTIAPDE-1193
- Bump sientia-dataops-library version from 1.4.1 to 1.4.2 in requirements.txt.
- Change GITHUB_BRANCH in values.yaml to reflect the new task SIENTIAPDE-1193 regarding datetime writing in temporal.
Update dependencies in requirements.txt and increment image tag in values.yaml
- Updated sientia-dataops-library dependency version from 1.4.0 to 1.4.1 in requirements.txt.
- Incremented image tag in values.yaml from 0.4.1 to 0.4.2.
Update dependencies and refactor logging imports for observability
- Updated the sientia-dataops-library dependency version to 1.4.0 in requirements.txt.
- Changed image tag in values.yaml from 0.3.2 to 0.4.1.
- Refactored logging imports across multiple files to use the new observability module instead of the temporal.utils.logger.
- Updated retry policy imports in workflow files to reflect the new module structure.
Update dependencies and configuration for improved functionality
- Updated sientia-dataops-library version from 1.3.7 to 1.3.8 in requirements.txt.
- Changed GITHUB_BRANCH in values.yaml to reflect new testing focus: SIENTIAPDE-1169.
- Modified notification level in gates.py from WARNING to ERROR for better error handling.
Update dependencies, modify replica count, and implement metrics tracking
- Updated sientia-dataops-library version from 1.3.5 to 1.3.7 in requirements.txt.
- Changed replicaCount in values.yaml from 5 to 3 and incremented image tag from 0.2.7 to 0.3.1.
- Added Prometheus metrics tracking in gates.py and worker.py, including a new write_metrics method.
- Configured Prometheus service and ServiceMonitor in values.yaml for metrics collection.
Update dependencies in requirements.txt and increment image tag in values.yaml
- Updated sientia-dataops-library version from 1.3.4 to 1.3.5 in requirements.txt.
- Incremented image tag in values.yaml from 0.2.5 to 0.2.7.
Update dependencies and enhance ML model retraining functionality
- Updated sientia-dataops-library version in requirements.txt from 1.3.3 to 1.3.4.
- Incremented image tag in values.yaml from 0.2.4 to 0.2.5 and added a new environment variable MONGODB_TTL_INDEX_HOURS.
- Introduced new methods in MLFlowRepository for model retraining and production model updates, including error handling and logging.
- Added retrain_model and update_production_model activities in mlflow.py to support model management workflows.
- Modified MongoDB connection settings in connectors_config.py for improved security and configuration flexibility.
Refactor OPC connection handling to improve error notifications and update GITHUB_BRANCH in values.yaml for MongoDB integration. Change requirements.txt to point to local dataops library path.
