SIENTIAPDE-1646 Merge origin/main into release/SIENTIAPDE-1646
Bring main up to date while keeping this branch's sync execution, PluginStore
runtime and sientia_model (SientiaMLflowRepository/SientiaModel) structure.
Resolution decisions:
- OPC: keep the sync asyncua.sync implementation; port main's server_uri
validation, e2e race fix (lock held until released) and the new unit tests
converted to sync.
- Model import workflow (.sientia bundles), joblib/transform flavors and the
local MLFlowRepository (model_repository.py) are dropped, with their tests,
docs, metrics, env vars and connectors config.
- Worker: drift/simple_metrics stay pinned to the 'core' runtime; port main's
HOSTNAME pod id, logger-based prometheus startup and sys.exit after finally.
- values.yaml removed, following main (SIENTIAPDE-1945).
- Deps: keep sientia_model; take main's fsspec/s3fs/botocore/boto3 pins and
sientia_do>=1.12.2; drop import-only deps (pynacl, cryptography, pymongo,
python-dotenv, pycaret, joblib, gcsfs).
- e2e: keep sientia_data schema and scenario input files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QTZPOC-20: Enable code_paths for sklearn models; improve worker shutdown handling
The MLFlowRepository now resolves and forwards code_paths to mlflow.sklearn.log_model, ensuring custom code dependencies for sklearn models are preserved and correctly reloaded. A new integration test validates this end-to-end functionality. Additionally, worker exception handling is refined to distinguish between deliberate shutdowns (SystemExit, KeyboardInterrupt) and unhandled exceptions, preventing unnecessary error logging for expected exits.
QTZPOC-20: Refactor ModelImport status engine for lazy initialization, add byte-counted size validation to bundle extraction, optimize MLflow run ID lookups, and ensure workflow cancellation propagates correctly.
QTZPOC-20: Refactor model import workflow, introduce atomic model ID generation, and clarify MinIO expiry.
Fuses bundle download, decryption, MLflow experiment creation, and artifact upload into a single activity to prevent cross-worker issues. Implements an atomic counter for MongoDB model IDs, replacing the max(id) + 1 scan to avoid collisions during concurrent imports. Adds comprehensive documentation for MinIO object expiry mechanisms and refines lifecycle rule application. Improves error handling with clearer responsibility classification and centralizes MLflow repository injection.
QTZPOC-20: Enhance system robustness and test coverage across bundle processing, activity error handling, and worker operations.
* Strengthen bundle reader validation with new tests for archive integrity, security vulnerabilities (e.g., path traversal, duplicate entries, special file types), and I/O error resilience.
* Improve error handling and status propagation in PI Web API, MLflow, and OPC activities, including partial write scenarios and OPC session management.
* Add comprehensive tests for worker startup, configuration, graceful shutdown, and Prometheus metrics server initialization.
* Introduce a new utility for intelligent DataFrame debugging, summarizing large dataframes for concise logging, and refine exception chain summarization.
* Update code coverage configuration to exclude @overload decorator lines.
QTZPOC-20: Refactor model repository tests for improved assertion logic and exception handling, including correcting pytest.raises exception access and refining mock verifications.
QTZPOC-20: Prioritize code/ directory download and sys.path injection before model loading, enabling MLflow flavors to stream models directly from URIs and simplifying joblib artifact handling.
QTZPOC-20: Unify artifact code directory handling for all model flavors and standardize joblib pickle path
Implemented a robust mechanism to ensure an artifact's code/ directory is prepended to sys.path before loading for sklearn, pyfunc, and joblib flavors. sklearn and pyfunc now resolve MLflow URIs to local directories first to facilitate this sys.path modification. Standardized the joblib flavor to explicitly read and write model.pkl within the artifact directory. Updated documentation and added comprehensive tests to reflect these changes.
QTZPOC-20: Implement explicit joblib model flavor and enhance model import workflow with extensive E2E tests.
This commit introduces a first-class joblib model flavor for direct loading and saving of Python pickle artifacts, including custom code within the artifact's code/ directory, bypassing MLflow's flavor loaders. The model import workflow's status database configuration is now streamlined to default to the POSTGRES_* server, with explicit IMPORT_STATUS_DB_* overrides.
Comprehensive end-to-end tests have been added for the model import workflow, covering success paths, gate rejections, digest enforcement, provisioning failures, status record integrity, cleanup, and retry/recovery scenarios using a real-clock environment. New E2
QTZPOC-13: Centralize MLflow joblib fallback logic, ensuring consistent model version resolution by reusing the initial run ID, improving logging, and updating dependencies.
QTZPOC-13: Introduce joblib fallback for MLflow model loading
Enhances model loading resilience by attempting to deserialize models directly with joblib if MLflow's native flavor loaders fail. This mechanism handles cases where .pkl files are present in the artifact.
Adds joblib as a core dependency and includes new end-to-end and unit tests to validate fallback behavior, covering success, no-fallback, and double-failure scenarios. Also updates .gitignore for .claude/ files and incorporates minor refactorings and linting adjustments across E2E tests and utility functions.
Enhance MLFlow run ID resolution with error handling for missing and invalid source URIs
- Added checks in `get_model_run_id` method to raise exceptions for models with missing or invalid source URIs.
- Introduced new test cases to validate error handling for these scenarios.
- Updated `requirements-light.txt` to include `mlflow` as a dependency.
Enhance OPC UA communication and metrics tracking
- Updated README.md to include new OPC UA Communication section and detailed metrics for session and write diagnostics.
- Added new metrics in laborious/metrics.py for tracking OPC UA session states and write attempts.
- Refactored OPC activity in laborious/activities/opc.py to handle session errors and improve error reporting.
- Updated e2e tests to cover new scenarios for OPC session/channel errors and reconnect handling.
- Modified .gitignore to include relatorio files and mlruns directory.
- Added ipykernel to requirements-dev.txt for Jupyter notebook support.
Update .gitignore, requirements, and enhance OPC UA error handling
- Added new entries to .gitignore for openspec and cursor directories.
- Updated sientia-dataops-library dependency version in requirements-light.txt from 1.10.4 to 1.12.0.
- Enhanced OPC UA communication by refining reconnect logic and error handling in opc_repository.py, including the introduction of a reconnect flag and improved session management.
- Updated tests to cover new reconnect scenarios and ensure robust error handling for protocol states.
SIENTIAPDE-1646 Enhance OPC integration and E2E testing framework
- Updated .gitignore to exclude '.cursor/*' for better file management.
- Added new marker in pyproject.toml for E2E tests using OPC.
- Introduced async OPC server fixtures and improved connection handling in conftest.py.
- Enhanced error handling in OPC activities and added metrics for session management in metrics.py.
- Updated E2E tests to cover new OPC scenarios, including session errors and reconnect handling.
- Refactored helper functions to improve prediction assertion logic in helpers.py.
- Documented new OPC scenarios in scenarios.md for clarity on expected outcomes.
Enhance OPC UA communication and metrics tracking
- Updated README.md to include new OPC UA Communication section and detailed metrics for session and write diagnostics.
- Added new metrics in laborious/metrics.py for tracking OPC UA session states and write attempts.
- Refactored OPC activity in laborious/activities/opc.py to handle session errors and improve error reporting.
- Updated e2e tests to cover new scenarios for OPC session/channel errors and reconnect handling.
- Modified .gitignore to include relatorio files and mlruns directory.
- Added ipykernel to requirements-dev.txt for Jupyter notebook support.
Refactor OPC and OpcRepository classes to remove logger dependency and streamline logging calls
- Removed logger attribute from OPC and OpcRepository classes, replacing direct logger calls with class methods for logging.
- Updated write_data method signatures to eliminate logger parameter, simplifying the interface.
- Adjusted related tests to reflect changes in method signatures and logging behavior.
SIENTIAPDE-1646 Update dependencies and refactor OPC integration
- Replaced `opcua` with `asyncua` in `requirements-local.txt` and `requirements.txt` to utilize the async capabilities.
- Updated `values.yaml` to change the GitHub branch from `feature/SIENTIAPDE-1646` to `release/SIENTIAPDE-1646`.
- Refactored `opc.py` and `opc_repository.py` to accommodate the new `asyncua` library, ensuring compatibility with the synchronous API.
- Adjusted tests in `test_opc_repository.py` to reflect changes in the client implementation and maintain functionality.
Update README, requirements, and E2E tests for improved configuration and functionality
- Enhanced the README with updated model configuration examples, including the addition of an alias for production.
- Removed the `requirements-light.txt` file and updated `requirements-local.txt` and `requirements.txt` to replace `asyncua` with `opcua`.
- Refactored E2E test scenarios to utilize scenario input files for better maintainability and clarity.
- Improved test coverage for MinIO offload functionality and added new helper functions for loading scenario inputs.
- Updated `values.yaml` to reflect new global configurations and environment variables for the laborious worker.
Enhance environment configuration and update dependencies
- Added new environment variables for PluginStore and MLflow configuration in `.env.example`, including `RUNTIME`, `STORE_BASE_URL`, `STORE_OWNER`, `STORE_REPO`, `STORE_BRANCH`, `STORE_USERNAME`, `STORE_PASSWORD`, `STORE_CACHE_TTL_SECONDS`, `PYPI_SERVER`, `PYPI_USERNAME`, and `PYPI_PASSWORD`.
- Updated `git-requirements-mapping.txt` to reflect changes in repository names.
- Modified `requirements-light.txt` and `requirements.txt` to upgrade `sientia-dataops-library` to version 1.12.0 and `sientia-mlops-library` to version 0.8.1.
- Updated `values.yaml` to include new environment variables for worker runtime and PluginStore configuration.
- Refactored E2E tests to utilize new MLflow repository stubs and PluginStore mocks for improved testing accuracy.
Remove code validation script and refactor imports in activities and workflows
- Deleted the `validate.sh` script, which was responsible for running code quality checks.
- Cleaned up import statements in `activities.py`, `gates.py`, `mlflow.py`, and `storage.py` by removing unused imports and organizing them.
- Refactored initialization methods in `MinioManager` and `MLFlow` classes for improved readability.
- Updated various workflows to ensure compatibility with the new structure and removed unnecessary comments.
- Enhanced test cases to accommodate changes in the activities and workflows, ensuring proper mocking of dependencies.
Implement MinIO Offload and Retention Features
- Added configuration options for MinIO retention hours and offload threshold in README.
- Introduced MinIO payload offloading for large DataFrame-derived payloads, storing them as parquet files.
- Updated activities to utilize MinIO for data loading and cleanup, including new methods for offloading and retention management.
- Refactored existing activities to integrate MinIO functionality, ensuring compatibility with previous workflows.
- Removed the legacy MinioRepository class, consolidating MinIO operations under a new manager structure.
- Updated requirements to use the latest version of the sientia-dataops-library.
Update sonar-project.properties to exclude all worker files from coverage and modify execution counts and timestamps in tests.ipynb. Add a new test for empty DataFrame handling in test_model_repository.py.
Refactor validation script and improve logging in API and model repository
- Updated validation script to include 'e2e/' directory in code formatting and linting checks.
- Enhanced error logging in API class to improve readability of error messages.
- Refactored debug logging in model repository for better structured output.
- Cleaned up import statements in various files for improved organization.
Enhance end-to-end tests for PredictionsBatch workflow scenarios
- Introduced mock repositories for PI Web API and OPC operations to improve test coverage.
- Updated test scenarios to handle partial write errors for PI Web API and OPC.
- Refactored existing tests to assert correct behavior under various error conditions.
- Enhanced logging and error handling in API and OPC activities to provide clearer feedback on failures.
- Removed outdated integration test file to streamline test suite.
Update sientia-mlops-library dependency to version 0.40.6 and refactor get_prediction_data method calls for improved readability in model_repository.py and test_model_repository.py.
Enhance prediction data retrieval in MLFlowRepository by adding predict_flavor parameter
- Updated get_prediction_data method to accept a predict_flavor argument, allowing for different prediction model handling.
- Adjusted calls to get_prediction_data throughout the codebase to include the new parameter.
- Added new test cases to validate behavior for different predict_flavor values, ensuring robust functionality.
SIENTIAPDE-1273
Enhance security analysis and SQL injection handling
- Added skip for potential SQL injection false positives in Bandit configuration.
- Updated validate.sh to use the pyproject.toml configuration for Bandit security analysis.
- Refactored code to replace ensure_dataframe utility with direct DataFrame usage in multiple activities, improving clarity and reducing dependencies.
- Removed the deprecated dataframe_utils module to streamline the codebase.
Enhance MLFlowRepository and Activities classes with new methods and metrics
- Added `check_artifact_exists` method to MLFlowRepository for verifying artifact presence in the MLflow Model Registry.
- Implemented `get_prediction_data` method in MLFlowRepository to retrieve prediction data from models.
- Updated Activities class to integrate ModelMetrics for improved metrics handling.
- Enhanced tests for artifact existence checks and prediction data retrieval, ensuring robust coverage for new functionalities.
- Updated various workflows to include `transform_table_name` in input data for better data handling.
Update logging in MLFlowRepository and OpcRepository to use unified logging methods
- Refactored logging calls in MLFlowRepository to replace `self.logger.info` and `self.logger.debug` with `self.info` and `self.debug` for consistency.
- Updated connection logging in OpcRepository to format the connection message properly.
- Adjusted test cases to reflect changes in logging behavior and ensure proper assertions.
Enhance metrics handling in Gates and MLFlow classes
- Added checks for `None` response times before emitting OPC writing metrics in the Gates class to prevent unnecessary metric emissions.
- Updated the MLFlow class to conditionally sort and drop duplicates based on the presence of the 'created_at' column, ensuring robustness in data processing.
- Adjusted corresponding tests to validate the new behavior in both classes.
Refactor monitoring and metrics integration across various components
- Removed coverage options from `pyproject.toml`.
- Updated prediction metrics in `README.md` to replace `pipeline_name` with `workflow_name`.
- Upgraded `sientia-dataops-library` dependency version in `requirements-light.txt` and `requirements.txt`.
- Enhanced metrics handling in `laborious` activities, including `Activities`, `Gates`, `MLFlow`, and `OPC`, to utilize a new `MetricsController`.
- Refactored metric emission methods to improve clarity and consistency across the codebase.
- Updated tests to reflect changes in metrics handling and ensure proper functionality.
Refactor OPC Write Method and Comment Out Error Handling Logic
- Reformatted the `write_opc_data` method for improved readability by adjusting its signature.
- Commented out the error handling logic in `OpcRepository` related to disconnection due to multiple errors, preserving the original functionality for future reference.
- Updated tests to reflect the commented-out error handling, ensuring clarity in the testing process.
Enhance OPC Metrics Handling and Refactor Write Operations
- Updated the OPC class to return response times for write operations, improving metrics tracking.
- Refactored the Gates activity to incorporate OPC metrics into the metrics writing process.
- Adjusted the manage_output_tags method in OpcRepository to return response times for each tag written.
- Modified tests to validate the new metrics structure and ensure correct behavior of the updated methods.
Refactor error handling in MLFlowRepository and update tests
- Improved error message formatting in MLFlowRepository for better readability.
- Updated test assertions to ensure correct calls to MLflow methods during experiment retrieval and creation.
Refactor OPC handling by removing pod_id from initialization and updating logging format
- Removed pod_id parameter from OPC class and repository initialization to streamline connection management.
- Updated logging statements for improved readability during disconnection attempts and error handling.
Refactor tests and update model_repository.py for clarity and consistency
- Added a blank line in model_repository.py for improved readability.
- Adjusted formatting in test_mlflow.py to streamline assertions.
- Updated ensure_bucket_exists method tests in test_minio_repository.py to reflect the new return value of None instead of True.
Enhance validation script and refactor code in various modules
- Updated the validation script to include automatic code formatting and linting fixes using Ruff.
- Removed the `clean_tmp_files` method from the Gates class to streamline functionality.
- Simplified conditional checks in the OpcRepository for better clarity and error handling.
- Added model ID to the minimal retrain workflow for improved tracking.
- Introduced new test cases for error handling in MLFlow and storage operations, ensuring robustness in repository interactions.
Update .gitignore and refactor metrics.py for improved logging and consistency
- Added coverage.xml to .gitignore to prevent tracking of coverage reports.
- Refactored metric labels in metrics.py for consistency in string formatting and improved readability.
- Enhanced logging messages in various activities to ensure uniformity in message formatting.
Add placeholder class 'Any' in test_model_repository.py and update invalid_cases to use it
- Introduced a new placeholder class 'Any' to be used in test cases.
- Updated the 'invalid_cases' list to replace integer keys with instances of the 'Any' class, enhancing test coverage for key types.
