SIENTIAPDE-1646 Merge origin/main into release/SIENTIAPDE-1646
Bring main up to date while keeping this branch's sync execution, PluginStore
runtime and sientia_model (SientiaMLflowRepository/SientiaModel) structure.
Resolution decisions:
- OPC: keep the sync asyncua.sync implementation; port main's server_uri
validation, e2e race fix (lock held until released) and the new unit tests
converted to sync.
- Model import workflow (.sientia bundles), joblib/transform flavors and the
local MLFlowRepository (model_repository.py) are dropped, with their tests,
docs, metrics, env vars and connectors config.
- Worker: drift/simple_metrics stay pinned to the 'core' runtime; port main's
HOSTNAME pod id, logger-based prometheus startup and sys.exit after finally.
- values.yaml removed, following main (SIENTIAPDE-1945).
- Deps: keep sientia_model; take main's fsspec/s3fs/botocore/boto3 pins and
sientia_do>=1.12.2; drop import-only deps (pynacl, cryptography, pymongo,
python-dotenv, pycaret, joblib, gcsfs).
- e2e: keep sientia_data schema and scenario input files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QTZPOC-20: Improve worker shutdown handling by allowing SystemExit and KeyboardInterrupt to propagate directly, preserving their original exit codes and ensuring cleanup.
QTZPOC-20: Enable code_paths for sklearn models; improve worker shutdown handling
The MLFlowRepository now resolves and forwards code_paths to mlflow.sklearn.log_model, ensuring custom code dependencies for sklearn models are preserved and correctly reloaded. A new integration test validates this end-to-end functionality. Additionally, worker exception handling is refined to distinguish between deliberate shutdowns (SystemExit, KeyboardInterrupt) and unhandled exceptions, preventing unnecessary error logging for expected exits.
QTZPOC-20: Refactor model import workflow, introduce atomic model ID generation, and clarify MinIO expiry.
Fuses bundle download, decryption, MLflow experiment creation, and artifact upload into a single activity to prevent cross-worker issues. Implements an atomic counter for MongoDB model IDs, replacing the max(id) + 1 scan to avoid collisions during concurrent imports. Adds comprehensive documentation for MinIO object expiry mechanisms and refines lifecycle rule application. Improves error handling with clearer responsibility classification and centralizes MLflow repository injection.
QTZPOC-20: Improve structured logging and reporting for all model import pipeline failures by introducing a dedicated activity and centralizing error classification. This ensures consistent observability, with all workflow failures now reported before cleanup. Also, adjusted import step timeouts and refined manual run script cleanup logic.
QTZPOC-20: Implement explicit joblib model flavor and enhance model import workflow with extensive E2E tests.
This commit introduces a first-class joblib model flavor for direct loading and saving of Python pickle artifacts, including custom code within the artifact's code/ directory, bypassing MLflow's flavor loaders. The model import workflow's status database configuration is now streamlined to default to the POSTGRES_* server, with explicit IMPORT_STATUS_DB_* overrides.
Comprehensive end-to-end tests have been added for the model import workflow, covering success paths, gate rejections, digest enforcement, provisioning failures, status record integrity, cleanup, and retry/recovery scenarios using a real-clock environment. New E2
QTZPOC-20: Implement model import workflow for .sientia bundles
Introduces the import_model workflow and its associated activities to enable importing .sientia bundles into MLflow and MongoDB. This includes:
* New core workflow and activity logic for handling bundle download, decryption, validation, MLflow provisioning (experiments, runs, registered versions), and MongoDB model document creation.
* Comprehensive E2E and unit test suites, including a dedicated test harness and a manual run script for development.
* Updated .env.example and README.md with new configurations and usage instructions.
* Ensures correct MLflow model source resolution for legacy loader compatibility and robust MongoDB document management.
* Refactors existing utility scripts into a scripts/ directory.
QTZPOC-20: Implement .sientia model bundle import workflow
Introduces a new Temporal workflow and activities to import .sientia model bundles. This enables:
- Secure decryption and validation of bundles through a seven-gate process.
- Provisioning imported models in MLflow and updating the MongoDB model listing.
- Comprehensive status tracking in the public.experiment_run table with new IMPORT_SUCCESS/IMPORT_ERROR states.
- Dedicated metrics, robust error handling, and configurable settings.
- Includes updated e2e database fixtures and extensive unit/integration tests for bundle format, reader, and overall workflow.
SIENTIAPDE-1646 Refactor MLFlow tests and update artifact handling
- Enhanced test cases for MLFlow to improve clarity and accuracy in data handling.
- Updated references in tests to use 'evaluation_data.csv' and 'test_data.csv' instead of 'retrain_input.csv' and 'train_data.csv'.
- Introduced a new helper function for creating prediction frames to streamline test setup.
Enhance MLFlow reference data handling and testing
- Updated the artifact handling in MLFlow to prioritize 'retrain_input.csv' over 'train_data.csv' when resolving reference data.
- Introduced new methods for resolving artifact names and locating downloaded CSV files.
- Modified the `get_reference_data` method to improve artifact resolution and error handling.
- Expanded unit tests to cover scenarios for missing artifacts and preference logic between retrain and train data CSVs.
Update values.yaml and worker.py for runtime installation logic
- Updated helm upgrade command version in values.yaml from 0.6.1 to 0.6.5 for deployment reference.
- Modified worker.py to adjust runtime installation logic, ensuring 'legacy' runtime is set to 'single' for compatibility.
Update .gitignore, requirements, and enhance OPC UA error handling
- Added new entries to .gitignore for openspec and cursor directories.
- Updated sientia-dataops-library dependency version in requirements-light.txt from 1.10.4 to 1.12.0.
- Enhanced OPC UA communication by refining reconnect logic and error handling in opc_repository.py, including the introduction of a reconnect flag and improved session management.
- Updated tests to cover new reconnect scenarios and ensure robust error handling for protocol states.
Enhance OPC UA communication and metrics tracking
- Updated README.md to include new OPC UA Communication section and detailed metrics for session and write diagnostics.
- Added new metrics in laborious/metrics.py for tracking OPC UA session states and write attempts.
- Refactored OPC activity in laborious/activities/opc.py to handle session errors and improve error reporting.
- Updated e2e tests to cover new scenarios for OPC session/channel errors and reconnect handling.
- Modified .gitignore to include relatorio files and mlruns directory.
- Added ipykernel to requirements-dev.txt for Jupyter notebook support.
Update .gitignore, requirements, and enhance OPC UA error handling
- Added new entries to .gitignore for openspec and cursor directories.
- Updated sientia-dataops-library dependency version in requirements-light.txt from 1.10.4 to 1.12.0.
- Enhanced OPC UA communication by refining reconnect logic and error handling in opc_repository.py, including the introduction of a reconnect flag and improved session management.
- Updated tests to cover new reconnect scenarios and ensure robust error handling for protocol states.
Enhance OPC UA communication and metrics tracking
- Updated README.md to include new OPC UA Communication section and detailed metrics for session and write diagnostics.
- Added new metrics in laborious/metrics.py for tracking OPC UA session states and write attempts.
- Refactored OPC activity in laborious/activities/opc.py to handle session errors and improve error reporting.
- Updated e2e tests to cover new scenarios for OPC session/channel errors and reconnect handling.
- Modified .gitignore to include relatorio files and mlruns directory.
- Added ipykernel to requirements-dev.txt for Jupyter notebook support.
Update README, requirements, and E2E tests for improved configuration and functionality
- Enhanced the README with updated model configuration examples, including the addition of an alias for production.
- Removed the `requirements-light.txt` file and updated `requirements-local.txt` and `requirements.txt` to replace `asyncua` with `opcua`.
- Refactored E2E test scenarios to utilize scenario input files for better maintainability and clarity.
- Improved test coverage for MinIO offload functionality and added new helper functions for loading scenario inputs.
- Updated `values.yaml` to reflect new global configurations and environment variables for the laborious worker.
Update dependencies and refactor MLFlow activities
- Replaced direct GitHub dependencies in `requirements.txt` with specific versioned packages for `sientia_do` and `sientia_model`.
- Refactored imports in `activities.py` to streamline the code structure.
- Enhanced the `MLFlow` class in `mlflow.py` by introducing a method to resolve model aliases, improving flexibility in model lookups.
- Simplified shutdown logic in `worker.py` for better readability.
- Added new tests for MLFlow activities and improved existing test coverage for data handling and model retraining processes.
Enhance environment configuration and update dependencies
- Added new environment variables for PluginStore and MLflow configuration in `.env.example`, including `RUNTIME`, `STORE_BASE_URL`, `STORE_OWNER`, `STORE_REPO`, `STORE_BRANCH`, `STORE_USERNAME`, `STORE_PASSWORD`, `STORE_CACHE_TTL_SECONDS`, `PYPI_SERVER`, `PYPI_USERNAME`, and `PYPI_PASSWORD`.
- Updated `git-requirements-mapping.txt` to reflect changes in repository names.
- Modified `requirements-light.txt` and `requirements.txt` to upgrade `sientia-dataops-library` to version 1.12.0 and `sientia-mlops-library` to version 0.8.1.
- Updated `values.yaml` to include new environment variables for worker runtime and PluginStore configuration.
- Refactored E2E tests to utilize new MLflow repository stubs and PluginStore mocks for improved testing accuracy.
Update dependencies and refactor input filter handling for consistency
- Updated sientia-dataops-library dependency version from 1.10.3 to 1.10.4 in requirements.txt.
- Refactored input filter handling in the Gates class to read policy and config keys in a case-insensitive manner.
- Updated test cases to ensure consistency in filter key naming conventions across various scenarios.
Remove `query_to_minio` method from Storage class and update worker activities to eliminate its usage. This change streamlines the codebase by removing unused functionality related to MinIO queries.
Implement MinIO Offload and Retention Features
- Added configuration options for MinIO retention hours and offload threshold in README.
- Introduced MinIO payload offloading for large DataFrame-derived payloads, storing them as parquet files.
- Updated activities to utilize MinIO for data loading and cleanup, including new methods for offloading and retention management.
- Refactored existing activities to integrate MinIO functionality, ensuring compatibility with previous workflows.
- Removed the legacy MinioRepository class, consolidating MinIO operations under a new manager structure.
- Updated requirements to use the latest version of the sientia-dataops-library.
Refactor validation script and improve logging in API and model repository
- Updated validation script to include 'e2e/' directory in code formatting and linting checks.
- Enhanced error logging in API class to improve readability of error messages.
- Refactored debug logging in model repository for better structured output.
- Cleaned up import statements in various files for improved organization.
Update environment variables in values.yaml and refactor worker.py for improved worker preparation
- Removed KAFKA_BOOTSTRAP_SERVERS from environment variables in values.yaml.
- Added PYPI_SERVER environment variable for library distribution.
- Refactored worker.py to replace resource tuner and poller behavior with a new prepare_worker function, streamlining worker initialization and enhancing code clarity.
Enhance README and Codebase with PI Web API Integration
- Updated README.md to include details about PI Web API integration, including configuration and export capabilities.
- Modified Activities class to incorporate PI Web API export operations and error handling.
- Added new API class for handling PI Web API interactions, including writing prediction and confidence data.
- Updated prediction workflows to support PI Web API output configuration.
- Enhanced worker and sub-workflows to include PI Web API in task queues and export processes.
- Improved documentation and error handling for PI Web API connections and configurations.
Refactor Activities and API Integration for PI Web API
- Reintroduced the API import in the Activities class for proper integration.
- Cleaned up whitespace and formatting in the API class and related tests for improved readability.
- Updated test cases to ensure consistent formatting in error messages and configuration structures for PI Web API.
- Enhanced connectors_config.py with additional whitespace for better organization.
Enhance Activities and Prediction Workflows with PI Web API Integration
- Updated the Activities class to include API integration, allowing for configuration of PI Web API parameters.
- Modified prediction workflows to support output configuration for PI Web API, enabling data writing to the API.
- Refactored connectors_config.py by removing unused PostgreSQL and MongoDB configuration functions.
- Added tests to validate the new PI Web API functionality in activities and workflows, ensuring robust integration and functionality.
Refactor import statements in worker.py for improved organization
- Moved the import of the os module to the appropriate section, enhancing clarity and consistency in the import structure.
Enhance resource management and configuration in Laborious worker
- Updated `values.yaml` to define resource limits and requests for better performance tuning.
- Modified environment variables in `worker.py` to support resource-based scaling and improved task queue management.
- Introduced new functions for creating resource tuners and poller behaviors, enhancing scalability and efficiency in handling workloads.
SIENTIAPDE-1273
Enhance security analysis and SQL injection handling
- Added skip for potential SQL injection false positives in Bandit configuration.
- Updated validate.sh to use the pyproject.toml configuration for Bandit security analysis.
- Refactored code to replace ensure_dataframe utility with direct DataFrame usage in multiple activities, improving clarity and reducing dependencies.
- Removed the deprecated dataframe_utils module to streamline the codebase.
Update dependencies and refactor data handling in various modules
- Updated sientia-dataops-library dependency version from 1.5.3 to 1.5.4 in requirements files.
- Updated sientia-mlops-library dependency version from 0.39.0 to 0.40.2 in requirements files.
- Refactored return types in Gates, MLFlow, and ModelMetrics classes to return dictionaries instead of DataFrames for improved compatibility with downstream systems.
- Removed the temporal_codec module as it is no longer needed for DataFrame serialization.
- Adjusted data handling in the Drift workflow to ensure proper data structure is maintained.
Refactor data handling in various modules to ensure DataFrame consistency
- Replaced direct DataFrame instantiation with `ensure_dataframe` utility in Gates, MLFlow, OPC, and ModelMetrics classes to standardize data handling.
- Updated return types in several asynchronous methods to return DataFrames instead of dictionaries for improved usability.
- Adjusted data export processes in workflows to convert DataFrames to dictionaries with `to_dict(orient='records')` for compatibility with downstream systems.
Refactor ModelMetrics return format and correct import name in worker module
- Changed the return format of the metrics data in ModelMetrics from a dictionary to a list for improved usability.
- Corrected the import statement for SimpleMetrics in the worker module to ensure consistency and clarity.
Update version and enhance metrics calculation in Laborious system
- Updated image tag in values.yaml from 1.1.0 to 1.1.1.
- Modified GITHUB_BRANCH environment variable for consistency.
- Added a new method `calculate_simple_metrics` in model_metrics.py to compute various model performance metrics including RMSE, MSE, MAE, and R2.
- Integrated the new metrics calculation into the worker setup, allowing for concurrent processing of simple metrics.
- Updated tests to cover the new metrics calculation functionality, ensuring comprehensive validation of the implementation.
Enhance MLFlowRepository and Activities classes with new methods and metrics
- Added `check_artifact_exists` method to MLFlowRepository for verifying artifact presence in the MLflow Model Registry.
- Implemented `get_prediction_data` method in MLFlowRepository to retrieve prediction data from models.
- Updated Activities class to integrate ModelMetrics for improved metrics handling.
- Enhanced tests for artifact existence checks and prediction data retrieval, ensuring robust coverage for new functionalities.
- Updated various workflows to include `transform_table_name` in input data for better data handling.
Enhance thread safety in MLFlowRepository model caching
- Introduced a reentrant lock to synchronize access to the model cache, ensuring thread safety during cache checks and updates.
- Updated the cache management logic to acquire the lock when checking for existing models and when updating the cache after downloading a new model.
- Reduced the maximum cached workflows in the worker configuration for improved resource management.
Update .gitignore and refactor metrics.py for improved logging and consistency
- Added coverage.xml to .gitignore to prevent tracking of coverage reports.
- Refactored metric labels in metrics.py for consistency in string formatting and improved readability.
- Enhanced logging messages in various activities to ensure uniformity in message formatting.
Enhance Gates and MLFlowRepository with new functionalities and improvements
- Added a new method `clean_tmp_files` in the Gates class to remove temporary files associated with model retraining.
- Updated MLFlowRepository methods to improve experiment handling, including dynamic parameter logging and model retrieval.
- Refactored model loading methods to streamline the process and enhance error handling.
- Improved logging for model operations and added support for model parameter retrieval.
- Adjusted minimal_retrain workflow to extend timeouts for activities and ensure proper model configuration handling.
Enhance MinIO integration and update environment configurations
- Added MinIO configuration parameters to .env.example and values.yaml for improved storage management.
- Updated requirements.txt to include necessary libraries for MinIO support.
- Refactored Activities class to utilize Storage for MinIO interactions.
- Enhanced MLFlow class to integrate MinIO for data retrieval during model retraining.
- Introduced build_minio_config function to streamline MinIO configuration setup.
- Updated minimal_retrain workflow to support data storage in MinIO.
Enhance MLFlow and MLFlowRepository with improved data handling and logging
- Refactored MLFlow class to sort data by 'created_at' and drop duplicates for better input preparation.
- Updated MLFlowRepository methods to include detailed logging for artifact downloads and model predictions.
- Introduced LzmaPayloadCodec for efficient payload compression in the worker, optimizing data handling for large payloads.
- Enhanced timestamp handling in treated data to ensure compatibility with model expectations.
Update model retraining and logging enhancements
- Changed the GITHUB_BRANCH value in values.yaml to 'main' for consistency.
- Refactored MLFlow class to improve timestamp handling and error messaging during model retraining.
- Enhanced MLFlowRepository methods to include metadata logging and improved model version retrieval.
- Updated minimal_retrain workflow to support extended timeout for activities and include model configuration in input data.
