Simplify merge in retrain: expose timestamp explicitly for join
Refactor evaluation_data merge to reset both frames' index into a
"timestamp" column before joining, instead of merging on index then
copying it out afterward. Same validate='one_to_one' safety check,
fewer intermediate steps.
Harden retrain merge and OPC write against silent failures
- pandas merge now uses validate='one_to_one' so duplicated timestamps
from wrapper.retrain fail loudly instead of silently multiplying rows
- OPC write_data raises OpcClientNotInitializedError instead of a bare
assert when client is torn down between validation and write, so
callers get a reported error instead of an AssertionError
- add regression test covering the torn-down client path
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SIENTIAPDE-1646 Merge origin/main into release/SIENTIAPDE-1646
Bring main up to date while keeping this branch's sync execution, PluginStore
runtime and sientia_model (SientiaMLflowRepository/SientiaModel) structure.
Resolution decisions:
- OPC: keep the sync asyncua.sync implementation; port main's server_uri
validation, e2e race fix (lock held until released) and the new unit tests
converted to sync.
- Model import workflow (.sientia bundles), joblib/transform flavors and the
local MLFlowRepository (model_repository.py) are dropped, with their tests,
docs, metrics, env vars and connectors config.
- Worker: drift/simple_metrics stay pinned to the 'core' runtime; port main's
HOSTNAME pod id, logger-based prometheus startup and sys.exit after finally.
- values.yaml removed, following main (SIENTIAPDE-1945).
- Deps: keep sientia_model; take main's fsspec/s3fs/botocore/boto3 pins and
sientia_do>=1.12.2; drop import-only deps (pynacl, cryptography, pymongo,
python-dotenv, pycaret, joblib, gcsfs).
- e2e: keep sientia_data schema and scenario input files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QTZPOC-20: Refactor ModelImport status engine for lazy initialization, add byte-counted size validation to bundle extraction, optimize MLflow run ID lookups, and ensure workflow cancellation propagates correctly.
QTZPOC-20: Refactor model import workflow, introduce atomic model ID generation, and clarify MinIO expiry.
Fuses bundle download, decryption, MLflow experiment creation, and artifact upload into a single activity to prevent cross-worker issues. Implements an atomic counter for MongoDB model IDs, replacing the max(id) + 1 scan to avoid collisions during concurrent imports. Adds comprehensive documentation for MinIO object expiry mechanisms and refines lifecycle rule application. Improves error handling with clearer responsibility classification and centralizes MLflow repository injection.
QTZPOC-20: Add skip_predict_timestamp flag to model_config to prevent timestamp reinsertion in prediction data, and document other ad-hoc behavioral flags.
QTZPOC-20: Improve structured logging and reporting for all model import pipeline failures by introducing a dedicated activity and centralizing error classification. This ensures consistent observability, with all workflow failures now reported before cleanup. Also, adjusted import step timeouts and refined manual run script cleanup logic.
QTZPOC-20: Implement explicit joblib model flavor and enhance model import workflow with extensive E2E tests.
This commit introduces a first-class joblib model flavor for direct loading and saving of Python pickle artifacts, including custom code within the artifact's code/ directory, bypassing MLflow's flavor loaders. The model import workflow's status database configuration is now streamlined to default to the POSTGRES_* server, with explicit IMPORT_STATUS_DB_* overrides.
Comprehensive end-to-end tests have been added for the model import workflow, covering success paths, gate rejections, digest enforcement, provisioning failures, status record integrity, cleanup, and retry/recovery scenarios using a real-clock environment. New E2
QTZPOC-20: Implement model import workflow for .sientia bundles
Introduces the import_model workflow and its associated activities to enable importing .sientia bundles into MLflow and MongoDB. This includes:
* New core workflow and activity logic for handling bundle download, decryption, validation, MLflow provisioning (experiments, runs, registered versions), and MongoDB model document creation.
* Comprehensive E2E and unit test suites, including a dedicated test harness and a manual run script for development.
* Updated .env.example and README.md with new configurations and usage instructions.
* Ensures correct MLflow model source resolution for legacy loader compatibility and robust MongoDB document management.
* Refactors existing utility scripts into a scripts/ directory.
QTZPOC-20: Implement .sientia model bundle import workflow
Introduces a new Temporal workflow and activities to import .sientia model bundles. This enables:
- Secure decryption and validation of bundles through a seven-gate process.
- Provisioning imported models in MLflow and updating the MongoDB model listing.
- Comprehensive status tracking in the public.experiment_run table with new IMPORT_SUCCESS/IMPORT_ERROR states.
- Dedicated metrics, robust error handling, and configurable settings.
- Includes updated e2e database fixtures and extensive unit/integration tests for bundle format, reader, and overall workflow.
Renamed the OPC.close asynchronous method to OPC.aclose to align with common Python conventions for asynchronous context managers and methods, improving clarity. All call sites and tests have been updated accordingly.
SIENTIAPDE-1646 Refactor MLFlow tests and update artifact handling
- Enhanced test cases for MLFlow to improve clarity and accuracy in data handling.
- Updated references in tests to use 'evaluation_data.csv' and 'test_data.csv' instead of 'retrain_input.csv' and 'train_data.csv'.
- Introduced a new helper function for creating prediction frames to streamline test setup.
SIENTIAPDE-1646 Update prediction data handling in MLFlow
- Renamed the prediction column in the MLFlow retraining process to improve clarity and consistency in the output data.
SIENTIAPDE-1646 Enhance MLFlow retraining process and evaluation data handling
- Updated the MLFlow class to merge prediction data with retrain data for improved evaluation.
- Renamed target column to "target" and added a timestamp column to the evaluation data.
- Logged both retrain input and evaluation data as artifacts in MLFlow.
Enhance MLFlow reference data handling and testing
- Updated the artifact handling in MLFlow to prioritize 'retrain_input.csv' over 'train_data.csv' when resolving reference data.
- Introduced new methods for resolving artifact names and locating downloaded CSV files.
- Modified the `get_reference_data` method to improve artifact resolution and error handling.
- Expanded unit tests to cover scenarios for missing artifacts and preference logic between retrain and train data CSVs.
Update requirements and enhance OPC communication handling
- Updated `requirements-local.txt` to use the latest versions of `sientia-dataops-library` (1.12.1) and `sientia-model-library` (0.10.0).
- Improved documentation in `opc-communication.md` to clarify the transition from async to sync implementation and added details on error classification and reconnection behavior.
- Refactored `init_opc` method in `opc.py` to enhance connection handling and logging, ensuring independent server initialization.
- Enhanced validation and writing methods in `opc.py` to provide better feedback and error handling for OPC server operations.
- Updated `opc_repository.py` to improve error payload construction and session management metrics.
SIENTIAPDE-1646 Enhance OPC integration and E2E testing framework
- Updated .gitignore to exclude '.cursor/*' for better file management.
- Added new marker in pyproject.toml for E2E tests using OPC.
- Introduced async OPC server fixtures and improved connection handling in conftest.py.
- Enhanced error handling in OPC activities and added metrics for session management in metrics.py.
- Updated E2E tests to cover new OPC scenarios, including session errors and reconnect handling.
- Refactored helper functions to improve prediction assertion logic in helpers.py.
- Documented new OPC scenarios in scenarios.md for clarity on expected outcomes.
SIENTIAPDE-1811 Implement OPC write error handling and refactor tag writing logic
- Introduced a new function `_apply_opc_write_error` to manage session and reconnect flags based on OPC write error responses.
- Refactored the `_write_tags_from_config` method to streamline the writing of OPC tags for both prediction and confidence data.
- Enhanced unit tests to cover various scenarios for OPC write errors, including session bad and reconnect in progress cases.
- Updated existing tests to validate the new logic and ensure robust error handling.
Enhance OPC UA communication and metrics tracking
- Updated README.md to include new OPC UA Communication section and detailed metrics for session and write diagnostics.
- Added new metrics in laborious/metrics.py for tracking OPC UA session states and write attempts.
- Refactored OPC activity in laborious/activities/opc.py to handle session errors and improve error reporting.
- Updated e2e tests to cover new scenarios for OPC session/channel errors and reconnect handling.
- Modified .gitignore to include relatorio files and mlruns directory.
- Added ipykernel to requirements-dev.txt for Jupyter notebook support.
Refactor OPC and OpcRepository classes to remove logger dependency and streamline logging calls
- Removed logger attribute from OPC and OpcRepository classes, replacing direct logger calls with class methods for logging.
- Updated write_data method signatures to eliminate logger parameter, simplifying the interface.
- Adjusted related tests to reflect changes in method signatures and logging behavior.
Implement new scheduling configurations for model retraining and drift analysis in input_sample.json
- Added three new schedule configurations: `minimal-retrain-test-runtime`, `drift-test-runtime`, and `simple-metrics-test-runtime`.
- Each configuration includes parameters such as model ID, workflow type, frequency, and specific queries for data retrieval.
- Enhanced the structure to support active status and updated timestamps for better tracking of schedule states.
SIENTIAPDE-1646 Add new scheduling configurations and remove outdated documentation
- Introduced new scheduling configurations for minimal retrain, drift analysis, and simple metrics in `input_sample.json`.
- Removed obsolete documentation files related to drift analysis and E2E test reports to streamline project resources.
- Updated E2E tests for minimal retrain to enhance reporting and error handling during model retraining processes.
SIENTIAPDE-1646 Update dependencies and refactor OPC integration
- Replaced `opcua` with `asyncua` in `requirements-local.txt` and `requirements.txt` to utilize the async capabilities.
- Updated `values.yaml` to change the GitHub branch from `feature/SIENTIAPDE-1646` to `release/SIENTIAPDE-1646`.
- Refactored `opc.py` and `opc_repository.py` to accommodate the new `asyncua` library, ensuring compatibility with the synchronous API.
- Adjusted tests in `test_opc_repository.py` to reflect changes in the client implementation and maintain functionality.
Update E2E test report and enhance drift analysis handling
- Updated the E2E test report metrics to reflect the latest test results, showing 47 collected tests with all passing.
- Removed outdated sections related to failed tests and their causes, streamlining the report.
- Implemented a regression fix in the drift analysis to handle empty merged frames, ensuring workflows skip export when no drift metrics are available.
- Enhanced the `insert_sample_data` and `insert_sample_prediction` functions to allow customizable timestamps for better test accuracy.
- Refactored E2E tests to improve clarity and maintainability, particularly in handling repeat scenarios with distinct timestamps.
Refactor ModelMetrics to utilize DriftAnalysis for drift detection
- Replaced ModelAnalysis with DriftAnalysis in the ModelMetrics class to enhance drift detection capabilities.
- Updated method signatures and documentation to reflect the changes in target_name and return values.
- Adjusted data handling to ensure compatibility with the new analysis methods and improved clarity in the drift metrics dataframe preparation.
Update README, requirements, and E2E tests for improved configuration and functionality
- Enhanced the README with updated model configuration examples, including the addition of an alias for production.
- Removed the `requirements-light.txt` file and updated `requirements-local.txt` and `requirements.txt` to replace `asyncua` with `opcua`.
- Refactored E2E test scenarios to utilize scenario input files for better maintainability and clarity.
- Improved test coverage for MinIO offload functionality and added new helper functions for loading scenario inputs.
- Updated `values.yaml` to reflect new global configurations and environment variables for the laborious worker.
Refactor MLFlow retraining logic to always use retrain method
- Removed the conditional logic for full retraining, ensuring the `retrain` method is always called.
- Updated the documentation in the `retrain_model` method to reflect the changes in retraining flow.
- Adjusted tests to verify that the `retrain` method is invoked correctly, while ensuring `train` is not called when the full retrain flag is set.
Update dependencies and refactor MLFlow activities
- Replaced direct GitHub dependencies in `requirements.txt` with specific versioned packages for `sientia_do` and `sientia_model`.
- Refactored imports in `activities.py` to streamline the code structure.
- Enhanced the `MLFlow` class in `mlflow.py` by introducing a method to resolve model aliases, improving flexibility in model lookups.
- Simplified shutdown logic in `worker.py` for better readability.
- Added new tests for MLFlow activities and improved existing test coverage for data handling and model retraining processes.
Enhance environment configuration and update dependencies
- Added new environment variables for PluginStore and MLflow configuration in `.env.example`, including `RUNTIME`, `STORE_BASE_URL`, `STORE_OWNER`, `STORE_REPO`, `STORE_BRANCH`, `STORE_USERNAME`, `STORE_PASSWORD`, `STORE_CACHE_TTL_SECONDS`, `PYPI_SERVER`, `PYPI_USERNAME`, and `PYPI_PASSWORD`.
- Updated `git-requirements-mapping.txt` to reflect changes in repository names.
- Modified `requirements-light.txt` and `requirements.txt` to upgrade `sientia-dataops-library` to version 1.12.0 and `sientia-mlops-library` to version 0.8.1.
- Updated `values.yaml` to include new environment variables for worker runtime and PluginStore configuration.
- Refactored E2E tests to utilize new MLflow repository stubs and PluginStore mocks for improved testing accuracy.
SIENTIAPDE-1712 Enhance logging across various classes by adding logger parameters and improving debug statements. This update includes adjustments in Gates, MLFlow, ModelMetrics, and MLFlowRepository classes for better traceability and observability during operations.
SIENTIAPDE-1712 Enhance logging in Gates, MLFlow, and Storage classes by integrating logger parameter for improved traceability. This update allows for better monitoring of operations and data handling across these components.
SIENTIAPDE-1712 Refactor debug logging for DataFrames across multiple classes. Introduced a new method to log DataFrame content conditionally based on row count in Gates, MLFlow, ModelMetrics, and MLFlowRepository classes, improving debugging capabilities while managing log output effectively.
SIENTIAPDE-1712 Implement debug logging for DataFrames in MLFlow and MLFlowRepository classes. Added a method to log DataFrame content conditionally based on row count, enhancing debugging capabilities while preventing excessive log output.
Update Gates and FormatAndExportPrediction classes to standardize timestamp usage
- Changed 'last_timestamp' to 'timestamp' in the Gates class for consistency in input data handling.
- Updated the FormatAndExportPrediction class to reflect the same change in the output data structure.
Update Gates and FormatAndExportPrediction classes to use 'last_timestamp' for improved data handling
- Modified the Gates class to utilize 'last_timestamp' when only one row is present, ensuring accurate timestamp assignment.
- Updated the FormatAndExportPrediction class to replace 'timestamp' with 'last_timestamp' in the output data structure.
Enhance E2E testing with MinIO support and update documentation
- Updated `requirements-dev.txt` to include MinIO support in testcontainers.
- Added a new fixture for MinIO container setup in `conftest.py` to facilitate E2E tests involving S3-compatible storage.
- Introduced a new test fixture for activities using a real MinIO container in `conftest.py`.
- Updated E2E test scenarios and documentation to reflect the integration of MinIO for offload uploads and clarified error handling in workflows.
- Refactored existing tests to improve clarity and maintainability.
Update dependencies and refactor input filter handling for consistency
- Updated sientia-dataops-library dependency version from 1.10.3 to 1.10.4 in requirements.txt.
- Refactored input filter handling in the Gates class to read policy and config keys in a case-insensitive manner.
- Updated test cases to ensure consistency in filter key naming conventions across various scenarios.
Refactor metrics and API handling for improved consistency and clarity
- Removed the SIENTIA_CORE_LABELS constant and replaced it with CORE_LABELS for uniformity across metrics.
- Updated the API class to ensure operation_type is always included in core labels for PI Web API metrics.
- Simplified metric tag handling in the Gates class by consolidating common tags into a single core_tags dictionary.
- Enhanced the PredictionProcess class to improve error handling and variable naming for clarity.
SIENTIAPDE-1712 Add last_timestamp parameter to MLFlow and Gates activities for enhanced tracking
- Introduced last_timestamp parameter in the MLFlow and Gates classes to improve tracking of data processing times.
- Updated MinioDataFramePayload to handle last_timestamp, ensuring it defaults to the maximum timestamp from the dataframe if not provided.
