- Delete release.yml (tarball-only, no deploy job).
- Add deploy.yml calling reusable-deploy-dataops.yml, bumping only the
legacy runtime's GITHUB_BRANCH via the new profile/profile_env_list_path
mechanism — basic is left untouched for now. workflow_dispatch has no
inputs.
- Add require-chart-ref-label.yml.
QTZPOC-20: Improve worker shutdown handling by allowing SystemExit and KeyboardInterrupt to propagate directly, preserving their original exit codes and ensuring cleanup.
QTZPOC-20: Enable code_paths for sklearn models; improve worker shutdown handling
The MLFlowRepository now resolves and forwards code_paths to mlflow.sklearn.log_model, ensuring custom code dependencies for sklearn models are preserved and correctly reloaded. A new integration test validates this end-to-end functionality. Additionally, worker exception handling is refined to distinguish between deliberate shutdowns (SystemExit, KeyboardInterrupt) and unhandled exceptions, preventing unnecessary error logging for expected exits.
QTZPOC-20: Refactor ModelImport status engine for lazy initialization, add byte-counted size validation to bundle extraction, optimize MLflow run ID lookups, and ensure workflow cancellation propagates correctly.
QTZPOC-20: Refactor model import workflow, introduce atomic model ID generation, and clarify MinIO expiry.
Fuses bundle download, decryption, MLflow experiment creation, and artifact upload into a single activity to prevent cross-worker issues. Implements an atomic counter for MongoDB model IDs, replacing the max(id) + 1 scan to avoid collisions during concurrent imports. Adds comprehensive documentation for MinIO object expiry mechanisms and refines lifecycle rule application. Improves error handling with clearer responsibility classification and centralizes MLflow repository injection.
QTZPOC-20: Enhance system robustness and test coverage across bundle processing, activity error handling, and worker operations.
* Strengthen bundle reader validation with new tests for archive integrity, security vulnerabilities (e.g., path traversal, duplicate entries, special file types), and I/O error resilience.
* Improve error handling and status propagation in PI Web API, MLflow, and OPC activities, including partial write scenarios and OPC session management.
* Add comprehensive tests for worker startup, configuration, graceful shutdown, and Prometheus metrics server initialization.
* Introduce a new utility for intelligent DataFrame debugging, summarizing large dataframes for concise logging, and refine exception chain summarization.
* Update code coverage configuration to exclude @overload decorator lines.
QTZPOC-20: Refactor model repository tests for improved assertion logic and exception handling, including correcting pytest.raises exception access and refining mock verifications.
QTZPOC-20: Add skip_predict_timestamp flag to model_config to prevent timestamp reinsertion in prediction data, and document other ad-hoc behavioral flags.
Confirmed with Vitor: the trimmed dependency list (dropping
psycopg2-binary, sqlalchemy, redis, botocore, s3fs, pyarrow, kaleido,
hyperopt, shap, pycurl as explicit top-level pins) is intentional,
already tested against the cluster's basic runtime, and meant to land
on release/SIENTIAPDE-1646 -- it just hadn't been pushed to the real
GitHub branch yet (only to a Gitea test mirror), which is why the
merge in the previous commit restored the untrimmed list. Reverting
that restoration; keeps this branch's sientia_model>=0.12.0 floor and
explicit scikit-learn==1.8.0 pin.
Resolves the requirements.txt conflict: keeps the full dependency list
from release/SIENTIAPDE-1646 (the earlier trim commit here was based
on a Gitea-only mirror copy that had diverged from this real branch,
not an accurate source — reverted that assumption). Keeps this
branch's sientia_model>=0.12.0 floor (needed for RegressionMetrics)
and explicit scikit-learn==1.8.0 pin (release leaves it unpinned;
this branch's cluster testing showed an explicit pin avoids a version
mismatch with the runtime's own scikit-learn install).
Also mirrors the same resolution in requirements-local.txt for
consistency, keeping this branch's sientia-model-library@0.13.1 pin.
Vitor trimmed release/SIENTIAPDE-1646's requirements.txt to drop
explicit pins for packages already pulled transitively by sientia_do/
sientia_model (psycopg2-binary, sqlalchemy, redis, botocore, s3fs,
pyarrow, kaleido, hyperopt, shap, pycurl) — confirmed all of these
install anyway via sientia_do/sientia_model's own deps in the cluster
pod logs. Fewer redundant top-level pins means fewer chances for a
version mismatch like the scikit-learn one from earlier today.
Also bumps sientia_model floor to >=0.14.0 to match, and mirrors the
same trim in requirements-local.txt for local-dev parity — but keeps
this branch's sientia-model-library@0.13.1 git pin there (release's
own requirements-local.txt is stale at @0.10.0, predating the
RegressionMetrics class this card depends on).
The sientia-dev test cluster's "basic" plugin runtime declares
scikit-learn==1.8.0, installed on top of this repo's own
requirements.txt at container startup. The old scikit-learn==1.5.2
pin here got silently overwritten by that later install, but the
son-approx model's pickled LinearRegression wrapper was recreated
against a version needing sklearn 1.8's API — mismatched envs caused
`ImportError: cannot import name 'validate_data' from
sklearn.utils.validation` on model load (found while testing
SIENTIAPDE-1986 against the real cluster). Matching the pin here to
1.8.0 removes the redundant conflicting install.
RegressionMetrics rounds every metric to 2 decimals (documented in
lib.md); the old test compared against unrounded closed-form values
with a 1e-6 tolerance, which the rewritten activity can no longer
satisfy. Round the expected values the same way before comparing.
Confirmed via real Docker-backed Postgres/MinIO/Mongo + Temporal test
environment: all 4 simple_metrics e2e scenarios pass.
QTZPOC-20: Prioritize code/ directory download and sys.path injection before model loading, enabling MLflow flavors to stream models directly from URIs and simplifying joblib artifact handling.
QTZPOC-20: Unify artifact code directory handling for all model flavors and standardize joblib pickle path
Implemented a robust mechanism to ensure an artifact's code/ directory is prepended to sys.path before loading for sklearn, pyfunc, and joblib flavors. sklearn and pyfunc now resolve MLflow URIs to local directories first to facilitate this sys.path modification. Standardized the joblib flavor to explicitly read and write model.pkl within the artifact directory. Updated documentation and added comprehensive tests to reflect these changes.
QTZPOC-20: Improve structured logging and reporting for all model import pipeline failures by introducing a dedicated activity and centralizing error classification. This ensures consistent observability, with all workflow failures now reported before cleanup. Also, adjusted import step timeouts and refined manual run script cleanup logic.
QTZPOC-20: Implement explicit joblib model flavor and enhance model import workflow with extensive E2E tests.
This commit introduces a first-class joblib model flavor for direct loading and saving of Python pickle artifacts, including custom code within the artifact's code/ directory, bypassing MLflow's flavor loaders. The model import workflow's status database configuration is now streamlined to default to the POSTGRES_* server, with explicit IMPORT_STATUS_DB_* overrides.
Comprehensive end-to-end tests have been added for the model import workflow, covering success paths, gate rejections, digest enforcement, provisioning failures, status record integrity, cleanup, and retry/recovery scenarios using a real-clock environment. New E2
QTZPOC-20: Implement model import workflow for .sientia bundles
Introduces the import_model workflow and its associated activities to enable importing .sientia bundles into MLflow and MongoDB. This includes:
* New core workflow and activity logic for handling bundle download, decryption, validation, MLflow provisioning (experiments, runs, registered versions), and MongoDB model document creation.
* Comprehensive E2E and unit test suites, including a dedicated test harness and a manual run script for development.
* Updated .env.example and README.md with new configurations and usage instructions.
* Ensures correct MLflow model source resolution for legacy loader compatibility and robust MongoDB document management.
* Refactors existing utility scripts into a scripts/ directory.
QTZPOC-20: Implement .sientia model bundle import workflow
Introduces a new Temporal workflow and activities to import .sientia model bundles. This enables:
- Secure decryption and validation of bundles through a seven-gate process.
- Provisioning imported models in MLflow and updating the MongoDB model listing.
- Comprehensive status tracking in the public.experiment_run table with new IMPORT_SUCCESS/IMPORT_ERROR states.
- Dedicated metrics, robust error handling, and configurable settings.
- Includes updated e2e database fixtures and extensive unit/integration tests for bundle format, reader, and overall workflow.
QTZPOC-13: Centralize MLflow joblib fallback logic, ensuring consistent model version resolution by reusing the initial run ID, improving logging, and updating dependencies.
Compares each computed metric against optional per-model thresholds
from model_config.simple_metrics_thresholds. Convention:
- {metric}_max: breach when value > threshold (rmse, mse, mae)
- {metric}_min: breach when value < threshold (r2)
Fires WARNING notification on breach. Missing thresholds = no alerting.
Schema designed to be extensible for Card 2 (Drift) thresholds.
SIENTIAPDE-1986
Reads optional simple_metrics_thresholds from model_config and passes
to calculate_simple_metrics. Threshold schema: {rmse_max, r2_min, ...}.
None when not configured (no alerting, no crash).
Updates workflow tests to expect the new key in the activity-call dict.
SIENTIAPDE-1986
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrites 8 existing tests to mock RegressionMetrics instead of verifying
manual numpy math. Adds 2 new tests:
- r2 excluded + warning when model_type is non-linear
- r2 included when model_type is absent (backward compat)
Replaces silent-discard test with ValueError propagation test.
SIENTIAPDE-1986
