QTZPOC-20: Add skip_predict_timestamp flag to model_config to prevent timestamp reinsertion in prediction data, and document other ad-hoc behavioral flags.
Confirmed with Vitor: the trimmed dependency list (dropping
psycopg2-binary, sqlalchemy, redis, botocore, s3fs, pyarrow, kaleido,
hyperopt, shap, pycurl as explicit top-level pins) is intentional,
already tested against the cluster's basic runtime, and meant to land
on release/SIENTIAPDE-1646 -- it just hadn't been pushed to the real
GitHub branch yet (only to a Gitea test mirror), which is why the
merge in the previous commit restored the untrimmed list. Reverting
that restoration; keeps this branch's sientia_model>=0.12.0 floor and
explicit scikit-learn==1.8.0 pin.
Resolves the requirements.txt conflict: keeps the full dependency list
from release/SIENTIAPDE-1646 (the earlier trim commit here was based
on a Gitea-only mirror copy that had diverged from this real branch,
not an accurate source — reverted that assumption). Keeps this
branch's sientia_model>=0.12.0 floor (needed for RegressionMetrics)
and explicit scikit-learn==1.8.0 pin (release leaves it unpinned;
this branch's cluster testing showed an explicit pin avoids a version
mismatch with the runtime's own scikit-learn install).
Also mirrors the same resolution in requirements-local.txt for
consistency, keeping this branch's sientia-model-library@0.13.1 pin.
Vitor trimmed release/SIENTIAPDE-1646's requirements.txt to drop
explicit pins for packages already pulled transitively by sientia_do/
sientia_model (psycopg2-binary, sqlalchemy, redis, botocore, s3fs,
pyarrow, kaleido, hyperopt, shap, pycurl) — confirmed all of these
install anyway via sientia_do/sientia_model's own deps in the cluster
pod logs. Fewer redundant top-level pins means fewer chances for a
version mismatch like the scikit-learn one from earlier today.
Also bumps sientia_model floor to >=0.14.0 to match, and mirrors the
same trim in requirements-local.txt for local-dev parity — but keeps
this branch's sientia-model-library@0.13.1 git pin there (release's
own requirements-local.txt is stale at @0.10.0, predating the
RegressionMetrics class this card depends on).
The sientia-dev test cluster's "basic" plugin runtime declares
scikit-learn==1.8.0, installed on top of this repo's own
requirements.txt at container startup. The old scikit-learn==1.5.2
pin here got silently overwritten by that later install, but the
son-approx model's pickled LinearRegression wrapper was recreated
against a version needing sklearn 1.8's API — mismatched envs caused
`ImportError: cannot import name 'validate_data' from
sklearn.utils.validation` on model load (found while testing
SIENTIAPDE-1986 against the real cluster). Matching the pin here to
1.8.0 removes the redundant conflicting install.
RegressionMetrics rounds every metric to 2 decimals (documented in
lib.md); the old test compared against unrounded closed-form values
with a 1e-6 tolerance, which the rewritten activity can no longer
satisfy. Round the expected values the same way before comparing.
Confirmed via real Docker-backed Postgres/MinIO/Mongo + Temporal test
environment: all 4 simple_metrics e2e scenarios pass.
QTZPOC-20: Prioritize code/ directory download and sys.path injection before model loading, enabling MLflow flavors to stream models directly from URIs and simplifying joblib artifact handling.
QTZPOC-20: Unify artifact code directory handling for all model flavors and standardize joblib pickle path
Implemented a robust mechanism to ensure an artifact's code/ directory is prepended to sys.path before loading for sklearn, pyfunc, and joblib flavors. sklearn and pyfunc now resolve MLflow URIs to local directories first to facilitate this sys.path modification. Standardized the joblib flavor to explicitly read and write model.pkl within the artifact directory. Updated documentation and added comprehensive tests to reflect these changes.
QTZPOC-20: Improve structured logging and reporting for all model import pipeline failures by introducing a dedicated activity and centralizing error classification. This ensures consistent observability, with all workflow failures now reported before cleanup. Also, adjusted import step timeouts and refined manual run script cleanup logic.
QTZPOC-20: Implement explicit joblib model flavor and enhance model import workflow with extensive E2E tests.
This commit introduces a first-class joblib model flavor for direct loading and saving of Python pickle artifacts, including custom code within the artifact's code/ directory, bypassing MLflow's flavor loaders. The model import workflow's status database configuration is now streamlined to default to the POSTGRES_* server, with explicit IMPORT_STATUS_DB_* overrides.
Comprehensive end-to-end tests have been added for the model import workflow, covering success paths, gate rejections, digest enforcement, provisioning failures, status record integrity, cleanup, and retry/recovery scenarios using a real-clock environment. New E2
QTZPOC-20: Implement model import workflow for .sientia bundles
Introduces the import_model workflow and its associated activities to enable importing .sientia bundles into MLflow and MongoDB. This includes:
* New core workflow and activity logic for handling bundle download, decryption, validation, MLflow provisioning (experiments, runs, registered versions), and MongoDB model document creation.
* Comprehensive E2E and unit test suites, including a dedicated test harness and a manual run script for development.
* Updated .env.example and README.md with new configurations and usage instructions.
* Ensures correct MLflow model source resolution for legacy loader compatibility and robust MongoDB document management.
* Refactors existing utility scripts into a scripts/ directory.
QTZPOC-20: Implement .sientia model bundle import workflow
Introduces a new Temporal workflow and activities to import .sientia model bundles. This enables:
- Secure decryption and validation of bundles through a seven-gate process.
- Provisioning imported models in MLflow and updating the MongoDB model listing.
- Comprehensive status tracking in the public.experiment_run table with new IMPORT_SUCCESS/IMPORT_ERROR states.
- Dedicated metrics, robust error handling, and configurable settings.
- Includes updated e2e database fixtures and extensive unit/integration tests for bundle format, reader, and overall workflow.
QTZPOC-13: Centralize MLflow joblib fallback logic, ensuring consistent model version resolution by reusing the initial run ID, improving logging, and updating dependencies.
Compares each computed metric against optional per-model thresholds
from model_config.simple_metrics_thresholds. Convention:
- {metric}_max: breach when value > threshold (rmse, mse, mae)
- {metric}_min: breach when value < threshold (r2)
Fires WARNING notification on breach. Missing thresholds = no alerting.
Schema designed to be extensible for Card 2 (Drift) thresholds.
SIENTIAPDE-1986
Reads optional simple_metrics_thresholds from model_config and passes
to calculate_simple_metrics. Threshold schema: {rmse_max, r2_min, ...}.
None when not configured (no alerting, no crash).
Updates workflow tests to expect the new key in the activity-call dict.
SIENTIAPDE-1986
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrites 8 existing tests to mock RegressionMetrics instead of verifying
manual numpy math. Adds 2 new tests:
- r2 excluded + warning when model_type is non-linear
- r2 included when model_type is absent (backward compat)
Replaces silent-discard test with ValueError propagation test.
SIENTIAPDE-1986
Replaces manual numpy if/elif chain with RegressionMetrics from
sientia_model. Fixes 3 known bugs in one pass:
- NaN now handled via _align_dropna (was silently propagated)
- r2 zero-variance uses sklearn r2_score (was divergent, pinned at 0.0)
- Unknown metric names raise ValueError (were silently dropped)
Also adds r2 lock: when model_type is known and non-linear,
r2 is excluded from calculation with a WARNING notification.
Drops the now-unused `numpy` import (the manual math it backed is gone,
and no other method in this file references it).
SIENTIAPDE-1986
Extracts model_type from model_config (optional, defaults to None when
absent) and passes it to calculate_simple_metrics. Required for the r2
lock - is_r2_supported() needs model_type to decide whether to compute r2.
Updates workflow tests to expect the new key in the activity-call dict.
RegressionMetrics class (needed for SIENTIAPDE-1986) was introduced in
sientia-model-library 0.12.0. The previous local pin (@0.10.0) predates
the class.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
QTZPOC-13: Introduce joblib fallback for MLflow model loading
Enhances model loading resilience by attempting to deserialize models directly with joblib if MLflow's native flavor loaders fail. This mechanism handles cases where .pkl files are present in the artifact.
Adds joblib as a core dependency and includes new end-to-end and unit tests to validate fallback behavior, covering success, no-fallback, and double-failure scenarios. Also updates .gitignore for .claude/ files and incorporates minor refactorings and linting adjustments across E2E tests and utility functions.
Disables the unraisableexception plugin in pytest due to a known bug in
pytest>=9.1 where it crashes with tracemalloc errors when multiple
unraisable exceptions occur close together.
Renamed the OPC.close asynchronous method to OPC.aclose to align with common Python conventions for asynchronous context managers and methods, improving clarity. All call sites and tests have been updated accordingly.
SIENTIAPDE-1646 Refactor MLFlow tests and update artifact handling
- Enhanced test cases for MLFlow to improve clarity and accuracy in data handling.
- Updated references in tests to use 'evaluation_data.csv' and 'test_data.csv' instead of 'retrain_input.csv' and 'train_data.csv'.
- Introduced a new helper function for creating prediction frames to streamline test setup.
SIENTIAPDE-1646 Update prediction data handling in MLFlow
- Renamed the prediction column in the MLFlow retraining process to improve clarity and consistency in the output data.
SIENTIAPDE-1646 Enhance MLFlow retraining process and evaluation data handling
- Updated the MLFlow class to merge prediction data with retrain data for improved evaluation.
- Renamed target column to "target" and added a timestamp column to the evaluation data.
- Logged both retrain input and evaluation data as artifacts in MLFlow.
Enhance MLFlow reference data handling and testing
- Updated the artifact handling in MLFlow to prioritize 'retrain_input.csv' over 'train_data.csv' when resolving reference data.
- Introduced new methods for resolving artifact names and locating downloaded CSV files.
- Modified the `get_reference_data` method to improve artifact resolution and error handling.
- Expanded unit tests to cover scenarios for missing artifacts and preference logic between retrain and train data CSVs.
Update values.yaml to add MLFLOW environment variables
- Added MLFLOW_HOST and MLFLOW_PORT environment variables to values.yaml for tracking configuration.
Update values.yaml and connectors_config.py for runtime and environment variable changes
- Renamed the runtime entry from "single" to "legacy" in values.yaml for clarity.
- Updated the helm upgrade command comment in values.yaml to reflect the correct path for deployment.
- Changed the environment variable name from "MLFLOW_HOST" and "MLFLOW_PORT" to "MLFLOW_URL" in connectors_config.py for consistency.
Update values.yaml and worker.py for runtime installation logic
- Updated helm upgrade command version in values.yaml from 0.6.1 to 0.6.5 for deployment reference.
- Modified worker.py to adjust runtime installation logic, ensuring 'legacy' runtime is set to 'single' for compatibility.
Update values.yaml to change runtime configurations and GitHub branch
- Renamed runtime entries from list format to key-value pairs for "basic" and "single".
- Updated the GITHUB_BRANCH environment variable from "feature/SIENTIAPDE-1646-legacy-laborious-worker" to "main".
- Added a helm upgrade command comment for deployment reference.
