Requirements follow the release; only sientia_model>=0.12.0 (RegressionMetrics),
model-library@0.13.1 and scikit-learn==1.8.0 (local) are kept from this branch.
Pin sqlalchemy<2.1 to fix psycopg driver break; use tmp_path in retrain
- sqlalchemy 2.1 defaults postgresql:// URLs to psycopg (v3), but sientia_do
only ships psycopg2-binary, breaking with "No module named psycopg"
- Replace hardcoded 'tmp' mkdtemp mock with pytest tmp_path fixture in
retrain_model tests to avoid writing to a fake path
SIENTIAPDE-1646 Merge origin/main into release/SIENTIAPDE-1646
Bring main up to date while keeping this branch's sync execution, PluginStore
runtime and sientia_model (SientiaMLflowRepository/SientiaModel) structure.
Resolution decisions:
- OPC: keep the sync asyncua.sync implementation; port main's server_uri
validation, e2e race fix (lock held until released) and the new unit tests
converted to sync.
- Model import workflow (.sientia bundles), joblib/transform flavors and the
local MLFlowRepository (model_repository.py) are dropped, with their tests,
docs, metrics, env vars and connectors config.
- Worker: drift/simple_metrics stay pinned to the 'core' runtime; port main's
HOSTNAME pod id, logger-based prometheus startup and sys.exit after finally.
- values.yaml removed, following main (SIENTIAPDE-1945).
- Deps: keep sientia_model; take main's fsspec/s3fs/botocore/boto3 pins and
sientia_do>=1.12.2; drop import-only deps (pynacl, cryptography, pymongo,
python-dotenv, pycaret, joblib, gcsfs).
- e2e: keep sientia_data schema and scenario input files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Confirmed with Vitor: the trimmed dependency list (dropping
psycopg2-binary, sqlalchemy, redis, botocore, s3fs, pyarrow, kaleido,
hyperopt, shap, pycurl as explicit top-level pins) is intentional,
already tested against the cluster's basic runtime, and meant to land
on release/SIENTIAPDE-1646 -- it just hadn't been pushed to the real
GitHub branch yet (only to a Gitea test mirror), which is why the
merge in the previous commit restored the untrimmed list. Reverting
that restoration; keeps this branch's sientia_model>=0.12.0 floor and
explicit scikit-learn==1.8.0 pin.
Resolves the requirements.txt conflict: keeps the full dependency list
from release/SIENTIAPDE-1646 (the earlier trim commit here was based
on a Gitea-only mirror copy that had diverged from this real branch,
not an accurate source — reverted that assumption). Keeps this
branch's sientia_model>=0.12.0 floor (needed for RegressionMetrics)
and explicit scikit-learn==1.8.0 pin (release leaves it unpinned;
this branch's cluster testing showed an explicit pin avoids a version
mismatch with the runtime's own scikit-learn install).
Also mirrors the same resolution in requirements-local.txt for
consistency, keeping this branch's sientia-model-library@0.13.1 pin.
Vitor trimmed release/SIENTIAPDE-1646's requirements.txt to drop
explicit pins for packages already pulled transitively by sientia_do/
sientia_model (psycopg2-binary, sqlalchemy, redis, botocore, s3fs,
pyarrow, kaleido, hyperopt, shap, pycurl) — confirmed all of these
install anyway via sientia_do/sientia_model's own deps in the cluster
pod logs. Fewer redundant top-level pins means fewer chances for a
version mismatch like the scikit-learn one from earlier today.
Also bumps sientia_model floor to >=0.14.0 to match, and mirrors the
same trim in requirements-local.txt for local-dev parity — but keeps
this branch's sientia-model-library@0.13.1 git pin there (release's
own requirements-local.txt is stale at @0.10.0, predating the
RegressionMetrics class this card depends on).
The sientia-dev test cluster's "basic" plugin runtime declares
scikit-learn==1.8.0, installed on top of this repo's own
requirements.txt at container startup. The old scikit-learn==1.5.2
pin here got silently overwritten by that later install, but the
son-approx model's pickled LinearRegression wrapper was recreated
against a version needing sklearn 1.8's API — mismatched envs caused
`ImportError: cannot import name 'validate_data' from
sklearn.utils.validation` on model load (found while testing
SIENTIAPDE-1986 against the real cluster). Matching the pin here to
1.8.0 removes the redundant conflicting install.
QTZPOC-20: Implement model import workflow for .sientia bundles
Introduces the import_model workflow and its associated activities to enable importing .sientia bundles into MLflow and MongoDB. This includes:
* New core workflow and activity logic for handling bundle download, decryption, validation, MLflow provisioning (experiments, runs, registered versions), and MongoDB model document creation.
* Comprehensive E2E and unit test suites, including a dedicated test harness and a manual run script for development.
* Updated .env.example and README.md with new configurations and usage instructions.
* Ensures correct MLflow model source resolution for legacy loader compatibility and robust MongoDB document management.
* Refactors existing utility scripts into a scripts/ directory.
QTZPOC-20: Implement .sientia model bundle import workflow
Introduces a new Temporal workflow and activities to import .sientia model bundles. This enables:
- Secure decryption and validation of bundles through a seven-gate process.
- Provisioning imported models in MLflow and updating the MongoDB model listing.
- Comprehensive status tracking in the public.experiment_run table with new IMPORT_SUCCESS/IMPORT_ERROR states.
- Dedicated metrics, robust error handling, and configurable settings.
- Includes updated e2e database fixtures and extensive unit/integration tests for bundle format, reader, and overall workflow.
RegressionMetrics class (needed for SIENTIAPDE-1986) was introduced in
sientia-model-library 0.12.0. The previous local pin (@0.10.0) predates
the class.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
QTZPOC-13: Introduce joblib fallback for MLflow model loading
Enhances model loading resilience by attempting to deserialize models directly with joblib if MLflow's native flavor loaders fail. This mechanism handles cases where .pkl files are present in the artifact.
Adds joblib as a core dependency and includes new end-to-end and unit tests to validate fallback behavior, covering success, no-fallback, and double-failure scenarios. Also updates .gitignore for .claude/ files and incorporates minor refactorings and linting adjustments across E2E tests and utility functions.
Update requirements and enhance OPC communication handling
- Updated `requirements-local.txt` to use the latest versions of `sientia-dataops-library` (1.12.1) and `sientia-model-library` (0.10.0).
- Improved documentation in `opc-communication.md` to clarify the transition from async to sync implementation and added details on error classification and reconnection behavior.
- Refactored `init_opc` method in `opc.py` to enhance connection handling and logging, ensuring independent server initialization.
- Enhanced validation and writing methods in `opc.py` to provide better feedback and error handling for OPC server operations.
- Updated `opc_repository.py` to improve error payload construction and session management metrics.
SIENTIAPDE-1646 Update dependencies and refactor OPC integration
- Replaced `opcua` with `asyncua` in `requirements-local.txt` and `requirements.txt` to utilize the async capabilities.
- Updated `values.yaml` to change the GitHub branch from `feature/SIENTIAPDE-1646` to `release/SIENTIAPDE-1646`.
- Refactored `opc.py` and `opc_repository.py` to accommodate the new `asyncua` library, ensuring compatibility with the synchronous API.
- Adjusted tests in `test_opc_repository.py` to reflect changes in the client implementation and maintain functionality.
Update README, requirements, and E2E tests for improved configuration and functionality
- Enhanced the README with updated model configuration examples, including the addition of an alias for production.
- Removed the `requirements-light.txt` file and updated `requirements-local.txt` and `requirements.txt` to replace `asyncua` with `opcua`.
- Refactored E2E test scenarios to utilize scenario input files for better maintainability and clarity.
- Improved test coverage for MinIO offload functionality and added new helper functions for loading scenario inputs.
- Updated `values.yaml` to reflect new global configurations and environment variables for the laborious worker.
Update dependencies and refactor MLFlow activities
- Replaced direct GitHub dependencies in `requirements.txt` with specific versioned packages for `sientia_do` and `sientia_model`.
- Refactored imports in `activities.py` to streamline the code structure.
- Enhanced the `MLFlow` class in `mlflow.py` by introducing a method to resolve model aliases, improving flexibility in model lookups.
- Simplified shutdown logic in `worker.py` for better readability.
- Added new tests for MLFlow activities and improved existing test coverage for data handling and model retraining processes.
