Simplify merge in retrain: expose timestamp explicitly for join
Refactor evaluation_data merge to reset both frames' index into a
"timestamp" column before joining, instead of merging on index then
copying it out afterward. Same validate='one_to_one' safety check,
fewer intermediate steps.
Harden retrain merge and OPC write against silent failures
- pandas merge now uses validate='one_to_one' so duplicated timestamps
from wrapper.retrain fail loudly instead of silently multiplying rows
- OPC write_data raises OpcClientNotInitializedError instead of a bare
assert when client is torn down between validation and write, so
callers get a reported error instead of an AssertionError
- add regression test covering the torn-down client path
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fix flaky shutdown assertions in test suite
GC-collected instances during tests also invoke patched shutdown,
making assert_called_once_with flaky. Count calls for this instance
specifically instead of asserting total call count.
Pin sqlalchemy<2.1 to fix psycopg driver break; use tmp_path in retrain
- sqlalchemy 2.1 defaults postgresql:// URLs to psycopg (v3), but sientia_do
only ships psycopg2-binary, breaking with "No module named psycopg"
- Replace hardcoded 'tmp' mkdtemp mock with pytest tmp_path fixture in
retrain_model tests to avoid writing to a fake path
Fix quality-gate repo list: remove space, swap mlops-library for model-l
Correct repo list breaks scan otherwise: space in name mismatched dep, wrong package name (mlops-library doesn't exist / renamed to model-library).
Use requirements.txt for quality-gate scan instead of light variant
Switch dependency scan input from requirements-light.txt to requirements.txt
so quality gate checks full dependency set, not partial one.
Simplify bandit nosec comments and mock metrics_controller with real spe
- drift.py/simple_metrics.py: move nosec rationale to a code comment above
the query instead of an inline suppression comment, keeping bandit happy
with a plain `# nosec`
- tests: replace bare `MagicMock()` with `MagicMock(spec=MetricsController)`
for metrics_controller across activities/repository tests so mocks fail
fast on attribute typos instead of silently accepting any call
SIENTIAPDE-1646 Merge origin/main into release/SIENTIAPDE-1646
Bring main up to date while keeping this branch's sync execution, PluginStore
runtime and sientia_model (SientiaMLflowRepository/SientiaModel) structure.
Resolution decisions:
- OPC: keep the sync asyncua.sync implementation; port main's server_uri
validation, e2e race fix (lock held until released) and the new unit tests
converted to sync.
- Model import workflow (.sientia bundles), joblib/transform flavors and the
local MLFlowRepository (model_repository.py) are dropped, with their tests,
docs, metrics, env vars and connectors config.
- Worker: drift/simple_metrics stay pinned to the 'core' runtime; port main's
HOSTNAME pod id, logger-based prometheus startup and sys.exit after finally.
- values.yaml removed, following main (SIENTIAPDE-1945).
- Deps: keep sientia_model; take main's fsspec/s3fs/botocore/boto3 pins and
sientia_do>=1.12.2; drop import-only deps (pynacl, cryptography, pymongo,
python-dotenv, pycaret, joblib, gcsfs).
- e2e: keep sientia_data schema and scenario input files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Card 16 of the SIENTIAPDE-2052 epic. This repo is a Flow B dataops
worker, it does not build its own image at all anymore, it runs off
a shared runtime image with code cloned from Gitea at deploy time. A
per repo image build Makefile pointing at aignosi.azurecr.io is dead
weight, no workflow calls make. Safe to remove.
- Delete release.yml (tarball-only, no deploy job).
- Add deploy.yml calling reusable-deploy-dataops.yml, bumping only the
legacy runtime's GITHUB_BRANCH via the new profile/profile_env_list_path
mechanism — basic is left untouched for now. workflow_dispatch has no
inputs.
- Add require-chart-ref-label.yml.
QTZPOC-20: Improve worker shutdown handling by allowing SystemExit and KeyboardInterrupt to propagate directly, preserving their original exit codes and ensuring cleanup.
QTZPOC-20: Enable code_paths for sklearn models; improve worker shutdown handling
The MLFlowRepository now resolves and forwards code_paths to mlflow.sklearn.log_model, ensuring custom code dependencies for sklearn models are preserved and correctly reloaded. A new integration test validates this end-to-end functionality. Additionally, worker exception handling is refined to distinguish between deliberate shutdowns (SystemExit, KeyboardInterrupt) and unhandled exceptions, preventing unnecessary error logging for expected exits.
QTZPOC-20: Refactor ModelImport status engine for lazy initialization, add byte-counted size validation to bundle extraction, optimize MLflow run ID lookups, and ensure workflow cancellation propagates correctly.
