Simplify merge in retrain: expose timestamp explicitly for join
Refactor evaluation_data merge to reset both frames' index into a
"timestamp" column before joining, instead of merging on index then
copying it out afterward. Same validate='one_to_one' safety check,
fewer intermediate steps.
Harden retrain merge and OPC write against silent failures
- pandas merge now uses validate='one_to_one' so duplicated timestamps
from wrapper.retrain fail loudly instead of silently multiplying rows
- OPC write_data raises OpcClientNotInitializedError instead of a bare
assert when client is torn down between validation and write, so
callers get a reported error instead of an AssertionError
- add regression test covering the torn-down client path
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fix flaky shutdown assertions in test suite
GC-collected instances during tests also invoke patched shutdown,
making assert_called_once_with flaky. Count calls for this instance
specifically instead of asserting total call count.
Pin sqlalchemy<2.1 to fix psycopg driver break; use tmp_path in retrain
- sqlalchemy 2.1 defaults postgresql:// URLs to psycopg (v3), but sientia_do
only ships psycopg2-binary, breaking with "No module named psycopg"
- Replace hardcoded 'tmp' mkdtemp mock with pytest tmp_path fixture in
retrain_model tests to avoid writing to a fake path
Fix quality-gate repo list: remove space, swap mlops-library for model-l
Correct repo list breaks scan otherwise: space in name mismatched dep, wrong package name (mlops-library doesn't exist / renamed to model-library).
Use requirements.txt for quality-gate scan instead of light variant
Switch dependency scan input from requirements-light.txt to requirements.txt
so quality gate checks full dependency set, not partial one.
Simplify bandit nosec comments and mock metrics_controller with real spe
- drift.py/simple_metrics.py: move nosec rationale to a code comment above
the query instead of an inline suppression comment, keeping bandit happy
with a plain `# nosec`
- tests: replace bare `MagicMock()` with `MagicMock(spec=MetricsController)`
for metrics_controller across activities/repository tests so mocks fail
fast on attribute typos instead of silently accepting any call
SIENTIAPDE-1646 Merge origin/main into release/SIENTIAPDE-1646
Bring main up to date while keeping this branch's sync execution, PluginStore
runtime and sientia_model (SientiaMLflowRepository/SientiaModel) structure.
Resolution decisions:
- OPC: keep the sync asyncua.sync implementation; port main's server_uri
validation, e2e race fix (lock held until released) and the new unit tests
converted to sync.
- Model import workflow (.sientia bundles), joblib/transform flavors and the
local MLFlowRepository (model_repository.py) are dropped, with their tests,
docs, metrics, env vars and connectors config.
- Worker: drift/simple_metrics stay pinned to the 'core' runtime; port main's
HOSTNAME pod id, logger-based prometheus startup and sys.exit after finally.
- values.yaml removed, following main (SIENTIAPDE-1945).
- Deps: keep sientia_model; take main's fsspec/s3fs/botocore/boto3 pins and
sientia_do>=1.12.2; drop import-only deps (pynacl, cryptography, pymongo,
python-dotenv, pycaret, joblib, gcsfs).
- e2e: keep sientia_data schema and scenario input files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Card 16 of the SIENTIAPDE-2052 epic. This repo is a Flow B dataops
worker, it does not build its own image at all anymore, it runs off
a shared runtime image with code cloned from Gitea at deploy time. A
per repo image build Makefile pointing at aignosi.azurecr.io is dead
weight, no workflow calls make. Safe to remove.
- Delete release.yml (tarball-only, no deploy job).
- Add deploy.yml calling reusable-deploy-dataops.yml, bumping only the
legacy runtime's GITHUB_BRANCH via the new profile/profile_env_list_path
mechanism — basic is left untouched for now. workflow_dispatch has no
inputs.
- Add require-chart-ref-label.yml.
QTZPOC-20: Improve worker shutdown handling by allowing SystemExit and KeyboardInterrupt to propagate directly, preserving their original exit codes and ensuring cleanup.
QTZPOC-20: Enable code_paths for sklearn models; improve worker shutdown handling
The MLFlowRepository now resolves and forwards code_paths to mlflow.sklearn.log_model, ensuring custom code dependencies for sklearn models are preserved and correctly reloaded. A new integration test validates this end-to-end functionality. Additionally, worker exception handling is refined to distinguish between deliberate shutdowns (SystemExit, KeyboardInterrupt) and unhandled exceptions, preventing unnecessary error logging for expected exits.
QTZPOC-20: Refactor ModelImport status engine for lazy initialization, add byte-counted size validation to bundle extraction, optimize MLflow run ID lookups, and ensure workflow cancellation propagates correctly.
QTZPOC-20: Refactor model import workflow, introduce atomic model ID generation, and clarify MinIO expiry.
Fuses bundle download, decryption, MLflow experiment creation, and artifact upload into a single activity to prevent cross-worker issues. Implements an atomic counter for MongoDB model IDs, replacing the max(id) + 1 scan to avoid collisions during concurrent imports. Adds comprehensive documentation for MinIO object expiry mechanisms and refines lifecycle rule application. Improves error handling with clearer responsibility classification and centralizes MLflow repository injection.
QTZPOC-20: Enhance system robustness and test coverage across bundle processing, activity error handling, and worker operations.
* Strengthen bundle reader validation with new tests for archive integrity, security vulnerabilities (e.g., path traversal, duplicate entries, special file types), and I/O error resilience.
* Improve error handling and status propagation in PI Web API, MLflow, and OPC activities, including partial write scenarios and OPC session management.
* Add comprehensive tests for worker startup, configuration, graceful shutdown, and Prometheus metrics server initialization.
* Introduce a new utility for intelligent DataFrame debugging, summarizing large dataframes for concise logging, and refine exception chain summarization.
* Update code coverage configuration to exclude @overload decorator lines.
QTZPOC-20: Refactor model repository tests for improved assertion logic and exception handling, including correcting pytest.raises exception access and refining mock verifications.
QTZPOC-20: Add skip_predict_timestamp flag to model_config to prevent timestamp reinsertion in prediction data, and document other ad-hoc behavioral flags.
