SIENTIAPDE-1646

SIENTIAPDE-1646 Refactor MLFlow tests and update artifact handling

- Enhanced test cases for MLFlow to improve clarity and accuracy in data handling.
- Updated references in tests to use 'evaluation_data.csv' and 'test_data.csv' instead of 'retrain_input.csv' and 'train_data.csv'.
- Introduced a new helper function for creating prediction frames to streamline test setup.

.env.example

@@ -11,6 +11,22 @@ MLFLOW_PORT="80"
 MLFLOW_USERNAME="aignosi"
 MLFLOW_PASSWORD="mlflow_password"
 
+# Worker runtime name for PluginStore.install_runtime (PredictionsBatch / MinimalRetrain workers).
+RUNTIME="single"
+
+# Plugin store (Git-backed catalog + runtime install).
+STORE_BASE_URL="http://gitea.sientia.svc.cluster.local:3000"
+STORE_OWNER="sientia"
+STORE_REPO="model-library-store"
+STORE_BRANCH="main"
+STORE_USERNAME=""
+STORE_PASSWORD=""
+STORE_CACHE_TTL_SECONDS=""
+
+PYPI_SERVER="http://library-distribution-server.library.svc.cluster.local:5000"
+PYPI_USERNAME=""
+PYPI_PASSWORD=""
+
 OPC_ID="1"
 OPC_URL="opc.tcp://sientia-opc-simulator-opc.sientia.svc.cluster.local:4840"
 

.gitignore

@@ -52,3 +52,7 @@ catboost_info/
 .ruff_cache/
 .mypy_cache/
 mlruns/
+
+relatorio*
+openspec/*
+.cursor/*

README.md

@@ -48,6 +48,7 @@ A comprehensive, Temporal-based ML orchestration system for industrial data proc
   - [Prediction Operation Metrics](#prediction-operation-metrics)
   - [OPC Export Metrics](#opc-export-metrics)
   - [Data Quality Metrics](#data-quality-metrics)
+- [OPC UA Communication](#opc-ua-communication)
 - [Configuration](#configuration-1)
   - [Environment Variables](#environment-variables)
   - [OPC Configuration](#opc-configuration)
@@ -163,8 +164,8 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
 #### **Data Services (`laborious/utils/`)**
 - `connectors_config.py`: Env-driven configuration builders
 - `models/minio_dataframe_payload.py`: MinIO-offloaded DataFrame payload model
-- `repository/model_repository.py`: MLFlow operations and retraining
+- ML models are loaded via `SientiaMLflowRepository` (wrapper-based, `@production` alias) constructed in `Activities` from `build_mlflow_config()`.
-- `repository/opc_repository.py`: OPC communication and writes
+- `repository/opc_repository.py`: OPC UA client, writes, session recovery (see [OPC UA Communication](#opc-ua-communication))
 - `repository/minio_manager.py`: MinIO object storage operations
 - `filters/conditional_filters.py` and `filters/mlflow_filters.py`
 
@@ -463,7 +464,7 @@ flowchart LR
   "source_table_name": "laborious_data",
   "target_table_name": "drift_metrics",
   "interval": 60,
-  "model_config": { "target": "temperature" },
+  "model_config": { "target": "temperature", "alias": "production" },
   "drift_metrics": ["kolmogorov_smirnov", "jensen_shannon", "wasserstein"],
   "chunk_period": "min"
 }
@@ -498,7 +499,7 @@ flowchart LR
   "data_table_name": "laborious_data",
   "target_table_name": "simple_metrics",
   "interval_minutes": 60,
-  "model_config": { "target": "temperature" },
+  "model_config": { "target": "temperature", "alias": "production" },
   "metrics": ["rmse", "mse", "mae", "r2"]
 }
 ```
@@ -730,7 +731,6 @@ tests/
 │       │   ├── test_conditional_filters.py
 │       │   └── test_mlflow_filters.py
 │       └── repository/
-│           ├── test_model_repository.py
 │           └── test_opc_repository.py
 ```
 
@@ -779,6 +779,15 @@ The Laborious system exposes comprehensive Prometheus metrics for operational vi
   - Labels: `pod_id`, `model_name`, `workflow_name`, `opc_server_id`
   - Buckets: [0.01, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0]
 
+OPC UA session and write diagnostics (Prometheus, `laborious/metrics.py`):
+
+- `opc_connections_initiated_total`, `opc_connections_failed_total`, `opc_connection_status`
+- `opc_session_created_total`, `opc_session_closed_total`, `opc_session_revised_timeout_milliseconds`
+- `opc_write_attempts_total` (label `result`: `OK` or exception name, e.g. `BadSessionIdInvalid`)
+- `opc_write_inter_arrival_over_session_timeout_total`
+
+See [OPC UA Communication](#opc-ua-communication) for semantics, concurrency, and confidence codes **12** / **14**.
+
 ### Data Quality Metrics
 - Filter pass/fail rates through notification system
 - MLFlow API response validation metrics
@@ -803,14 +812,27 @@ The Laborious system exposes comprehensive Prometheus metrics for operational vi
 | `MLFLOW_PORT` | MLFlow server port | `5080` | Yes |
 | `MLFLOW_USERNAME` | MLFlow username | `aignosi` | Yes |
 | `MLFLOW_PASSWORD` | MLFlow password | `aignosi` | Yes |
+| `RUNTIME` | Plugin store runtime name installed at worker boot (required for model-loading workers) | `single` | Yes* |
+| `STORE_BASE_URL` | Plugin store Git server base URL | `http://localhost:3000` | Yes* |
+| `STORE_OWNER` | Plugin store repository owner | `sientia` | Yes* |
+| `STORE_REPO` | Plugin store repository name | `model-library-store` | Yes* |
+| `STORE_BRANCH` | Optional branch for the store repository | `main` | No |
+| `STORE_USERNAME` | HTTP username for the Git store | `None` | No |
+| `STORE_PASSWORD` | HTTP password/token for the Git store | `None` | No |
+| `STORE_CACHE_TTL_SECONDS` | Optional cache TTL for store metadata | `None` | No |
+| `PYPI_SERVER` | Private PyPI index URL for runtime wheels | `http://localhost:5000` | Yes* |
+| `PYPI_USERNAME` | Optional PyPI basic-auth username | `None` | No |
+| `PYPI_PASSWORD` | Optional PyPI basic-auth password | `None` | No |
 | `OPC_CONFIG` | OPC server configuration (JSON) | `{}` | No |
+
+\* `RUNTIME`, PluginStore (`STORE_*`), and `PYPI_SERVER` are required for workers that install a runtime and load `SientiaModel` wrappers (`PredictionsBatch`, `MinimalRetrain`). Workers that only run drift/simple-metrics style jobs may omit them when those workflows are deployed separately.
 | `OPC_ID` | OPC server identifier | `1` | No |
 | `OPC_URL` | OPC server URL | `opc.tcp://localhost:4840` | No |
 | `OPC_SERVER_URI` | OPC server URI | `opc.tcp://localhost:4840` | No |
 | `OPC_CERT_PATH` | OPC client certificate path | `None` | No |
 | `OPC_PRIVATE_KEY_PATH` | OPC private key path | `None` | No |
 | `OPC_SERVER_CERT_PATH` | OPC server certificate path | `None` | No |
-| `OPC_RECONNECTION_INTERVAL` | OPC reconnection interval (ms) | `120` | No |
+| `OPC_RECONNECTION_INTERVAL` | Minimum seconds between OPC reconnects | `120` | No |
 | `PI_WEB_API_BASE_URL` | PI Web API server base URL | `None` | No |
 | `PI_WEB_API_AUTH_TYPE` | PI Web API authentication type (basic/bearer) | `None` | No |
 | `PI_WEB_API_AUTH_TOKEN` | PI Web API authentication token | `None` | No |
@@ -903,6 +925,12 @@ Legacy MinIO object layout (relative key):
 `training_datasets/{model_name}/{object_prefix}_{timestamp}.parquet` where `object_prefix` is sanitized
 (slashes replaced by underscores) to keep a stable model-level directory.
 
+## OPC UA Communication
+
+Full reference: **[docs/opc-communication.md](docs/opc-communication.md)** (connection lifecycle, Tier-1 `Bad*` reconnect, connection lock / session readiness, metrics, PostgreSQL confidence **12** vs **14**, tests).
+
+Implementation plan: [`.cursor/plans/opc_bad_reconnect_ac4c6045.plan.md`](.cursor/plans/opc_bad_reconnect_ac4c6045.plan.md).
+
 ### OPC Configuration
 
 For multiple OPC servers, use the `OPC_CONFIG` environment variable:
@@ -941,7 +969,7 @@ For single OPC server, use individual environment variables:
 
 ### PI Web API Configuration
 
-PI Web API configuration is built from environment variables using the `build_api_config` function from `sientia_do.connectors_config`. The configuration includes:
+PI Web API configuration is built from environment variables using the `build_api_config` function from `sientia_do.utils.connectors_config`. The configuration includes:
 
 - `PI_WEB_API_BASE_URL`: Base URL of the PI Web API server
 - `PI_WEB_API_AUTH_TYPE`: Authentication type ('basic' or 'bearer')
@@ -974,9 +1002,30 @@ Where:
 
 MongoDB pipeline configuration:
 
-#### Predictions Batch Workflow configuration sample
+#### MongoDB input samples (updated)
 
-This is the configuration for the Predictions Batch Workflow, to be inserted into the MongoDB pipeline collection.
+Updated examples are available in `input_sample.json` at the repository root.
+The sample already reflects the runtime-aware and alias-based flow:
+
+- `model_config` uses `target`, `retention_minutes`, and `alias`.
+- `transform_flavor` / `predict_flavor` are not used anymore.
+
+Example model document:
+
+```json
+{
+  "id": "4",
+  "name": "vcm-nox",
+  "active": false,
+  "model_config": {
+    "alias": "production",
+    "retention_minutes": 60,
+    "target": "CI-W3W01A3"
+  }
+}
+```
+
+Example predictions_batch schedule document:
 
 ```json
 {
@@ -986,41 +1035,67 @@ This is the configuration for the Predictions Batch Workflow, to be inserted int
   "frequency": "30s",
   "max_retry_policy": 1,
   "query": "select * from sientia_data.laborious_data where model_id = 1 and \"timestamp\" > NOW() - INTERVAL '5 minutes' order by \"timestamp\" desc limit 30;",
+  "retention_time": 60,
   "write_tags": [
     {
-      "server_id": "server1",
+      "server_id": "1",
       "type": "prediction",
       "addr": "ns=2;i=5",
       "data_type": "double"
     },
     {
-      "server_id": "server1",
+      "server_id": "1",
       "type": "confidence",
-      "addr": "ns=2;i=6",
+      "addr": "ns=2;i=5",
       "data_type": "double"
     }
   ],
-  "input_filters": {
+  "input_filters": [
-    "EMPTY_DATA": {"POLICY": "STOP"},
+    {
-    "SPECIFIC_VARIABLES_NULL_VALUES": {
+      "filter_name": "EMPTY_DATA",
-      "POLICY": "CONTINUE",
+      "policy": "STOP"
-      "config": {"variables": ["Counter"]}
+    },
+    {
+      "filter_name": "SPECIFIC_VARIABLES_NULL_VALUES",
+      "policy": "CONTINUE",
+      "config": {
+        "variables": ["Counter"]
       }
+    }
+  ],
+  "mlflow_transform_filters": [
+    {
+      "filter_name": "API_ERROR",
+      "policy": "REPEAT"
     },
-  "mlflow_transform_filters": {
+    {
-    "API_ERROR": {"POLICY": "REPEAT"},
+      "filter_name": "NAN_VALUES",
-    "NAN_VALUES": {"POLICY": "STOP"}
+      "policy": "STOP"
-  },
+    }
-  "mlflow_predict_filters": {
+  ],
-    "API_ERROR": {"POLICY": "CONTINUE"}
+  "mlflow_predict_filters": [
-  },
+    {
+      "filter_name": "API_ERROR",
+      "policy": "CONTINUE"
+    }
+  ],
   "path_priority": ["STOP", "CONTINUE", "REPEAT"],
   "active": true,
   "updated_at": {
-    "$date": "2025-09-16T10:00:00.000Z"
+    "$date": "2026-01-27T17:35:01.600Z"
   },
-  "datetime_columns": ["timestamp", "created_at"],
+  "save_transform": false,
-  "predictions_storage_policy": "lts:1"
+  "pi_web_api_output_config": {
+    "endpoint": "/streamsets/value",
+    "prediction_tags": {},
+    "confidence_tags": {}
+  },
+  "model_config": {
+    "alias": "production",
+    "retention_minutes": 60,
+    "target": "CI-W3W01A3"
+  },
+  "datetime_columns": ["timestamp", "created_at"]
 }
 ```
 
@@ -1038,11 +1113,8 @@ This is the configuration created by the Orchestrator in Temporal.
     "EMPTY_DATA":{"config":{},"policy":"STOP"}
   },
   "model_config":{
-    "is_compressed":true,
+    "target":"sensor_or_label_column",
-    "predict_flavor":"pyfunc",
+    "retention_minutes":60
-    "retention_minutes":60,
-    "retention_target":"artifact",
-    "transform_function_keyword":"transform"
   },
   "model_id":"352",
   "model_name":"courier",
@@ -1081,8 +1153,7 @@ laborious/
 │       ├── prediction_process.py # Core prediction workflow
 │       └── format_and_export_prediction.py # Export workflow
 ├── worker/                  # Worker implementation
-│   ├── worker.py           # Main worker orchestrator
+│   └── worker.py           # Entrypoint; workers built via `sientia_do.temporal.worker.prepare_worker`
-│   └── prepare_worker.py   # Worker factory with autoscaling config
 ├── utils/                   # Utility functions
 │   ├── connectors_config.py # Environment-driven config builders
 │   ├── models/             # Data models
@@ -1091,7 +1162,6 @@ laborious/
 │   │   ├── conditional_filters.py # Conditional data filters
 │   │   └── mlflow_filters.py # MLFlow response filters
 │   └── repository/         # Data access layer
-│       ├── model_repository.py # MLFlow model operations
 │       ├── opc_repository.py # OPC server operations
 │       └── minio_manager.py # MinIO object storage operations
 ├── metrics.py               # Prometheus metrics definitions
@@ -1118,7 +1188,7 @@ laborious/
 2. **MLFlow Connection Issues**
    - Verify MLFlow server is running and accessible
    - Check authentication credentials and permissions
-   - Ensure model names and versions exist
+   - Ensure model names exist and the expected alias (for example `production`) is registered
 
 3. **Database Connection Issues**
    - Verify PostgreSQL service is running
@@ -1126,8 +1196,10 @@ laborious/
    - Ensure proper connection pool configuration
 
 4. **OPC Connection Failures**
-   - Verify OPC server is accessible
+   - See [docs/opc-communication.md](docs/opc-communication.md)
+   - Verify OPC server is accessible and `OPC_RECONNECTION_INTERVAL` is appropriate
    - Check certificate and key file paths
+   - Correlate `opc_write_attempts_total` with `opc_session_*` metrics; count session errors via `prediction_confidence = 14`
    - Review OPC server logs for connection issues
 
 5. **PI Web API Connection Failures**

docs/E2E_TEST_REPORT.md

@@ -0,0 +1,149 @@
+# E2E test run report
+
+**Date:** 2026-05-08  
+**Command:** `source venv/bin/activate && rtk pytest e2e/ -v --tb=short`  
+**Environment:** Linux, Python 3.11.15, pytest 9.0.3  
+
+## Summary
+
+| Metric | Count |
+|--------|------:|
+| Collected | 43 |
+| **Passed** | **37** |
+| **Failed** | **6** |
+
+Full pytest output (compressed by `rtk`) was written to:
+
+`~/.local/share/rtk/tee/1778268193_pytest.log`
+
+---
+
+## Failed tests (6)
+
+1. `e2e/test_drift.py::test_drift_happy_path_persists_all_columns_with_reference_data`
+2. `e2e/test_drift.py::test_drift_uses_30pct_fallback_when_reference_unavailable`
+3. `e2e/test_drift.py::test_drift_chunk_period_seconds_preserves_seconds_in_chunk_start_date`
+4. `e2e/test_predictions_batch_prediction_process.py::test_scenario_2_1_3_input_gate_triggers_repeat`
+5. `e2e/test_predictions_batch_prediction_process.py::test_scenario_2_2_3_transform_gate_triggers_repeat`
+6. `e2e/test_predictions_batch_prediction_process.py::test_scenario_2_3_3_predict_gate_triggers_repeat`
+
+**Follow-up:** Jensen–Shannon NULLs for the drift happy-path scenario are fully traced (histogram out-of-range + `density=True`, vs NannyML’s leftover bin) in [DRIFT_JS_NULL_INVESTIGATION.md](./DRIFT_JS_NULL_INVESTIGATION.md).
+
+---
+
+## Failure group A — Drift: `drift_metrics.value` NOT NULL (2 tests)
+
+### Error
+
+`psycopg2.errors.NotNullViolation`: null value in column `value` of relation `sientia_data.drift_metrics` violates not-null constraint.
+
+Example failing row (from logs): `feature=sensor_2`, `method=jensen_shannon`, `value=null`, with `kolmogorov_smirnov` / `wasserstein` populated for the same chunk.
+
+The bulk INSERT built by `Activities.export_data_to_postgres` includes parameters such as `'value__4': None` for `jensen_shannon` on a given chunk.
+
+### Root cause
+
+`calculate_drift` (real `DriftAnalysis` + `ModelMetrics.get_drift_metrics`) can emit **NaN / missing** values for some metric methods (here **Jensen–Shannon**) on some features/chunks. Pandas/SQLAlchemy turns that into SQL `NULL`, while the E2E schema (mirroring production) defines:
+
+```sql
+value numeric NOT NULL
+```
+
+in `e2e/db_schema.sql` for `sientia_data.drift_metrics`.
+
+### Recommended fixes (pick one consistent with product rules)
+
+1. **Application layer (preferred if NULLs are never valid in production):** Before export, sanitize the drift dataframe — e.g. drop rows where `value` is null/NaN, or replace with a defined sentinel (only if product agrees), or skip emitting that method row when the statistic is undefined.
+2. **Analytics layer:** Harden the Jensen–Shannon (and similar) paths so they always return a finite float for the supported inputs, or explicitly map “undefined” to an agreed numeric convention.
+3. **Schema (only if product allows missing metrics):** Align DDL with reality by making `value` nullable — **only** if production and downstream consumers already expect missing metrics; the E2E comment in `test_drift.py` suggests `feature`/`timestamp` nullable cases exist, but `value` is still listed in `NON_NULL_DRIFT_COLUMNS`.
+
+### Tests affected
+
+- `test_drift_happy_path_persists_all_columns_with_reference_data`
+- `test_drift_uses_30pct_fallback_when_reference_unavailable`
+
+---
+
+## Failure group B — Drift: chunk period seconds (1 test)
+
+### Error
+
+`AssertionError: expected at least one drift row to be persisted` (`e2e/test_drift.py:493`).
+
+The workflow run completed without failing the test via `WorkflowFailureError`, but **no rows** were found in `sientia_data.drift_metrics` for the model.
+
+### Likely cause
+
+In `Drift.run`, persistence runs only when `if drift_data:` is truthy (`laborious/workflows/drift.py`). An **empty** drift result skips `export_data_to_postgres`, so the table stays empty.
+
+Probable reasons:
+
+- With **`chunk_period='s'`** and only **three** target rows (30 s spacing), `calculate_drift` / `DriftAnalysis` may produce **no output rows** (insufficient data per chunk or internal filters).
+- Less likely here: time-window mismatch — timestamps are built from `datetime.now(UTC)` with `interval: 60` minutes from `drift_base.json`, so data should still fall in the window.
+
+### Recommended fixes
+
+1. **Test data:** Increase the number of second-spaced points (and/or span multiple chunk boundaries) so the analyzer reliably emits at least one chunk row.
+2. **Product code:** If sub-minute chunking is required to always produce metrics when any data exists, adjust `ModelMetrics` / `DriftAnalysis` integration for small-N second buckets.
+3. **Diagnostics:** Run the same scenario with `pytest -s` and confirm logs for “empty `drift_data`” vs export errors.
+
+Solução a ser aplicada:
+Dividir em dois testes:
+
+1. Teste com dados suficientes para produzir pelo menos uma linha de drift
+2. Teste com dados insuficientes para produzir pelo menos uma linha de drift, mas ja esperando os erros e validando que nao foi persistido nada
+
+### Test affected
+
+- `test_drift_chunk_period_seconds_preserves_seconds_in_chunk_start_date`
+
+---
+
+## Failure group C — Prediction REPEAT path: unique constraint on predictions (3 tests)
+
+### Error
+
+`psycopg2.errors.UniqueViolation`: duplicate key value violates unique constraint **`unique_model_id_timestamp`** on `sientia_data.predictions` (`model_id`, `timestamp`).
+
+### Root cause
+
+REPEAT is handled by `Activities.repeat_last_prediction` (registered from `sientia_do.temporal.activities.postgres_sync.Postgres`, via `Storage` inheritance in `laborious/activities/storage.py`). The E2E tests seed a prior row with a **fixed** timestamp:
+
+```python
+# e2e/test_predictions_batch_prediction_process.py — insert_sample_prediction
+VALUES ({model_id}, '2024-01-01 12:00:00+00:00', ...)
+```
+
+The helper `assert_repeat` expects **two** rows with the same `(model_id, prediction, prediction_confidence, prediction_status)` but compares **only** those columns — not `timestamp` (`e2e/helpers.py`). So the intended behavior is: **duplicate business payload**, not necessarily **duplicate primary unique key** `(model_id, timestamp)`.
+
+If `repeat_last_prediction` **INSERT**s a copy using the **same** `timestamp` as the last prediction, Postgres correctly rejects the second insert.
+
+### Recommended fixes
+
+1. **Implement REPEAT as UPSERT:** Use `ON CONFLICT (model_id, timestamp) DO UPDATE` (or the project’s existing `export_data_to_postgres` `on_conflict` pattern used in `FormatAndExportPrediction`) when writing the repeated prediction — if the product definition of REPEAT is “refresh same logical slot.”
+2. **Insert with the current batch timestamp:** Copy numeric/status fields from the last row but set `timestamp` to the **new** batch instant (e.g. the workflow’s `last_timestamp` / slice timestamp). This matches `assert_repeat`, which does not assert on `timestamp`.
+3. **Test-only change (weakest):** Relax assertions or change seed data — only if production behavior is “duplicate key is expected” (unlikely).
+
+Solução a ser aplicada:
+Alterar o teste para usar o timestamp da ultima execucao do batch, ao inves do timestamp do primeiro batch.
+2. **Insert with the current batch timestamp:** Copy numeric/status fields from the last row but set `timestamp` to the **new** batch instant (e.g. the workflow’s `last_timestamp` / slice timestamp). This matches `assert_repeat`, which does not assert on `timestamp`.
+
+### Tests affected
+
+- `test_scenario_2_1_3_input_gate_triggers_repeat`
+- `test_scenario_2_2_3_transform_gate_triggers_repeat`
+- `test_scenario_2_3_3_predict_gate_triggers_repeat`
+
+---
+
+## Passing areas (sanity check)
+
+All scenarios in `test_child_workflows_e2e.py`, `test_minimal_retrain.py`, `test_minio_offload.py`, `test_predictions_batch_format_export.py`, `test_predictions_batch_main_workflow.py` (except the three REPEAT cases above), and `test_simple_metrics.py` **passed** in this run.
+
+---
+
+## Suggested order of work
+
+1. Fix **drift `value` NULL** — unblocks two high-value drift E2Es and may clarify the chunk-seconds scenario if exports start succeeding consistently.
+2. Fix **`repeat_last_prediction` uniqueness** — unblocks three prediction-process E2Es; implementation likely lives in **`sientia_do`** Postgres activities, not in this repo.
+3. Revisit **`test_drift_chunk_period_seconds`** data volume / expectations after drift export is stable.

docs/opc-communication.md

@@ -0,0 +1,164 @@
+# OPC UA communication (Laborious)
+
+Laborious exports predictions to OPC UA servers through `OpcRepository` ([`laborious/utils/repository/opc_repository.py`](../laborious/utils/repository/opc_repository.py)) and the synchronous Temporal activity layer in [`laborious/activities/opc.py`](../laborious/activities/opc.py). The repository uses `asyncua.sync.Client` (asyncio on a background thread) so activities remain blocking without `async def`.
+
+OPC reconnect, write error classification (`opc_error_kind`), and activity confidence/comment behavior are converted from the **async** implementation on `main` at `fcc8920a8be4` (`asyncua.Client` + `asyncio` reconnect task → `threading` reconnect thread). Re-convert with `scripts/convert_opc_async_to_sync.py` when `main` OPC files change.
+
+Implementation plan for session/channel recovery on Tier-1 `Bad*` errors: [`.cursor/plans/opc_bad_reconnect_ac4c6045.plan.md`](../.cursor/plans/opc_bad_reconnect_ac4c6045.plan.md).
+
+## Architecture
+
+```text
+Worker (long-lived)
+  └── OpcRepository per OPC server id (from OPC_CONFIG / env)
+        ├── connect / disconnect / validate_connection (read-only)
+        ├── _connect_locked / _reconnect_locked (under _connection_lock)
+        ├── write_data (single attempt per call)
+        └── background reconnect on Tier-1 Bad*, closed protocol, or stale session
+
+Temporal activity write_opc_data
+  └── OPC.manage_output_tags → write_data per tag (sequential per activity)
+```
+
+One worker process holds one `OpcRepository` instance per configured server. Multiple Temporal activities can call `write_data` concurrently on the same repository.
+
+## Connection lifecycle
+
+| Phase | Behavior |
+|-------|----------|
+| Startup | `init_opc()` creates repositories and calls `connect()` → `_connect_locked()` |
+| Steady state | `validate_connection()` is read-only (`protocol.state` only); `_session_ready` is checked in `write_data` |
+| Tier-1 Bad* / protocol closed / session not ready | `_start_reconnect(reason)` → `_run_reconnect` (thread) → `_reconnect_locked()` (respects `reconnection_interval`) |
+| Write | `write_data()` checks in-flight reconnect thread, `_session_ready`, validates, then one `get_node` + `write_value` |
+| Shutdown | `disconnect()` sets `_allow_reconnect = False`, then tears down session |
+
+### Session and channel timeouts
+
+Requested session and secure-channel lifetime: **10 minutes** (`OPC_UA_SESSION_AND_CHANNEL_TIMEOUT_MS` in `opc_repository.py`). The server may revise these values; negotiated values are logged after connect and exposed as `opc_session_revised_timeout_milliseconds`.
+
+### Reconnection interval
+
+`OPC_RECONNECTION_INTERVAL` is in **seconds** (default `120`). It gates **background** reconnect after Tier-1 `Bad*` (`last_reconnection_time` is updated only in `_reconnect_locked()`). It limits load on the OPC server when many workflows fail at once.
+
+## Concurrency: connection lock and session readiness
+
+To allow **multiple concurrent writes** when the session is healthy, but **block all writes** while the connection is being torn down or re-established:
+
+| Primitive | Role |
+|-----------|------|
+| `_connection_lock` (`threading.Lock`) | Held for the entire `disconnect` → `connect` path. Only one connection-maintenance task at a time. |
+| `_session_ready` (`threading.Event`) | Set when a session is ready for writes; cleared before reconnect starts and set again after a successful connect. |
+| `_allow_reconnect` | Cleared in `disconnect()` so shutdown does not spawn reconnect threads |
+
+**Connection methods (caller holds `_connection_lock` for `_*_locked` helpers):**
+
+| Method | Role |
+|--------|------|
+| `_create_client()` | Create asyncua `Client` + optional `set_security`; raises if `client` already exists |
+| `_open_session()` | `client.connect()` + metrics; raises if session already open or client missing |
+| `_connect_locked()` | `_create_client()` (when needed) + `_open_session()`; raises if already connected |
+| `_disconnect_locked()` | Teardown session and clear `client` |
+| `_reconnect_locked()` | `_disconnect_locked()` + `_connect_locked()`; sets `last_reconnection_time` |
+
+Public `connect()` / `disconnect()` acquire the lock and call `_connect_locked()` / `_disconnect_locked()`.
+
+**Write path (`write_data`):**
+
+1. If a reconnect **thread** is alive → `reconnect_in_progress`.
+2. If `_session_ready` is cleared → schedule `SessionNotReady` reconnect; return `reconnect_in_progress` or `connection_lost`.
+3. If `validate_connection()` fails (protocol closed) → schedule `ProtocolClosed` reconnect; return `connection_lost`.
+4. Single `get_node` + `write_value` (no retry in the same call).
+
+**Reconnect path (`_run_reconnect`):**
+
+1. `_start_reconnect` clears `_session_ready` and starts a daemon thread when the interval allows and `_allow_reconnect` is true.
+2. `with _connection_lock:` → `_reconnect_locked()`.
+3. `_session_ready` is set on successful `_open_session()`.
+
+A second `_connect_locked()` while a session is already open raises `OpcSessionAlreadyConnectedError` (disconnect first).
+
+**asyncua note:** Concurrent `write_value` on the same session is only safe if the stack tolerates it. If production shows issues, serialize writes while keeping the connection lock semantics above.
+
+## Tier-1 `Bad*` errors and reconnect
+
+When the server invalidates the session (e.g. `BadSessionIdInvalid`) but the client still sees transport as open, `write_data` fails once, records the OPC status in metrics, and **schedules** reconnect if:
+
+- The exception is a `UaStatusCodeError` whose name is in `RECONNECTABLE_OPC_BAD_NAMES` (see plan), and
+- `reconnection_interval` has elapsed since `last_reconnection_time`, and
+- No reconnect task is already running.
+
+There is **no write retry**: the failed export is not sent again in the same activity.
+
+## Prediction confidence and PostgreSQL comments
+
+| `prediction_confidence` | Meaning |
+|-------------------------|---------|
+| (unchanged) | Successful OPC export |
+| **12** | Generic OPC write failure (`OPC_WRITTING_ERROR_CONFIDENCE`) |
+| **14** | Tier-1 session/channel `Bad*` on export (`OPC_SESSION_BAD_CONFIDENCE`) |
+| **14** | Write while reconnect in progress (`OPC_SESSION_BAD_CONFIDENCE`, comment `OPC UA reconnect in progress`) |
+| **13** | PI Web API write failure (separate path) |
+
+Session/channel errors use a stable comment for counting:
+
+```text
+OPC UA session/channel error: BadSessionIdInvalid
+```
+
+Reconnect-in-progress exports use:
+
+```text
+OPC UA reconnect in progress
+```
+
+Example SQL:
+
+```sql
+SELECT count(*) FROM predictions WHERE prediction_confidence = 14;
+SELECT count(*) FROM predictions WHERE comments LIKE 'OPC UA session/channel error:%';
+```
+
+## Prometheus metrics (`opc_*`)
+
+Defined in [`laborious/metrics.py`](../laborious/metrics.py). Do not rename in production without a dashboard migration.
+
+| Metric | Purpose |
+|--------|---------|
+| `opc_connections_initiated_total` | Connection attempts |
+| `opc_connections_failed_total` | Failed connects |
+| `opc_connection_status` | Gauge 1=connected, 0=disconnected |
+| `opc_session_created_total` | Session established after connect |
+| `opc_session_closed_total` | Disconnect initiated |
+| `opc_session_revised_timeout_milliseconds` | Negotiated session timeout (ms) |
+| `opc_write_attempts_total` | Per write; label `result` = `OK` or exception name |
+| `opc_write_inter_arrival_over_session_timeout_total` | Successful writes spaced longer than revised session timeout |
+
+Legacy activity metrics: `laborious_prediction_opc_writing_count`, `laborious_prediction_opc_writing_response_time_monitor`.
+
+## Environment variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `OPC_CONFIG` | — | JSON map of server configs (overrides single-server env) |
+| `OPC_ID` | `1` | Server id |
+| `OPC_URL` | `opc.tcp://localhost:4840` | Endpoint |
+| `OPC_SERVER_NAME` | `default_server` | Label for metrics/logs |
+| `OPC_SERVER_URI` | same as URL | Application URI / cert SAN |
+| `OPC_CERT_PATH` | — | Client certificate (secure mode) |
+| `OPC_PRIVATE_KEY_PATH` | — | Client private key |
+| `OPC_SERVER_CERT_PATH` | — | Server certificate |
+| `OPC_RECONNECTION_INTERVAL` | `120` | Minimum seconds between reconnects |
+
+## Operations checklist
+
+- Correlate `BadSessionIdInvalid` in `opc_write_attempts_total` with `opc_session_closed_total` / `opc_session_created_total` (reconnect may finish after the row is stored with confidence 14).
+- Use confidence **14** and comment prefix for session invalidation rates; use **12** for other OPC failures.
+- Respect `OPC_RECONNECTION_INTERVAL` under parallel load; bursts of confidence 14 are expected until the next successful cycle.
+
+## Related tests
+
+- Unit: [`tests/laborious/utils/repository/test_opc_repository.py`](../tests/laborious/utils/repository/test_opc_repository.py)
+- Unit: [`tests/laborious/activities/test_opc.py`](../tests/laborious/activities/test_opc.py)
+- E2E (mock OPC): [`e2e/test_predictions_batch_format_export.py`](../e2e/test_predictions_batch_format_export.py)
+- E2E (in-process asyncua server + real `OpcRepository`): [`e2e/test_opc_real_server.py`](../e2e/test_opc_real_server.py) — scenarios 3.1.2, 3.2.2, 3.2.4, 3.2.5
+- Scenarios: [`e2e/scenarios.md`](../e2e/scenarios.md)

docs/sientia_model_drift_jensen_shannon_change.md

@@ -0,0 +1,55 @@
+# Specification: `sientia_model` — Jensen–Shannon drift (`DriftAnalysis`)
+
+This document describes what **`sientia_model.analytics.drift_analysis.DriftAnalysis`** should change so downstream consumers (e.g. Laborious `calculate_drift` → Postgres `drift_metrics.value NOT NULL`) no longer receive **NaN** for Jensen–Shannon on valid finite data.
+
+## Scope
+
+- **File:** `sientia_model/analytics/drift_analysis.py`
+- **Method:** `_jensen_shannon_distance(self, ref: np.ndarray, cur: np.ndarray, bins: int = 20) -> float`
+- **Callers:** `detect_univariate_drift` uses this for the `jensen_shannon` method; results are written to `value` in the drift dataframe.
+
+## Problem
+
+The current implementation:
+
+1. Builds bin edges from **`ref`** only: `np.histogram(ref, bins=bins, density=True)`.
+2. Builds the chunk histogram with **`density=True`** on the same edges: `np.histogram(cur, bins=edges, density=True)`.
+
+When **every** value in **`cur`** falls **outside** the closed support implied by those edges (typical case: production chunk drifted above the reference max or below the reference min), NumPy yields **all-zero counts** for `cur`. With **`density=True`**, normalization does **0/0**, producing **NaN** for the whole histogram, which propagates to **`float('nan')`** in `detect_univariate_drift` → **SQL NULL** where `value` is `NOT NULL`.
+
+This appears in Laborious when:
+
+- `model_config.target` excludes the main target column from univariate features, so another feature (e.g. `sensor_2`) is compared chunk-by-chunk against the full reference series for that feature.
+- Reference and current ranges do not overlap for some chunks (strong drift or different scaling).
+
+Other methods (`kolmogorov_smirnov`, `wasserstein`) do not use this histogram+density path, so they can stay finite while **Jensen–Shannon** alone becomes null.
+
+## Required behavior
+
+1. **Finite output** for finite `ref` and `cur` after removing non-finite values, whenever both sides have **at least one** usable sample.
+2. **Explicit handling of out-of-range chunk mass:** probability mass from `cur` that does not fall into any bin defined from `ref` must still be represented (so the chunk distribution sums to 1), analogous to NannyML’s continuous JS approach (tail / “leftover” mass).
+3. **Missing values:** drop `NaN` from `ref` and `cur` before computing. If either side is **empty** after that, return **`float('nan')`** (callers may filter or map; schema may still forbid null — product decision outside this spec).
+
+## Recommended algorithm (replace current body)
+
+1. `ref = np.asarray(ref, float); cur = np.asarray(cur, float)`.
+2. `ref = ref[~np.isnan(ref)]; cur = cur[~np.isnan(cur)]`.
+3. If `ref.size == 0` or `cur.size == 0`: return `float('nan')`.
+4. `hist_ref, edges = np.histogram(ref, bins=bins)` — **counts**, not `density=True`.
+5. `p = hist_ref.astype(float) / ref.size` (reference bin probabilities).
+6. `hist_cur, _ = np.histogram(cur, bins=edges)`; `q = hist_cur.astype(float) / cur.size`.
+7. `leftover = 1.0 - float(np.sum(q))`. If `leftover > 1e-15` (tolerance for float noise), append **`leftover`** to `q` and **`0.0`** to `p` so both remain proper discrete distributions over the same extended support.
+8. Apply small smoothing (existing module constant `EPSILON` is fine): add `EPSILON` to `p` and `q`, renormalize each to sum 1.
+9. `m = 0.5 * (p + q)`; compute symmetric JS via KL terms as today, e.g. `inner = 0.5 * (sum(p*log(p/m)) + sum(q*log(q/m)))`.
+10. Return `sqrt(max(inner, 0.0))` to guard against tiny negative `inner` from floating-point error.
+
+## Non-goals / notes
+
+- **Numerical parity** with the old `density=True` implementation is not required; parity with **NannyML** or **scipy** JS is desirable but optional. The priority is **finite, interpretable** drift when the chunk is outside the reference histogram range.
+- **Multivariate** drift in the same file is unchanged by this spec.
+- **Tests** in `sientia_model` should cover: (a) chunk entirely above reference max, (b) entirely below reference min, (c) overlapping range, (d) `ref` or `cur` all-NaN after cleaning.
+
+## Reference (external)
+
+- NannyML continuous JS uses count-based bin probabilities and a **leftover** mass bin; see `ContinuousJensenShannonDistance` in `nannyml/drift/univariate/methods.py` (`_calculate`, `leftover = 1 - np.sum(...)`).
+- Historical NaN-in-reference issue: [NannyML#339](https://github.com/NannyML/nannyml/issues/339) / [#340](https://github.com/NannyML/nannyml/pull/340) (orthogonal to out-of-range mass, but relevant for input cleaning).

e2e/conftest.py

@@ -1,37 +1,53 @@
-"""
+"""Pytest configuration and fixtures for E2E tests."""
-Pytest configuration and fixtures for E2E tests.
-"""
 
-from unittest.mock import AsyncMock, MagicMock, patch
+import asyncio
-from io import BytesIO
+import time
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+from unittest.mock import MagicMock
 
 import pandas as pd
 import pytest
 import pytest_asyncio
-from sqlalchemy import create_engine, text
+from sqlalchemy import create_engine
+from testcontainers.core.container import DockerContainer
 from testcontainers.minio import MinioContainer
 from testcontainers.postgres import PostgresContainer
 from temporalio.testing import WorkflowEnvironment
 from temporalio.worker import Worker
 
+from e2e.opc_test_server import OpcE2ETestServer
 from laborious.activities.activities import Activities
+from laborious.workflows.drift import Drift
+from laborious.workflows.minimal_retrain import MinimalRetrain
 from laborious.workflows.predictions_batch import PredictionsBatch
+from laborious.workflows.simple_metrics import SimpleMetrics
+from laborious.workflows.sub_workflows.format_and_export_prediction import (
+    FormatAndExportPrediction,
+)
 from laborious.workflows.sub_workflows.prediction_process import PredictionProcess
-from laborious.workflows.sub_workflows.format_and_export_prediction import FormatAndExportPrediction
 from sientia_do.notifications.handlers import CoreNotificationHandler
 from sientia_do.observability.logger import Logger
 from sientia_do.observability.metrics_controller import MetricsController
 
-# Test constants
+# Single source of truth for the test database schema. Mirrors the production
-TEST_MONGODB_CONNECTION_STRING = 'mongodb://localhost:27017'
+# DDL for ``sientia_data`` so any production change can be pasted directly into
-TEST_DATABASE_NAME = 'test_db'
+# this file (see ``e2e/db_schema.sql``) without touching Python.
+DB_SCHEMA_SQL_PATH = Path(__file__).parent / 'db_schema.sql'
+
+
+@pytest_asyncio.fixture(scope='session')
+def postgres_container():
+    """PostgreSQL testcontainer used by all E2E tests."""
+    postgres = PostgresContainer('postgres:15')
+    postgres.start()
+    yield postgres
+    postgres.stop()
 
 
 @pytest_asyncio.fixture(scope='session')
 def minio_container():
-    """
+    """MinIO testcontainer used by E2E offload and payload retrieval paths."""
-    MinIO S3-compatible storage for E2E tests that exercise real offload uploads.
-    """
     minio = MinioContainer()
     minio.start()
     yield minio
@@ -39,468 +55,202 @@ def minio_container():
 
 
 @pytest_asyncio.fixture(scope='session')
-def postgres_container():
+def mongo_container():
-    """
+    """MongoDB testcontainer used by real CoreNotificationHandler."""
-    Create a PostgreSQL container using testcontainers.
+    mongo = DockerContainer('mongo:7').with_exposed_ports(27017)
-    
+    mongo.start()
-    This fixture creates a real PostgreSQL database in a Docker container
+    yield mongo
-    that will be used for all tests in the session.
+    mongo.stop()
-    """
-    postgres = PostgresContainer('postgres:15')
-    postgres.start()
-    yield postgres
-    postgres.stop()
 
 
 @pytest_asyncio.fixture
 def postgres_engine(postgres_container):
-    """
+    """SQLAlchemy engine bound to the PostgreSQL testcontainer."""
-    Create SQLAlchemy engine for PostgreSQL test database.
-    
-    This fixture creates a connection to the PostgreSQL container
-    created by the postgres_container fixture.
-    """
     engine = create_engine(postgres_container.get_connection_url())
-    
     yield engine
-    
     engine.dispose()
 
 
 def _create_schema_and_tables(engine):
     """
-    Helper function to create schema and tables in the given engine.
+    Create all schemas/tables required by workflow and activity paths.
 
-    Creates predictions_schema with:
+    Loads the DDL from ``e2e/db_schema.sql`` (single source of truth that
-    - laborious_data: Input data table for queries
+    mirrors the production schema). The SQL file is executed via the raw
-    - predictions: Output predictions table
+    DBAPI cursor so multi-statement DDL is supported.
-    - transformed_data: Output transformed data table
     """
-    # Use begin() to ensure transaction is properly committed
+    sql_text = DB_SCHEMA_SQL_PATH.read_text(encoding='utf-8')
     with engine.begin() as conn:
-        # Create predictions_schema
+        conn.exec_driver_sql(sql_text)
-        conn.execute(text("CREATE SCHEMA IF NOT EXISTS predictions_schema"))
-        
-        # Create laborious_data table (input data from sensors)
-        create_laborious_data_sql = """
-            CREATE TABLE IF NOT EXISTS predictions_schema.laborious_data (
-                id SERIAL NOT NULL,
-                model_id int4 NOT NULL,
-                variable text NOT NULL,
-                value numeric NULL,
-                "timestamp" timestamptz NOT NULL,
-                created_at timestamptz NOT NULL,
-                PRIMARY KEY (id)
-            );
-        """
-        conn.execute(text(create_laborious_data_sql))
-        
-        # Create predictions table
-        create_predictions_sql = """
-            CREATE TABLE if not exists predictions_schema.predictions (
-                id SERIAL NOT NULL ,
-                model_id int4 NOT NULL,
-                prediction numeric NULL,
-                prediction_confidence numeric NOT NULL,
-                response_time numeric NOT NULL,
-                prediction_status text NOT NULL,
-                "timestamp" timestamptz NOT NULL,
-                created_at timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
-                "comments" text NULL,
-                PRIMARY KEY (id, created_at)
-            );
-        """
-        conn.execute(text(create_predictions_sql))
-        
-        # Create transformed_data table
-        create_transformed_sql = """
-            CREATE TABLE IF NOT EXISTS predictions_schema.transformed_data (
-                id SERIAL NOT NULL,
-                model_id int4 NOT NULL,
-                variable text NOT NULL,
-                value numeric NULL,
-                "timestamp" timestamptz NOT NULL,
-                created_at timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
-                PRIMARY KEY (id)
-            );
-        """
-        conn.execute(text(create_transformed_sql))
 
 
 @pytest_asyncio.fixture(autouse=True)
 def setup_postgres_schema_and_tables(postgres_engine):
-    """
+    """Ensure required schema and tables exist before each E2E test."""
-    Automatically create necessary schema and tables before each test.
-    
-    This fixture runs automatically (autouse=True) and ensures
-    that the predictions_schema and tables exist with the correct structure.
-    """
     _create_schema_and_tables(postgres_engine)
     yield
 
 
 @pytest_asyncio.fixture
 def mock_logger():
-    """Mock logger for testing."""
+    """Logger double with readable console output for E2E runs."""
-    def message(message):
+    logger = MagicMock(spec=Logger)
-        print(f"[LOG] {message}")
+    logger.info = MagicMock(side_effect=lambda msg: print(f'[LOG] {msg}'))
-    def custom_message(message, _metadata={}):
+    logger.debug = MagicMock(side_effect=lambda msg: print(f'[LOG] {msg}'))
-        print(f"[LOG] {message}")
+    logger.error = MagicMock(side_effect=lambda msg: print(f'[LOG] {msg}'))
-    logger = MagicMock()
+    logger.warning = MagicMock(side_effect=lambda msg: print(f'[LOG] {msg}'))
-    logger.info = MagicMock(
+    logger.custom_info = MagicMock(side_effect=lambda msg, _meta=None: print(f'[LOG] {msg}'))
-        side_effect=message
+    logger.custom_debug = MagicMock(side_effect=lambda msg, _meta=None: print(f'[LOG] {msg}'))
-    )
+    logger.custom_error = MagicMock(side_effect=lambda msg, _meta=None: print(f'[LOG] {msg}'))
-    logger.debug = MagicMock(
+    logger.custom_warning = MagicMock(side_effect=lambda msg, _meta=None: print(f'[LOG] {msg}'))
-        side_effect=message
-    )
-    logger.error = MagicMock(
-        side_effect=message
-    )
-    logger.warning = MagicMock(
-        side_effect=message
-    )
-    logger.custom_info = MagicMock(
-        side_effect=custom_message
-    )
-    logger.custom_debug = MagicMock(
-        side_effect=custom_message
-    )
-    logger.custom_error = MagicMock(
-        side_effect=custom_message
-    )
-    logger.custom_warning = MagicMock(
-        side_effect=custom_message
-    )
     return logger
 
 
-@pytest_asyncio.fixture
-def mock_mongo_client():
-    """
-    Mock MongoDB client to avoid real connections.
-    
-    This fixture mocks the pymongo.MongoClient used by CoreNotificationHandler,
-    allowing us to use a real NotificationHandler instance without connecting to MongoDB.
-    """
-    mock_client = MagicMock()
-    mock_db = MagicMock()
-    mock_collection = MagicMock()
-    
-    # Configure the mock chain: client[database] -> db[collection] -> collection
-    mock_client.__getitem__.return_value = mock_db
-    mock_db.__getitem__.return_value = mock_collection
-    
-    # Mock server_info() to avoid connection attempts
-    mock_client.server_info = MagicMock()
-    
-    # Mock insert_one for notifications
-    mock_collection.insert_one = MagicMock()
-    
-    return mock_client
-
-
-@pytest.fixture
-def notification_inserts(mock_mongo_client):
-    """
-    Mongo insert_one mock used by CoreNotificationHandler for notification persistence.
-
-    Yields:
-        MagicMock for insert_one, reset before each test.
-    """
-    mock_db = mock_mongo_client.__getitem__.return_value
-    mock_collection = mock_db.__getitem__.return_value
-    mock_collection.insert_one.reset_mock()
-    yield mock_collection.insert_one
-
-
-@pytest_asyncio.fixture
-def notification_handler(mock_logger, mock_mongo_client):
-    """
-    Create a real NotificationHandler instance with mocked MongoDB client.
-    
-    This fixture creates a real CoreNotificationHandler instance but mocks
-    the underlying MongoDB connection to avoid real database connections.
-    """
-    # Patch MongoClient where it's imported in the handlers module
-    with patch('sientia_do.notifications.handlers.MongoClient', return_value=mock_mongo_client):
-        handler = CoreNotificationHandler(
-            connection_string=TEST_MONGODB_CONNECTION_STRING,
-            database=TEST_DATABASE_NAME,
-            logger=mock_logger,
-            project_name='laborious',
-        )
-        yield handler
-        handler.shutdown()
-
-
 @pytest_asyncio.fixture
 def metrics_controller(mock_logger):
-    """Create a real MetricsController instance."""
+    """Real metrics controller for E2E observability paths."""
     return MetricsController(logger=mock_logger)
 
 
 @pytest_asyncio.fixture
-def mock_minio_repository():
+def notification_handler(mock_logger, mongo_container):
-    """Mock MinIO repository for object storage operations."""
+    """Real notification handler using MongoDB testcontainer."""
-    mock_repo = MagicMock()
+    mongo_port = mongo_container.get_exposed_port(27017)
+    handler = CoreNotificationHandler(
+        connection_string=f'mongodb://localhost:{mongo_port}',
+        database='test_db',
+        logger=mock_logger,
+        project_name='laborious',
+    )
+    try:
+        yield handler
+    finally:
+        handler.shutdown()
 
-    # Provide at least valid parquet bytes so that MinioDataFramePayload.retrieve()
-    # can decode the payload if offloading is exercised in an integration scenario.
-    parquet_df = pd.DataFrame({'a': [1]})
-    parquet_buffer = BytesIO()
-    parquet_df.to_parquet(parquet_buffer, engine='pyarrow', index=True)
-    parquet_bytes = parquet_buffer.getvalue()
 
-    # sientia_do MinioRepository API
+@pytest.fixture
-    mock_repo.bucket = 'test-bucket'
+def notification_inserts(notification_handler):
-    mock_repo.upload_file = AsyncMock(
+    """Spy on real Mongo insert calls issued by notification handler."""
-        side_effect=lambda file_bytes, relative_key, content_type='application/octet-stream', bucket=None, metadata=None: {
+    collection = notification_handler.mongo_collection
-            'minio_object_name': f'sientia/streamlit-connectors/{relative_key}',
+    original_insert_one = collection.insert_one
-            'original_filename': relative_key.rsplit('/', 1)[-1],
+    spy = MagicMock(wraps=original_insert_one)
-            'uploaded_at': '2024-01-01T00:00:00Z',
+    collection.insert_one = spy
-            'sha256_hash': 'deadbeef',
+    try:
+        yield spy
+    finally:
+        collection.insert_one = original_insert_one
+
+
+class _FakeModelWrapper:
+    """External MLflow wrapper double used by repository stub."""
+
+    def __init__(self):
+        self.transform = MagicMock(side_effect=self._default_transform)
+        self.predict = MagicMock(side_effect=self._default_predict)
+
+    @staticmethod
+    def _default_transform(data: pd.DataFrame):
+        result = pd.DataFrame(
+            {
+                'feature_1': [0.234] * len(data),
+                'feature_2': [0.783] * len(data),
             }
         )
-    mock_repo.download_file = AsyncMock(return_value=parquet_bytes)
+        result.index = data.index
-    mock_repo.list_objects = AsyncMock(return_value=[])
+        return result, {}
-    mock_repo.delete_file = AsyncMock()
-    mock_repo.close = MagicMock()
 
-    return mock_repo
+    @staticmethod
+    def _default_predict(_params: dict, data: pd.DataFrame):
+        pred = pd.DataFrame([0.5] * len(data), columns=['placeholder'])
+        pred.index = data.index
+        return pred, {}
 
 
 @pytest_asyncio.fixture
-def mock_pi_web_api_repository():
+def mlflow_repository_stub():
-    """Mock PI Web API repository for PI Web API operations."""
+    """External MLflow repository stub."""
-    mock_repo = MagicMock()
+    repo = MagicMock()
+    wrapper = _FakeModelWrapper()
+    repo.stub_wrapper = wrapper
+    repo.get_cached_model = MagicMock(return_value=wrapper)
+    repo._client = MagicMock()
+    return repo
 
-    async def _write_value(web_ids, value, metadata=None, **kwargs):
-        """
-        Mirror successful PI writes: one response item per requested web_id.
 
-        write_pi_web_api_data passes the list into process_pi_web_api_response (not a
+class _FakePIWebAPIClient:
-        wrapped {'Items': ...} envelope).
+    """External PI Web API client stub with deterministic responses."""
-        """
+
+    def __init__(self):
+        self._responses = None
+        self.write_value = MagicMock(side_effect=self._write_value)
+        self.close = MagicMock()
+
+    def set_side_effect(self, side_effect):
+        self._responses = side_effect
+
+    def _write_value(self, web_ids, value, metadata=None, **kwargs):
+        if isinstance(self._responses, Exception):
+            raise self._responses
+        if isinstance(self._responses, list):
+            item = self._responses.pop(0)
+            if isinstance(item, Exception):
+                raise item
+            return item
+        if callable(self._responses):
+            return self._responses(web_ids=web_ids, value=value, metadata=metadata, **kwargs)
         return [{'WebId': wid, 'Errors': []} for wid in web_ids]
 
-    mock_repo.write_value = AsyncMock(side_effect=_write_value)
-    mock_repo.close = MagicMock()
-    return mock_repo
 
 @pytest_asyncio.fixture
-def mock_opc_repository():
+def pi_web_api_client_stub():
-    """Mock OPC repository for OPC operations."""
+    """PI Web API stub fixture."""
-    mock_repo = MagicMock()
+    return _FakePIWebAPIClient()
-    mock_repo.write_data = AsyncMock(
-        return_value=(True, {'response_time': 0.1})
-    )
-    mock_repo.disconnect = AsyncMock()
-    return mock_repo
-
-@pytest_asyncio.fixture
-def patch_create_engine(postgres_engine):
-    """Patch create_engine to return test postgres_engine."""
-    with patch('sientia_do.temporal.activities.postgres.create_engine', return_value=postgres_engine):
-        yield
 
 
 @pytest_asyncio.fixture
-def patch_minio_repository(mock_minio_repository):
+def opc_repository_stub():
-    """Patch MinioRepository to return mock."""
+    """OPC external dependency stub."""
-    # Patch where Activities resolves the symbol (import binds the original class).
+    repo = MagicMock()
-    with patch('laborious.activities.activities.MinioRepository', return_value=mock_minio_repository):
+    repo.write_data = MagicMock(return_value=(True, {'response_time': 0.1}))
-        yield
+    repo.disconnect = MagicMock()
-
+    return repo
-@pytest_asyncio.fixture
-def patch_pi_web_api_repository(mock_pi_web_api_repository):
-    """Patch MLflowRepository to return mock."""
-    with patch('laborious.activities.api.PIWebAPIClient', return_value=mock_pi_web_api_repository):
-        yield
-
-@pytest_asyncio.fixture
-def mock_mlflow_models():
-    """Create mock models for MLflow load_model methods."""
-    # Mock transform model - returns DataFrame with same index as input
-    mock_transform_model = MagicMock()
-    def mock_transform_predict(data):
-        num_rows = max(len(data), 1) if hasattr(data, '__len__') else 1
-        print(data.to_csv())
-        print(data.index)
-        result = pd.DataFrame({
-            'feature_1': [0.234] * num_rows,
-            'feature_2': [0.783] * num_rows,
-        })
-        result.index = data.index
-        return result
-    mock_transform_model.predict = MagicMock(side_effect=mock_transform_predict)
-    
-    # Mock predict model - returns array/list of predictions
-    mock_predict_model = MagicMock()
-    def mock_predict_predict(data):
-        num_rows = max(len(data), 1) if hasattr(data, '__len__') else 1
-        return [0.5] * num_rows
-    mock_predict_model.predict = MagicMock(side_effect=mock_predict_predict)
-    
-    # Mock PyFuncModel for compressed models
-    mock_pyfunc_model = MagicMock()
-    mock_pyfunc_model._model_impl = MagicMock()
-    mock_pyfunc_model._model_impl.python_model = mock_transform_model
-    
-    return {
-        'transform_model': mock_transform_model,
-        'predict_model': mock_predict_model,
-        'pyfunc_model': mock_pyfunc_model,
-    }
 
 
 @pytest_asyncio.fixture
-def patch_mlflow(mock_mlflow_models):
+def plugin_store_stub():
-    """Patch mlflow module in repository with load_model mocks."""
+    """Plugin store external dependency stub."""
-    mock_mlflow = MagicMock()
+    return MagicMock()
-    
-    # Mock sklearn.load_model
-    def mock_sklearn_load_model(model_uri):
-        if 'data_model' in model_uri or 'transform' in model_uri.lower():
-            return mock_mlflow_models['transform_model']
-        return mock_mlflow_models['predict_model']
-    mock_mlflow.sklearn = MagicMock()
-    mock_mlflow.sklearn.load_model = MagicMock(side_effect=mock_sklearn_load_model)
-    
-    # Mock pyfunc.load_model
-    def mock_pyfunc_load_model(model_uri):
-        if 'artifacts' in model_uri or 'tmp' in model_uri:
-            return mock_mlflow_models['pyfunc_model']
-        if 'data_model' in model_uri or 'transform' in model_uri.lower():
-            return mock_mlflow_models['transform_model']
-        return mock_mlflow_models['predict_model']
-    mock_mlflow.pyfunc = MagicMock()
-    mock_mlflow.pyfunc.load_model = MagicMock(side_effect=mock_pyfunc_load_model)
-    
-    # Mock pytorch.load_model
-    mock_mlflow.pytorch = MagicMock()
-    mock_mlflow.pytorch.load_model = MagicMock(return_value=mock_mlflow_models['predict_model'])
-    
-    # Mock other mlflow methods that might be called
-    mock_mlflow.set_tracking_uri = MagicMock()
-    mock_mlflow.get_run = MagicMock(return_value=MagicMock(info=MagicMock(artifact_uri='mlflow-artifacts:/test_run_id')))
-    mock_mlflow.tracking = MagicMock()
-    mock_mlflow.tracking.MlflowClient = MagicMock(return_value=MagicMock(
-        search_registered_models=MagicMock(return_value=[MagicMock(name='test_model')]),
-        search_model_versions=MagicMock(return_value=[MagicMock(
-            current_stage='Production',
-            version='1',
-            source='runs:/artifacts/test_run_id'
-        )])
-    ))
-    
-    with patch('laborious.utils.repository.model_repository.mlflow', new=mock_mlflow):
-        yield mock_mlflow
 
 
 @pytest_asyncio.fixture(scope='function')
 async def test_activities(
-    postgres_engine,
-    postgres_container,
-    mock_logger,
-    notification_handler,
-    metrics_controller,
-    mock_minio_repository,
-    patch_create_engine,
-    patch_minio_repository,
-    patch_mlflow,
-    patch_pi_web_api_repository,
-    mock_opc_repository
-):
-    """
-    Create Activities instance with test dependencies.
-
-    This fixture creates a real Activities instance with:
-    - PostgreSQL database (via testcontainers)
-    - Mocked MinIO client
-    - Real NotificationHandler and MetricsController (with mocked underlying services)
-    """
-    activities = Activities(
-        postgres_config={
-            'host': 'localhost',
-            'port': postgres_container.get_exposed_port(5432),
-            'user': 'test',
-            'password': 'test',
-            'dbname': 'test',
-            'min_connections': 1,
-            'max_connections': 5,
-        },
-        mlflow_config={
-            'host': 'http://localhost',
-            'port': '5000',
-            'username': 'test',
-            'password': 'test',
-        },
-        minio_config={
-            # Host:port only; Minio() prepends http(s):// from the secure flag.
-            'endpoint_url': 'localhost:9000',
-            'access_key': 'test',
-            'secret_key': 'test',
-            'default_bucket': 'test-bucket',
-            'retention_hours': 24,
-            'secure': False,
-        },
-        opc_config={},
-        pi_web_api_config={
-            'base_url': 'http://localhost:8080',
-            'auth_type': 'bearer',
-            'auth_token': 'test_token',
-        },
-        logger=mock_logger,
-        notification_handler=notification_handler,
-    )
-
-    activities.opc_repository = {
-        '1': mock_opc_repository,
-    }
-
-    try:
-        yield activities
-    finally:
-        # Cleanup - ALWAYS runs, even if test fails
-        await activities.shutdown()
-
-
-@pytest_asyncio.fixture(scope='function')
-async def test_activities_real_minio(
-    postgres_engine,
     postgres_container,
     minio_container,
     mock_logger,
     notification_handler,
     metrics_controller,
-    patch_create_engine,
+    mlflow_repository_stub,
-    patch_mlflow,
+    plugin_store_stub,
-    patch_pi_web_api_repository,
+    pi_web_api_client_stub,
-    mock_opc_repository,
+    opc_repository_stub,
 ):
-    """
+    """Activities with real infra and external-system stubs only."""
-    Activities with a real MinIO testcontainer (no MinioRepository patch) for offload tests.
-    """
     minio_client = minio_container.get_client()
     if not minio_client.bucket_exists('test-bucket'):
         minio_client.make_bucket('test-bucket')
     minio_port = minio_container.get_exposed_port(9000)
+
     activities = Activities(
         postgres_config={
             'host': 'localhost',
-            'port': postgres_container.get_exposed_port(5432),
+            'port': int(postgres_container.get_exposed_port(5432)),
-            'user': 'test',
+            'user': postgres_container.username,
-            'password': 'test',
+            'password': postgres_container.password,
-            'dbname': 'test',
+            'dbname': postgres_container.dbname,
             'min_connections': 1,
             'max_connections': 5,
         },
-        mlflow_config={
+        plugin_store=plugin_store_stub,
-            'host': 'http://localhost',
-            'port': '5000',
-            'username': 'test',
-            'password': 'test',
-        },
         minio_config={
             'endpoint_url': f'localhost:{minio_port}',
             'access_key': 'minioadmin',
@@ -517,17 +267,76 @@ async def test_activities_real_minio(
         },
         logger=mock_logger,
         notification_handler=notification_handler,
+        metrics_controller=metrics_controller,
+        mlflow_repository=mlflow_repository_stub,
     )
-    activities.opc_repository = {'1': mock_opc_repository}
+    activities.pi_web_api_client = pi_web_api_client_stub
+    activities.opc_repository = {'1': opc_repository_stub}
     try:
         yield activities
     finally:
-        await activities.shutdown()
+        activities.shutdown()
+
+
+@pytest_asyncio.fixture(scope='function')
+async def test_activities_real_minio(
+    postgres_container,
+    minio_container,
+    mock_logger,
+    notification_handler,
+    metrics_controller,
+    mlflow_repository_stub,
+    plugin_store_stub,
+    pi_web_api_client_stub,
+    opc_repository_stub,
+):
+    """Compatibility alias for offload tests."""
+    minio_client = minio_container.get_client()
+    if not minio_client.bucket_exists('test-bucket'):
+        minio_client.make_bucket('test-bucket')
+    minio_port = minio_container.get_exposed_port(9000)
+
+    activities = Activities(
+        postgres_config={
+            'host': 'localhost',
+            'port': int(postgres_container.get_exposed_port(5432)),
+            'user': postgres_container.username,
+            'password': postgres_container.password,
+            'dbname': postgres_container.dbname,
+            'min_connections': 1,
+            'max_connections': 5,
+        },
+        plugin_store=plugin_store_stub,
+        minio_config={
+            'endpoint_url': f'localhost:{minio_port}',
+            'access_key': 'minioadmin',
+            'secret_key': 'minioadmin',
+            'default_bucket': 'test-bucket',
+            'retention_hours': 24,
+            'secure': False,
+        },
+        opc_config={},
+        pi_web_api_config={
+            'base_url': 'http://localhost:8080',
+            'auth_type': 'bearer',
+            'auth_token': 'test_token',
+        },
+        logger=mock_logger,
+        notification_handler=notification_handler,
+        metrics_controller=metrics_controller,
+        mlflow_repository=mlflow_repository_stub,
+    )
+    activities.pi_web_api_client = pi_web_api_client_stub
+    activities.opc_repository = {'1': opc_repository_stub}
+    try:
+        yield activities
+    finally:
+        activities.shutdown()
 
 
 def _worker_activity_list(test_activities: Activities):
+    """List of registered activity callables used by Temporal worker in E2E."""
     return [
-        test_activities.load_custom_query,
         test_activities.load_query_with_minio_offload,
         test_activities.cleanup_minio_objects_expired,
         test_activities.input_gate,
@@ -549,7 +358,7 @@ def _worker_activity_list(test_activities: Activities):
 
 @pytest_asyncio.fixture(scope='function')
 async def temporal_test_env():
-    """Create Temporal test environment."""
+    """Temporal test environment with time-skipping."""
     env = await WorkflowEnvironment.start_time_skipping()
     async with env:
         yield env
@@ -557,23 +366,238 @@ async def temporal_test_env():
 
 @pytest_asyncio.fixture(scope='function')
 async def temporal_worker(temporal_test_env, test_activities):
-    """Create Temporal worker with test activities."""
+    """Temporal worker for full predictions-batch and child workflows."""
+    with ThreadPoolExecutor(max_workers=32) as activity_executor:
         async with Worker(
             temporal_test_env.client,
             task_queue='test-queue',
             workflows=[PredictionsBatch, PredictionProcess, FormatAndExportPrediction],
             activities=_worker_activity_list(test_activities),
+            activity_executor=activity_executor,
         ) as worker:
             yield worker
 
 
 @pytest_asyncio.fixture(scope='function')
 async def temporal_worker_real_minio(temporal_test_env, test_activities_real_minio):
-    """Temporal worker backed by Activities using real MinIO testcontainer."""
+    """Temporal worker alias for tests that emphasize MinIO behavior."""
+    with ThreadPoolExecutor(max_workers=32) as activity_executor:
         async with Worker(
             temporal_test_env.client,
             task_queue='test-queue',
             workflows=[PredictionsBatch, PredictionProcess, FormatAndExportPrediction],
             activities=_worker_activity_list(test_activities_real_minio),
+            activity_executor=activity_executor,
         ) as worker:
             yield worker
+
+
+def _drift_worker_activity_list(test_activities: Activities):
+    """Activity callables registered on the drift Temporal worker."""
+    return [
+        test_activities.load_custom_query,
+        test_activities.get_reference_data,
+        test_activities.calculate_drift,
+        test_activities.export_data_to_postgres,
+    ]
+
+
+@pytest_asyncio.fixture(scope='function')
+async def temporal_worker_drift(temporal_test_env, test_activities):
+    """Temporal worker registered with the Drift workflow and its activities."""
+    with ThreadPoolExecutor(max_workers=32) as activity_executor:
+        async with Worker(
+            temporal_test_env.client,
+            task_queue='test-queue',
+            workflows=[Drift],
+            activities=_drift_worker_activity_list(test_activities),
+            activity_executor=activity_executor,
+        ) as worker:
+            yield worker
+
+
+def _simple_metrics_worker_activity_list(test_activities: Activities):
+    """Activity callables registered on the simple-metrics Temporal worker."""
+    return [
+        test_activities.load_custom_query,
+        test_activities.calculate_simple_metrics,
+        test_activities.export_data_to_postgres,
+    ]
+
+
+@pytest_asyncio.fixture(scope='function')
+async def temporal_worker_simple_metrics(temporal_test_env, test_activities):
+    """Temporal worker registered with the SimpleMetrics workflow and its activities."""
+    with ThreadPoolExecutor(max_workers=32) as activity_executor:
+        async with Worker(
+            temporal_test_env.client,
+            task_queue='test-queue',
+            workflows=[SimpleMetrics],
+            activities=_simple_metrics_worker_activity_list(test_activities),
+            activity_executor=activity_executor,
+        ) as worker:
+            yield worker
+
+
+def _minimal_retrain_worker_activity_list(test_activities: Activities):
+    """Activity callables registered on the minimal-retrain Temporal worker."""
+    return [
+        test_activities.load_query_with_minio_offload,
+        test_activities.retrain_model,
+        test_activities.update_production_model,
+        test_activities.format_retrain_report,
+        test_activities.export_data_to_postgres,
+    ]
+
+
+@pytest_asyncio.fixture(scope='function')
+async def temporal_worker_minimal_retrain(temporal_test_env, test_activities):
+    """Temporal worker registered with the MinimalRetrain workflow and its activities."""
+    with ThreadPoolExecutor(max_workers=32) as activity_executor:
+        async with Worker(
+            temporal_test_env.client,
+            task_queue='test-queue',
+            workflows=[MinimalRetrain],
+            activities=_minimal_retrain_worker_activity_list(test_activities),
+            activity_executor=activity_executor,
+        ) as worker:
+            yield worker
+
+
+def _connect_activities_to_opc_server(activities: Activities, server_url: str) -> None:
+    """
+    Initialize OPC repositories and block until the E2E server session is ready.
+
+    Runs synchronously (typically via ``asyncio.to_thread``) so the asyncua test
+    server event loop is not blocked during ``Client.connect()``.
+
+    Args:
+        activities (Activities): Worker activities under test.
+        server_url (str): ``opc.tcp://`` URL from ``OpcE2ETestServer``.
+    """
+    activities.init_opc()
+    repo = activities.opc_repository['1']
+    deadline = time.monotonic() + 30.0
+    while time.monotonic() < deadline:
+        if repo._session_ready.is_set():
+            return
+        connected, _ = repo.connect()
+        if connected:
+            return
+        time.sleep(0.5)
+    raise RuntimeError(f'Could not connect OpcRepository to OPC E2E server at {server_url}')
+
+
+def _build_e2e_opc_config(server_url: str) -> dict[str, dict]:
+    """
+    OPC server config for E2E Activities pointing at an in-process asyncua server.
+
+    Args:
+        server_url (str): ``opc.tcp://`` endpoint from ``OpcE2ETestServer``.
+
+    Return:
+        dict: ``opc_config`` payload for ``Activities`` (server id ``1``).
+    """
+    return {
+        '1': {
+            'id': '1',
+            'server_name': 'e2e_opcua',
+            'url': server_url,
+            'server_uri': server_url,
+            'cert_path': None,
+            'private_key_path': None,
+            'server_cert_path': None,
+            'reconnection_interval': 0,
+        }
+    }
+
+
+@pytest_asyncio.fixture
+async def opc_e2e_server():
+    """In-process asyncua server with writable prediction/confidence nodes."""
+    server = OpcE2ETestServer()
+    await server.start()
+    await asyncio.sleep(0.5)
+    try:
+        yield server
+    finally:
+        await server.stop()
+
+
+@pytest_asyncio.fixture(scope='function')
+async def test_activities_real_opc(
+    postgres_container,
+    minio_container,
+    mock_logger,
+    notification_handler,
+    metrics_controller,
+    mlflow_repository_stub,
+    plugin_store_stub,
+    pi_web_api_client_stub,
+    opc_e2e_server: OpcE2ETestServer,
+):
+    """Activities with real OpcRepository connected to the in-process OPC UA server."""
+    minio_client = minio_container.get_client()
+    if not minio_client.bucket_exists('test-bucket'):
+        minio_client.make_bucket('test-bucket')
+    minio_port = minio_container.get_exposed_port(9000)
+
+    activities = Activities(
+        postgres_config={
+            'host': 'localhost',
+            'port': int(postgres_container.get_exposed_port(5432)),
+            'user': postgres_container.username,
+            'password': postgres_container.password,
+            'dbname': postgres_container.dbname,
+            'min_connections': 1,
+            'max_connections': 5,
+        },
+        plugin_store=plugin_store_stub,
+        minio_config={
+            'endpoint_url': f'localhost:{minio_port}',
+            'access_key': 'minioadmin',
+            'secret_key': 'minioadmin',
+            'default_bucket': 'test-bucket',
+            'retention_hours': 24,
+            'secure': False,
+        },
+        opc_config=_build_e2e_opc_config(opc_e2e_server.url),
+        pi_web_api_config={
+            'base_url': 'http://localhost:8080',
+            'auth_type': 'bearer',
+            'auth_token': 'test_token',
+        },
+        logger=mock_logger,
+        notification_handler=notification_handler,
+        metrics_controller=metrics_controller,
+        mlflow_repository=mlflow_repository_stub,
+    )
+    activities.pi_web_api_client = pi_web_api_client_stub
+    await asyncio.to_thread(_connect_activities_to_opc_server, activities, opc_e2e_server.url)
+    try:
+        yield activities
+    finally:
+        await asyncio.to_thread(_teardown_real_opc_activities, activities)
+
+
+def _teardown_real_opc_activities(activities: Activities) -> None:
+    """Disconnect OPC sessions and shut down activities (sync, for asyncio.to_thread)."""
+    for opc_repo in activities.opc_repository.values():
+        opc_repo.disconnect()
+    activities.shutdown()
+
+
+@pytest_asyncio.fixture(scope='function')
+async def temporal_worker_real_opc(temporal_test_env, test_activities_real_opc):
+    """Temporal worker using real OpcRepository against the in-process OPC UA server."""
+    with ThreadPoolExecutor(max_workers=32) as activity_executor:
+        async with Worker(
+            temporal_test_env.client,
+            task_queue='test-queue',
+            workflows=[PredictionsBatch, PredictionProcess, FormatAndExportPrediction],
+            activities=_worker_activity_list(test_activities_real_opc),
+            activity_executor=activity_executor,
+        ) as worker:
+            yield worker
+
+

e2e/db_schema.sql

@@ -0,0 +1,109 @@
+-- =============================================================================
+-- E2E test database schema for the ``sientia_data`` namespace.
+--
+-- Mirrors the production DDL one-to-one so any change in production can be
+-- pasted directly into this file. The conftest fixture loads this SQL into the
+-- testcontainers Postgres before each test run.
+--
+-- Notes on differences from production:
+--  * Tables that are partitioned in production (e.g. ``simple_metrics``,
+--    ``transformed_data``, ``drift_metrics``) are created as plain tables
+--    here because the test suite does not exercise partition pruning.
+--  * Indexes are intentionally omitted; tests rely on functional behavior,
+--    not query plans.
+-- =============================================================================
+
+CREATE SCHEMA IF NOT EXISTS sientia_data;
+
+-- -----------------------------------------------------------------------------
+-- sientia_data.laborious_data
+-- -----------------------------------------------------------------------------
+CREATE TABLE IF NOT EXISTS sientia_data.laborious_data (
+    model_id    int4 NOT NULL,
+    variable    text NOT NULL,
+    value       numeric NULL,
+    "timestamp" timestamptz NOT NULL,
+    created_at  timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
+    CONSTRAINT unique_timestamp_variable
+        UNIQUE (model_id, "timestamp", variable)
+);
+
+-- -----------------------------------------------------------------------------
+-- sientia_data.predictions
+-- -----------------------------------------------------------------------------
+CREATE TABLE IF NOT EXISTS sientia_data.predictions (
+    model_id              int4 NOT NULL,
+    prediction            numeric NULL,
+    prediction_confidence numeric NOT NULL,
+    response_time         numeric NOT NULL,
+    prediction_status     text NOT NULL,
+    "timestamp"           timestamptz NOT NULL,
+    created_at            timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
+    "comments"            text NULL,
+    CONSTRAINT unique_model_id_timestamp
+        UNIQUE (model_id, "timestamp")
+);
+
+-- -----------------------------------------------------------------------------
+-- sientia_data.transformed_data
+--   Production: PARTITION BY RANGE (created_at). Tests use a plain table.
+-- -----------------------------------------------------------------------------
+CREATE TABLE IF NOT EXISTS sientia_data.transformed_data (
+    id          SERIAL NOT NULL,
+    model_id    int4 NOT NULL,
+    variable    text NOT NULL,
+    value       numeric NULL,
+    "timestamp" timestamptz NOT NULL,
+    created_at  timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
+    PRIMARY KEY (id, created_at)
+);
+
+-- -----------------------------------------------------------------------------
+-- sientia_data.drift_metrics
+--   Production: PARTITION BY RANGE (created_at). Tests use a plain table.
+-- -----------------------------------------------------------------------------
+CREATE TABLE IF NOT EXISTS sientia_data.drift_metrics (
+    id               SERIAL NOT NULL,
+    model_id         text NOT NULL,
+    feature          text NULL,
+    method           text NOT NULL,
+    value            numeric NOT NULL,
+    alert            bool NOT NULL,
+    chunk_index      int4 NOT NULL,
+    chunk_start_date text NOT NULL,
+    chunk_end_date   text NOT NULL,
+    accurate         bool NOT NULL,
+    "timestamp"      timestamptz NULL,
+    created_at       timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
+    PRIMARY KEY (id, created_at)
+);
+
+-- -----------------------------------------------------------------------------
+-- sientia_data.simple_metrics
+--   Production: PARTITION BY RANGE (created_at). Tests use a plain table.
+-- -----------------------------------------------------------------------------
+CREATE TABLE IF NOT EXISTS sientia_data.simple_metrics (
+    id               SERIAL NOT NULL,
+    model_id         text NOT NULL,
+    metric           text NOT NULL,
+    value            numeric NOT NULL,
+    "timestamp"      timestamptz NULL,
+    data_size        int4 NOT NULL,
+    interval_minutes int4 NOT NULL,
+    created_at       timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
+    PRIMARY KEY (id, created_at)
+);
+
+-- -----------------------------------------------------------------------------
+-- sientia_data.log_retrain
+--   No primary key in production; all columns nullable.
+-- -----------------------------------------------------------------------------
+CREATE TABLE IF NOT EXISTS sientia_data.log_retrain (
+    mlflow_experiment_id int8 NULL,
+    mlflow_run_id        text NULL,
+    model_id             text NULL,
+    model_name           text NULL,
+    status               text NULL,
+    "timestamp"          timestamptz NULL,
+    "version"            text NULL
+);

e2e/helpers.py

@@ -3,13 +3,59 @@ Shared helpers for E2E tests (Temporal workflows + PostgreSQL).
 """
 
 import asyncio
+import json
 from datetime import datetime
 from decimal import Decimal
+from pathlib import Path
 from typing import Any
 
 from sqlalchemy import text
 from sqlalchemy.engine import Engine
 
+SCENARIO_INPUTS_DIR = Path(__file__).parent / 'scenario_inputs'
+
+
+def _replace_template_values(payload: Any, model_id: int) -> Any:
+    """
+    Replace string placeholders in scenario payloads with the concrete model id.
+
+    Args:
+        payload: JSON-like structure loaded from scenario input file.
+        model_id: Model id used to render template placeholders.
+
+    Return:
+        Any: Payload with ``{{MODEL_ID}}`` replaced where applicable.
+    """
+    if isinstance(payload, dict):
+        return {key: _replace_template_values(value, model_id) for key, value in payload.items()}
+    if isinstance(payload, list):
+        return [_replace_template_values(item, model_id) for item in payload]
+    if isinstance(payload, str):
+        if payload == '{{MODEL_ID}}':
+            return model_id
+        return payload.replace('{{MODEL_ID}}', str(model_id))
+    return payload
+
+
+def load_scenario_input(file_name: str, model_id: int | None = None) -> dict[str, Any]:
+    """
+    Load a scenario input JSON from ``e2e/scenario_inputs``.
+
+    Args:
+        file_name: JSON file name inside ``e2e/scenario_inputs``.
+        model_id: Optional model id used to render ``{{MODEL_ID}}`` placeholders.
+
+    Return:
+        dict[str, Any]: Input payload ready to be passed to workflow/activity calls.
+    """
+    file_path = SCENARIO_INPUTS_DIR / file_name
+    with file_path.open('r', encoding='utf-8') as f:
+        payload = json.load(f)
+
+    if model_id is not None:
+        return _replace_template_values(payload, model_id)
+    return payload
+
 
 async def start_and_await_workflow(client, workflow_run, input_data: dict, workflow_id: str, timeout: float = 60.0):
     """
@@ -34,7 +80,17 @@ async def start_and_await_workflow(client, workflow_run, input_data: dict, workf
     return await asyncio.wait_for(handle.result(), timeout=timeout)
 
 
-def insert_sample_data(postgres_engine: Engine, model_id: int, values: list[Any]) -> None:
+DEFAULT_BATCH_TIMESTAMP = '2024-01-01 12:00:00+00:00'
+DEFAULT_PREDICTION_HISTORY_TIMESTAMP = '2024-01-01 12:00:00+00:00'
+
+
+def insert_sample_data(
+    postgres_engine: Engine,
+    model_id: int,
+    values: list[Any],
+    *,
+    data_timestamp: str = DEFAULT_BATCH_TIMESTAMP,
+) -> None:
     """
     Replace laborious_data rows for a model_id with one row per value (sensor_1..n).
 
@@ -42,22 +98,106 @@ def insert_sample_data(postgres_engine: Engine, model_id: int, values: list[Any]
         postgres_engine: SQLAlchemy engine.
         model_id: Model id column value.
         values: Per-sensor values; use string 'NULL' for SQL NULL.
+        data_timestamp: Timestamp and created_at for every inserted row; drives
+            ``last_timestamp`` on the MinIO/query payload (max row time).
     """
     with postgres_engine.begin() as conn:
-        conn.execute(text(f'DELETE FROM predictions_schema.laborious_data WHERE model_id = {model_id}'))
+        conn.execute(text(f'DELETE FROM sientia_data.laborious_data WHERE model_id = {model_id}'))
         values_sql = []
         for i, value in enumerate(values):
             values_sql.append(f"""
-                ({model_id}, 'sensor_{i + 1}', {value}, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00')
+                ({model_id}, 'sensor_{i + 1}', {value}, '{data_timestamp}', '{data_timestamp}')
             """)
         insert_sql = f"""
-            INSERT INTO predictions_schema.laborious_data (model_id, variable, value, timestamp, created_at)
+            INSERT INTO sientia_data.laborious_data (model_id, variable, value, timestamp, created_at)
             VALUES
                 {', '.join(values_sql)}
         """
         conn.execute(text(insert_sql))
 
 
+def insert_sample_prediction(
+    postgres_engine: Engine,
+    model_id: int,
+    *,
+    prediction_timestamp: str = DEFAULT_PREDICTION_HISTORY_TIMESTAMP,
+) -> tuple[int, Decimal, Decimal, str]:
+    """
+    Insert a single historical prediction row for REPEAT scenarios.
+
+    Args:
+        postgres_engine: SQLAlchemy engine.
+        model_id: Model id.
+        prediction_timestamp: Row ``timestamp`` (unique with model_id in tests).
+
+    Return:
+        tuple: (model_id, prediction, prediction_confidence, prediction_status) for assertions.
+    """
+    with postgres_engine.begin() as conn:
+        conn.execute(text(f'DELETE FROM sientia_data.predictions WHERE model_id = {model_id}'))
+        insert_sql = f"""
+            INSERT INTO sientia_data.predictions (
+                model_id, timestamp, prediction, prediction_confidence, prediction_status, comments, response_time
+            )
+            VALUES (
+                {model_id}, '{prediction_timestamp}', 10, 0, 'Good', '', 0.1
+            )
+        """
+        conn.execute(text(insert_sql))
+    return (model_id, Decimal(10), Decimal(0), 'Good')
+
+
+def workflow_failure_message_chain(exc: BaseException) -> list[str]:
+    """
+    Collect ``str()`` / ``message`` from an exception and its ``__cause__`` chain.
+
+    Args:
+        exc: Root exception (e.g. from ``pytest.raises``).
+
+    Return:
+        list[str]: Messages from root to innermost cause.
+    """
+    messages: list[str] = []
+    current: BaseException | None = exc
+    while current is not None:
+        messages.append(getattr(current, 'message', None) or str(current) or repr(current))
+        current = current.__cause__
+    return messages
+
+
+def assert_postgres_unique_violation_in_chain(exc: BaseException) -> None:
+    """
+    Assert the exception chain mentions Postgres unique-constraint violation.
+
+    Args:
+        exc: Workflow or activity error from Temporal.
+
+    Raises:
+        AssertionError: If no link in the chain looks like UniqueViolation.
+    """
+    chain = ' | '.join(workflow_failure_message_chain(exc))
+    assert 'UniqueViolation' in chain or 'unique_model_id_timestamp' in chain, (
+        f'Expected unique constraint violation in error chain, got: {chain}'
+    )
+
+
+def assert_prediction_row_count(postgres_engine: Engine, model_id: int, expected: int) -> None:
+    """
+    Assert how many prediction rows exist for a model_id.
+
+    Args:
+        postgres_engine: SQLAlchemy engine.
+        model_id: Model id filter.
+        expected: Expected row count.
+    """
+    with postgres_engine.connect() as conn:
+        n = conn.execute(
+            text('SELECT COUNT(*) FROM sientia_data.predictions WHERE model_id = :m'),
+            {'m': model_id},
+        ).scalar()
+    assert n == expected, f'Expected {expected} prediction rows, got {n}'
+
+
 def assert_prediction(
     postgres_engine: Engine,
     model_id: int,
@@ -65,6 +205,7 @@ def assert_prediction(
     prediction_confidence: int | Decimal = 0,
     prediction_status: str = 'Good',
     comments: str = '',
+    comments_contains: str | None = None,
 ) -> None:
     """
     Assert exactly one prediction row exists for model_id with expected columns.
@@ -75,15 +216,14 @@ def assert_prediction(
         prediction: Expected prediction value.
         prediction_confidence: Expected confidence (int or Decimal for numeric column).
         prediction_status: Expected status string.
-        comments: Expected comments string.
+        comments: Expected exact comments string (ignored when ``comments_contains`` is set).
+        comments_contains: When set, assert this substring appears in comments.
     """
-    import pytest
-
     with postgres_engine.connect() as conn:
         result_query = conn.execute(
             text(
                 f'SELECT model_id, prediction, prediction_confidence, prediction_status, comments '
-                f'FROM predictions_schema.predictions WHERE model_id = {model_id} '
+                f'FROM sientia_data.predictions WHERE model_id = {model_id} '
                 f'ORDER BY created_at ASC'
             )
         )
@@ -98,7 +238,13 @@ def assert_prediction(
             str(prediction_confidence)
         ), f'Expected prediction_confidence={prediction_confidence}, got {row[2]}'
         assert row[3] == prediction_status, f"Expected prediction_status='{prediction_status}', got {row[3]}"
-        assert row[4] == comments, f"Expected comments='{comments}', got {row[4]}"
+        actual_comments = row[4] or ''
+        if comments_contains is not None:
+            assert comments_contains in actual_comments, (
+                f"Expected comments to contain '{comments_contains}', got '{actual_comments}'"
+            )
+        else:
+            assert actual_comments == comments, f"Expected comments='{comments}', got '{actual_comments}'"
 
 
 def assert_continue(
@@ -112,7 +258,7 @@ def assert_continue(
         result_query = conn.execute(
             text(
                 f'SELECT model_id, prediction, prediction_confidence, prediction_status, comments '
-                f'FROM predictions_schema.predictions WHERE model_id = {model_id}'
+                f'FROM sientia_data.predictions WHERE model_id = {model_id}'
             )
         )
         prediction_rows = result_query.fetchall()
@@ -132,7 +278,7 @@ def assert_stop(postgres_engine: Engine, model_id: int) -> None:
 
     with postgres_engine.connect() as conn:
         result_query = conn.execute(
-            text(f'SELECT COUNT(*) FROM predictions_schema.predictions WHERE model_id = {model_id}')
+            text(f'SELECT COUNT(*) FROM sientia_data.predictions WHERE model_id = {model_id}')
         )
         count = result_query.scalar()
         assert count == 0, f'Expected no predictions, but found {count} records'
@@ -155,7 +301,7 @@ def assert_repeat(postgres_engine: Engine, model_id: int, last_prediction: tuple
         result_query = conn.execute(
             text(
                 f'SELECT model_id, prediction, prediction_confidence, prediction_status '
-                f'FROM predictions_schema.predictions WHERE model_id = {model_id} '
+                f'FROM sientia_data.predictions WHERE model_id = {model_id} '
                 f'ORDER BY created_at ASC'
             )
         )
@@ -172,3 +318,51 @@ def assert_repeat(postgres_engine: Engine, model_id: int, last_prediction: tuple
 def make_workflow_id(prefix: str) -> str:
     """Build a unique workflow id using a prefix and current timestamp."""
     return f'{prefix}-{datetime.now().timestamp()}'
+
+
+def insert_target_data_for_drift(
+    postgres_engine: Engine,
+    model_id: int,
+    timestamps: list[str],
+    variables_values: dict[str, list[float]],
+) -> None:
+    """
+    Insert one row per (timestamp, variable) pair into ``laborious_data``.
+
+    Used by drift scenarios that need wide-format input where the pivot keeps a
+    full row for every timestamp.
+
+    Args:
+     - postgres_engine: SQLAlchemy engine bound to the test container.
+     - model_id: Model id stamped on every row.
+     - timestamps: ISO-8601 strings used both as ``timestamp`` and ``created_at``.
+     - variables_values: Mapping of variable name to a list of values; each list
+       must be the same length as ``timestamps``.
+    """
+    for var_name, values in variables_values.items():
+        if len(values) != len(timestamps):
+            raise ValueError(
+                f"Variable '{var_name}' has {len(values)} values but {len(timestamps)} timestamps"
+            )
+
+    rows_sql = []
+    for index, ts in enumerate(timestamps):
+        for var_name, values in variables_values.items():
+            rows_sql.append(
+                f"({model_id}, '{var_name}', {values[index]}, '{ts}', '{ts}')"
+            )
+
+    with postgres_engine.begin() as conn:
+        conn.execute(
+            text(f'DELETE FROM sientia_data.laborious_data WHERE model_id = {model_id}')
+        )
+        if rows_sql:
+            conn.execute(
+                text(
+                    'INSERT INTO sientia_data.laborious_data '
+                    '(model_id, variable, value, "timestamp", created_at) VALUES '
+                    + ', '.join(rows_sql)
+                )
+            )
+
+

e2e/opc_test_server.py

@@ -0,0 +1,189 @@
+"""
+In-process OPC UA server for E2E tests (asyncua).
+
+Provides writable prediction/confidence nodes and optional write faults
+(Tier-1 BadSessionIdInvalid via PreWrite callback).
+"""
+
+from __future__ import annotations
+
+import socket
+from dataclasses import dataclass
+from typing import TYPE_CHECKING
+
+from asyncua import Server, ua
+from asyncua.common.callback import CallbackType
+from asyncua.common.utils import ServiceError
+
+if TYPE_CHECKING:
+    from asyncua.common.node import Node
+
+
+UNKNOWN_NODE_ID = 'ns=99;i=9999'
+
+
+@dataclass(frozen=True)
+class OpcE2ENodeIds:
+    """NodeId strings used in opc_output_config for E2E workflows."""
+
+    prediction: str
+    confidence: str
+    unknown: str = UNKNOWN_NODE_ID
+
+
+class OpcE2ETestServer:
+    """
+    Ephemeral asyncua server with Laborious E2E variables and controllable faults.
+
+    Args:
+        host: Bind address (default 127.0.0.1).
+    """
+
+    def __init__(self, host: str = '127.0.0.1') -> None:
+        self._host = host
+        self._server: Server | None = None
+        self._prediction_node: Node | None = None
+        self._confidence_node: Node | None = None
+        self._session_bad_on_write = False
+        self._url: str | None = None
+        self._node_ids: OpcE2ENodeIds | None = None
+
+    @property
+    def url(self) -> str:
+        if self._url is None:
+            raise RuntimeError('OPC E2E server is not started')
+        return self._url
+
+    @property
+    def node_ids(self) -> OpcE2ENodeIds:
+        if self._node_ids is None:
+            raise RuntimeError('OPC E2E server is not started')
+        return self._node_ids
+
+    def set_session_bad_on_write(self, enabled: bool) -> None:
+        """
+        When enabled, every client Write is rejected with BadSessionIdInvalid.
+
+        Args:
+            enabled (bool): Turn Tier-1 session fault injection on or off.
+        """
+        self._session_bad_on_write = enabled
+
+    async def start(self) -> OpcE2ENodeIds:
+        """
+        Start the OPC UA server on a free TCP port.
+
+        Return:
+            OpcE2ENodeIds: NodeId strings for prediction and confidence tags.
+        """
+        port = _free_port(self._host)
+        self._url = f'opc.tcp://{self._host}:{port}/freeopcua/server/'
+
+        server = Server()
+        server.set_endpoint(self._url)
+        await server.init()
+        server.iserver.callback_service.addListener(
+            CallbackType.PreWrite,
+            self._pre_write_callback,
+        )
+
+        idx = await server.register_namespace('http://sientia.test/laborious-e2e')
+        e2e_object = await server.nodes.objects.add_object(idx, 'LaboriousE2E')
+        prediction = await e2e_object.add_variable(
+            idx,
+            'Prediction',
+            ua.Variant(0.0, ua.VariantType.Float),
+        )
+        confidence = await e2e_object.add_variable(
+            idx,
+            'Confidence',
+            ua.Variant(0.0, ua.VariantType.Float),
+        )
+        await prediction.set_writable()
+        await confidence.set_writable()
+
+        await server.start()
+        self._server = server
+        self._prediction_node = prediction
+        self._confidence_node = confidence
+        self._node_ids = OpcE2ENodeIds(
+            prediction=prediction.nodeid.to_string(),
+            confidence=confidence.nodeid.to_string(),
+        )
+        return self._node_ids
+
+    async def stop(self) -> None:
+        """Stop the OPC UA server and release the listening port."""
+        if self._server is not None:
+            await self._server.stop()
+            self._server = None
+        self._prediction_node = None
+        self._confidence_node = None
+        self._url = None
+        self._node_ids = None
+        self._session_bad_on_write = False
+
+    async def read_prediction(self) -> float:
+        """
+        Read the current prediction variable value from the address space.
+
+        Return:
+            float: Stored prediction value.
+        """
+        if self._prediction_node is None:
+            raise RuntimeError('OPC E2E server is not started')
+        value = await self._prediction_node.read_value()
+        return float(value)
+
+    async def read_confidence(self) -> float:
+        """
+        Read the current confidence variable value from the address space.
+
+        Return:
+            float: Stored confidence value.
+        """
+        if self._confidence_node is None:
+            raise RuntimeError('OPC E2E server is not started')
+        value = await self._confidence_node.read_value()
+        return float(value)
+
+    async def _pre_write_callback(self, _event, _service) -> None:
+        if self._session_bad_on_write:
+            raise ServiceError(ua.StatusCodes.BadSessionIdInvalid)
+
+
+def _free_port(host: str) -> int:
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
+        sock.bind((host, 0))
+        return int(sock.getsockname()[1])
+
+
+def build_opc_output_config(
+    node_ids: OpcE2ENodeIds,
+    *,
+    prediction_tag: str | None = None,
+    confidence_tag: str | None = None,
+    prediction_only: bool = False,
+    server_key: str = '1',
+) -> dict[str, dict]:
+    """
+    Build opc_output_config for PredictionsBatch using real server NodeIds.
+
+    Args:
+        node_ids (OpcE2ENodeIds): Node ids from OpcE2ETestServer.
+        prediction_tag (str | None): Override prediction NodeId (default: node_ids.prediction).
+        confidence_tag (str | None): Override confidence NodeId (default: node_ids.confidence).
+        prediction_only (bool): When True, omit confidence_tags (single write per activity).
+        server_key (str): OPC server id key in opc_output_config.
+
+    Return:
+        dict: opc_output_config payload for workflow input.
+    """
+    pred = prediction_tag if prediction_tag is not None else node_ids.prediction
+    conf = confidence_tag if confidence_tag is not None else node_ids.confidence
+    server_config: dict = {
+        'prediction_tags': {pred: {'data_type': 'float'}},
+    }
+    if not prediction_only:
+        server_config['confidence_tags'] = {conf: {'data_type': 'float'}}
+    return {server_key: server_config}

e2e/scenario_inputs/drift_base.json

@@ -0,0 +1,14 @@
+{
+  "schedule_name": "test-schedule",
+  "model_name": "test_model",
+  "model_id": "{{MODEL_ID}}",
+  "schema": "sientia_data",
+  "source_table_name": "laborious_data",
+  "target_table_name": "drift_metrics",
+  "interval": 60,
+  "drift_metrics": ["kolmogorov_smirnov", "jensen_shannon", "wasserstein"],
+  "chunk_period": "min",
+  "model_config": {
+    "target": "sensor_1"
+  }
+}

e2e/scenario_inputs/format_export_base.json

@@ -0,0 +1,37 @@
+{
+  "schedule_name": "test-schedule",
+  "model_name": "test_model",
+  "model_id": "{{MODEL_ID}}",
+  "query": "SELECT timestamp, variable, value, created_at FROM sientia_data.laborious_data WHERE model_id = {{MODEL_ID}}",
+  "schema": "sientia_data",
+  "table_name": "predictions",
+  "transform_table_name": "transformed_data",
+  "input_filters": {
+    "EMPTY_DATA": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "mlflow_transform_filters": {
+    "API_ERROR": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "mlflow_predict_filters": {
+    "API_ERROR": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "path_priority": ["STOP", "CONTINUE", "REPEAT"],
+  "opc_output_config": {},
+  "pi_web_api_output_config": {},
+  "save_transform": true,
+  "prediction_store_policy": "lts:1",
+  "model_config": {
+    "retention_minutes": 0,
+    "target": "sensor_1"
+  },
+  "datetime_columns": ["timestamp", "created_at"]
+}

e2e/scenario_inputs/main_happy_path.json

@@ -0,0 +1,45 @@
+{
+  "metadata": {
+    "metadata": {
+      "model_id": "{{MODEL_ID}}",
+      "model_name": "test_model",
+      "schedule_name": "test-schedule",
+      "workflow_name": "predictions_batch"
+    }
+  },
+  "schedule_name": "test-schedule",
+  "model_name": "test_model",
+  "model_id": "{{MODEL_ID}}",
+  "query": "SELECT timestamp, variable, value, created_at FROM sientia_data.laborious_data WHERE model_id = {{MODEL_ID}}",
+  "schema": "sientia_data",
+  "table_name": "predictions",
+  "transform_table_name": "transformed_data",
+  "input_filters": {
+    "EMPTY_DATA": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "mlflow_transform_filters": {
+    "API_ERROR": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "mlflow_predict_filters": {
+    "API_ERROR": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "path_priority": ["STOP", "CONTINUE", "REPEAT"],
+  "opc_output_config": {},
+  "pi_web_api_output_config": {},
+  "save_transform": true,
+  "prediction_store_policy": "lts:1",
+  "model_config": {
+    "target": "sensor_1",
+    "retention_minutes": 0
+  },
+  "datetime_columns": ["timestamp", "created_at"]
+}

e2e/scenario_inputs/main_invalid_datetime.json

@@ -0,0 +1,45 @@
+{
+  "metadata": {
+    "metadata": {
+      "model_id": "{{MODEL_ID}}",
+      "model_name": "test_model",
+      "schedule_name": "test-schedule",
+      "workflow_name": "predictions_batch"
+    }
+  },
+  "schedule_name": "test-schedule",
+  "model_name": "test_model",
+  "model_id": "{{MODEL_ID}}",
+  "query": "SELECT timestamp, variable, value FROM sientia_data.laborious_data WHERE model_id = {{MODEL_ID}}",
+  "schema": "sientia_data",
+  "table_name": "predictions",
+  "transform_table_name": "transformed_data",
+  "input_filters": {
+    "EMPTY_DATA": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "mlflow_transform_filters": {
+    "API_ERROR": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "mlflow_predict_filters": {
+    "API_ERROR": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "path_priority": ["STOP", "CONTINUE", "REPEAT"],
+  "opc_output_config": {},
+  "pi_web_api_output_config": {},
+  "save_transform": true,
+  "prediction_store_policy": "lts:1",
+  "model_config": {
+    "target": "sensor_1",
+    "retention_minutes": 0
+  },
+  "datetime_columns": ["nonexistent_column"]
+}

e2e/scenario_inputs/main_missing_required.json

@@ -0,0 +1,16 @@
+{
+  "metadata": {
+    "metadata": {
+      "model_id": "{{MODEL_ID}}",
+      "model_name": "test_model",
+      "schedule_name": "test-schedule",
+      "workflow_name": "predictions_batch"
+    }
+  },
+  "schedule_name": "test-schedule",
+  "model_name": "test_model",
+  "model_id": "{{MODEL_ID}}",
+  "schema": "sientia_data",
+  "table_name": "predictions",
+  "transform_table_name": "transformed_data"
+}

e2e/scenario_inputs/main_sql_error.json

@@ -0,0 +1,44 @@
+{
+  "metadata": {
+    "metadata": {
+      "model_id": "{{MODEL_ID}}",
+      "model_name": "test_model",
+      "schedule_name": "test-schedule",
+      "workflow_name": "predictions_batch"
+    }
+  },
+  "schedule_name": "test-schedule",
+  "model_name": "test_model",
+  "model_id": "{{MODEL_ID}}",
+  "query": "SELECT * FROM nonexistent_table WHERE invalid_syntax =",
+  "schema": "sientia_data",
+  "table_name": "predictions",
+  "transform_table_name": "transformed_data",
+  "input_filters": {
+    "EMPTY_DATA": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "mlflow_transform_filters": {
+    "API_ERROR": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "mlflow_predict_filters": {
+    "API_ERROR": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "path_priority": ["STOP", "CONTINUE", "REPEAT"],
+  "opc_output_config": {},
+  "pi_web_api_output_config": {},
+  "save_transform": true,
+  "prediction_store_policy": "lts:1",
+  "model_config": {
+    "target": "sensor_1",
+    "retention_minutes": 0
+  }
+}

e2e/scenario_inputs/minimal_retrain_base.json

@@ -0,0 +1,12 @@
+{
+  "schedule_name": "test-schedule",
+  "model_name": "test_model",
+  "model_id": "{{MODEL_ID}}",
+  "query": "SELECT timestamp, variable, value, created_at FROM sientia_data.laborious_data WHERE model_id = {{MODEL_ID}}",
+  "schema": "sientia_data",
+  "table_name": "log_retrain",
+  "datetime_columns": ["timestamp", "created_at"],
+  "model_config": {
+    "target": "sensor_1"
+  }
+}

e2e/scenario_inputs/minio_offload_load_query.json

@@ -0,0 +1,11 @@
+{
+  "metadata": {
+    "schedule_name": "test-schedule",
+    "model_name": "test_model",
+    "model_id": "{{MODEL_ID}}",
+    "workflow_name": "predictions_batch"
+  },
+  "query": "SELECT timestamp, variable, value, created_at FROM sientia_data.laborious_data WHERE model_id = {{MODEL_ID}}",
+  "model_name": "test_model",
+  "datetime_columns": ["timestamp", "created_at"]
+}

e2e/scenario_inputs/minio_offload_workflow.json

@@ -0,0 +1,37 @@
+{
+  "schedule_name": "test-schedule",
+  "model_name": "test_model",
+  "model_id": "{{MODEL_ID}}",
+  "query": "SELECT timestamp, variable, value, created_at FROM sientia_data.laborious_data WHERE model_id = {{MODEL_ID}}",
+  "schema": "sientia_data",
+  "table_name": "predictions",
+  "transform_table_name": "transformed_data",
+  "input_filters": {
+    "EMPTY_DATA": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "mlflow_transform_filters": {
+    "API_ERROR": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "mlflow_predict_filters": {
+    "API_ERROR": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "path_priority": ["STOP", "CONTINUE", "REPEAT"],
+  "opc_output_config": {},
+  "pi_web_api_output_config": {},
+  "save_transform": false,
+  "prediction_store_policy": "lts:1",
+  "model_config": {
+    "retention_minutes": 0,
+    "target": "sensor_1"
+  },
+  "datetime_columns": ["timestamp", "created_at"]
+}

e2e/scenario_inputs/prediction_process_base.json

@@ -0,0 +1,39 @@
+{
+  "schedule_name": "test-schedule",
+  "model_name": "test_model",
+  "model_id": "{{MODEL_ID}}",
+  "query": "SELECT timestamp, variable, value, created_at FROM sientia_data.laborious_data WHERE model_id = {{MODEL_ID}}",
+  "schema": "sientia_data",
+  "table_name": "predictions",
+  "transform_table_name": "transformed_data",
+  "input_filters": {
+    "SPECIFIC_VARIABLES_NULL_VALUES": {
+      "POLICY": "CONTINUE",
+      "CONFIG": {
+        "variables": ["sensor_1"]
+      }
+    }
+  },
+  "mlflow_transform_filters": {
+    "API_ERROR": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "mlflow_predict_filters": {
+    "API_ERROR": {
+      "POLICY": "STOP",
+      "CONFIG": {}
+    }
+  },
+  "path_priority": ["STOP", "CONTINUE", "REPEAT"],
+  "opc_output_config": {},
+  "pi_web_api_output_config": {},
+  "save_transform": true,
+  "prediction_store_policy": "lts:1",
+  "model_config": {
+    "retention_minutes": 0,
+    "target": "sensor_1"
+  },
+  "datetime_columns": ["timestamp", "created_at"]
+}

e2e/scenario_inputs/simple_metrics_base.json

@@ -0,0 +1,14 @@
+{
+  "schedule_name": "test-schedule",
+  "model_name": "test_model",
+  "model_id": "{{MODEL_ID}}",
+  "schema": "sientia_data",
+  "predictions_table_name": "predictions",
+  "data_table_name": "laborious_data",
+  "target_table_name": "simple_metrics",
+  "interval_minutes": 60,
+  "metrics": ["rmse", "mse", "mae", "r2"],
+  "model_config": {
+    "target": "sensor_target"
+  }
+}

e2e/scenarios.md

@@ -1,520 +1,455 @@
-# Test Scenarios for Predictions Batch Workflow
+# E2E Scenario Documentation - Predictions Batch
 
-This document describes all possible test scenarios for the `predictions_batch` workflow and its child workflows `prediction_process` and `format_and_export_prediction`.
+This document describes the end-to-end scenarios for `predictions_batch` and its child workflows:
+`prediction_process` and `format_and_export_prediction`.
 
-## Running automated E2E tests (`e2e/`)
+It is a functional reference of scenario behavior, inputs, and expected outcomes.
 
-- **Runtime**: Docker (or a Docker-compatible daemon) must be available so [testcontainers](https://testcontainers.com/) can start **PostgreSQL** and **MinIO** containers.
+## Execution Context
-- **Dependencies**: install dev requirements (includes `testcontainers[postgres,minio]`).
-- **Invocation**: run only integration-marked tests, for example: `pytest e2e/ -m integration`.
-- **MinIO tests**: `e2e/test_minio_offload.py` exercises real S3 uploads; other E2E modules continue to mock MinIO on the worker used by most scenarios.
 
-## Workflow Overview
+- Tests run under `e2e/` and are marked with `@pytest.mark.integration`.
+- PostgreSQL and MinIO are provisioned with testcontainers.
+- `test_minio_offload.py` uses real MinIO I/O; other scenario suites may use stubs/mocks for optional outputs.
+- Real OPC UA scenarios use `@pytest.mark.opc` and an in-process asyncua server (`e2e/test_opc_real_server.py`).
 
-The `predictions_batch` workflow:
+### Local validation
-1. Loads data using a custom SQL query
+
-2. Prepares prediction configuration
+Use the existing project virtualenv and the shared `validate` script for unit/quality gates; run E2E separately (Docker required).
-3. Delegates to `prediction_process` child workflow which:
+
-   - Retrieves last timestamp for incremental processing
+```bash
-   - Applies input data quality gates
+source ./venv/bin/activate
-   - Executes MLFlow transform operation
+
-   - Validates transform response
+# Auto-fix + static checks (no pytest)
-   - Executes MLFlow predict operation
+validate --fix --project-name=laborious
-   - Validates predict response
+
-   - Delegates to `format_and_export_prediction` child workflow
+# Full unit + quality gate
-4. The `format_and_export_prediction` workflow:
+validate --project-name=laborious
-   - Formats prediction data (normal or default)
+
-   - Exports to PI Web API (optional)
+# E2E (integration)
-   - Exports to OPC server (optional)
+pytest e2e/ --override-ini testpaths=e2e -m integration
-   - Exports to PostgreSQL
+
-   - Writes metrics
+# E2E (real OPC server only)
+pytest e2e/test_opc_real_server.py --override-ini testpaths=e2e -m opc
+```
 
 ---
 
-## 1. Predictions Batch - Main Workflow Scenarios
+## 1. Main Workflow Scenarios
+Source: `e2e/test_predictions_batch_main_workflow.py`
 
-### 1.1 Success Scenarios
+### 1.1.1 Happy Path - Complete Success
+**Summary**: Full workflow succeeds with valid query and default gate behavior.
 
-#### Scenario 1.1.1: Happy Path - Complete Success
+**Description**:
-**Description**: Workflow completes successfully with valid SQL query and all activities succeed
+- Query returns rows for a model.
+- `prediction_process` runs transform and predict paths.
+- Final prediction and transformed data are persisted.
 
-**Input**:
+**Expected Outcome**:
-- Valid `schedule_name`, `model_name`, `model_id`
+- Exactly one prediction row is created.
-- Valid `query` returning non-empty DataFrame
+- Transform rows are created.
-- Valid `schema`, `table_name`, `transform_table_name`
+- Confidence/status/comments are success values.
-- Optional `datetime_columns` for timestamp parsing
-- Optional `input_filters`, `mlflow_transform_filters`, `mlflow_predict_filters`
-- Optional `path_priority`, `opc_output_config`, `pi_web_api_output_config`
 
-**Expected Behavior**:
+### 1.2.1 SQL Query Execution Error
-- `load_custom_query` returns DataFrame with data
+**Summary**: Invalid SQL leads to no persisted prediction.
-- Workflow prepares prediction input with all configurations
-- `prediction_process` child workflow executes successfully
-- All gates pass with no issues
-- Transform and predict operations succeed
-- Data exported to PostgreSQL
-- Metrics written
 
-**Assertions**:
+**Description**:
-- SQL query executed once
+- Input query is invalid.
-- `prediction_process` workflow called with correct parameters
+- Load step fails and workflow follows error/short-circuit path.
-- Data exists in PostgreSQL (predictions table)
+
-- Metrics recorded
+**Expected Outcome**:
-- No errors raised
+- No prediction rows for the model.
+- Workflow does not require retry-loop assumptions in assertions.
+
+### 1.2.2 Missing Required Parameters
+**Summary**: Missing required fields prevent workflow completion path.
+
+**Description**:
+- Required input key (e.g. `query`) is omitted.
+- Workflow fails to produce actionable input for child flow.
+
+**Expected Outcome**:
+- No prediction rows are persisted.
+- Workflow handle may require explicit terminate in E2E harness.
+
+### 1.2.3 Invalid Datetime Column Specification (de-prioritized)
+**Summary**: Legacy invalid datetime-column case is retained only as low-priority legacy coverage.
+
+**Description**:
+- `datetime_columns` references non-existing columns.
+- Behavior may vary by query shape and parser fallback.
+
+**Expected Outcome**:
+- No predictions persisted in the covered legacy assertion path.
+- Scenario is not considered primary behavior coverage.
 
 ---
 
-### 1.2 Error Scenarios
+## 2. Prediction Process Scenarios
+Source: `e2e/test_predictions_batch_prediction_process.py`
 
-#### Scenario 1.2.1: SQL Query Execution Error
+### 2.1 Input Gate Path Decisions
-**Description**: SQL query fails due to syntax error or connection issue
 
-**Input**:
+#### 2.1.1 CONTINUE
-- Invalid SQL query (syntax error)
+**Summary**: Input filter flags quality issue but allows continuation via default path.
-- Or database connection unavailable
 
-**Expected Behavior**:
+**Description**:
-- `load_custom_query` raises exception (caught by Temporal retry policy)
+- Input gate returns `CONTINUE`.
-- Notification sent with SQL error details
+- MLFlow transform/predict are skipped.
-- After retries, activity may return empty data or workflow may fail
+- Export path persists default-style prediction with warning context.
-- If empty data returned, workflow completes with early exit via input gate
 
-**Assertions**:
+#### 2.1.2 STOP
-- Error notification sent
+**Summary**: Input filter blocks processing.
-- Workflow completes (either fails or exits early)
+
-- No data in predictions table
+**Description**:
+- Input gate returns `STOP`.
+- Workflow exits without export.
+
+#### 2.1.3 REPEAT with history
+**Summary**: Prior prediction is reused.
+
+**Description**:
+- Input gate returns `REPEAT`.
+- `repeat_last_prediction` path is executed using existing historical row.
+
+#### 2.1.4 REPEAT without history
+**Summary**: Repeat requested but no previous prediction exists.
+
+**Description**:
+- Input gate returns `REPEAT`.
+- No prior row is available to duplicate.
+
+**Expected Outcome**:
+- No new prediction rows are created for the model.
+
+### 2.2 Transform Gate Decisions
+
+#### 2.2.1 CONTINUE on transform response error
+**Summary**: Transform response is degraded, but workflow continues.
+
+#### 2.2.2 STOP on transform response error
+**Summary**: Transform response error blocks downstream processing.
+
+#### 2.2.3 REPEAT on transform response error
+**Summary**: Transform response error triggers repeat-last-prediction path.
+
+#### 2.2.4 STOP on transform content NaN
+**Summary**: Content gate (`NAN_VALUES`) blocks on all-NaN transform payload.
+
+### 2.3 Predict Gate Decisions
+
+#### 2.3.1 CONTINUE on predict response error
+**Summary**: Predict response degraded; workflow exports with degraded metadata.
+
+#### 2.3.2 STOP on predict response error
+**Summary**: Predict response error blocks export.
+
+#### 2.3.3 REPEAT on predict response error
+**Summary**: Predict response error routes to repeat-last-prediction.
+
+### 2.4.1 Priority Conflict Resolution
+**Summary**: Deterministic selection when multiple filters produce different flags.
+
+**Description**:
+- Multiple filters may produce `STOP`, `CONTINUE`, and/or `REPEAT`.
+- `path_priority` defines precedence.
+
+**Expected Outcome**:
+- Highest-priority flag is applied consistently.
+- Executed branch matches configured priority ordering.
 
 ---
 
-#### Scenario 1.2.2: Missing Required Parameters
+## 3. Format and Export Scenarios
-**Description**: Essential parameters missing from input
+Source: `e2e/test_predictions_batch_format_export.py`
 
-**Input**:
+### 3.1 Output Combination Scenarios
-- Missing `query` or `model_id` or `schema` or `table_name`
 
-**Expected Behavior**:
+#### 3.1.1 Default prediction export
-- Workflow or activity raises KeyError or validation error
+**Summary**: Non-`None` path flag uses `format_default_prediction`.
-- Workflow fails immediately
 
-**Assertions**:
+**Description**:
-- Workflow fails with parameter error
+- Default prediction is generated.
-- Error notification sent
+- Transform export is skipped.
-- No child workflow called
+- Optional outputs (PI/OPC) still execute when configured.
+
+#### 3.1.2 OPC only
+**Summary**: Postgres + OPC writes, PI Web API disabled.
+
+#### 3.1.3 PI Web API only
+**Summary**: Postgres + PI writes, OPC disabled.
+
+#### 3.1.4 Postgres only
+**Summary**: Both optional outputs disabled; only Postgres persistence and metrics.
+
+#### 3.1.5 No transformed data export
+**Summary**: Prediction is persisted; transformed table is not written.
+
+### 3.2 Degraded-but-successful Completion
+
+#### 3.2.1 PI Web API write error
+**Summary**: PI write failure does not fail workflow.
+
+**Expected Outcome**:
+- Workflow completes.
+- Prediction persisted with degraded confidence/comments (PI error semantics).
+
+#### 3.2.2 OPC write error
+**Summary**: OPC write failure does not fail workflow.
+
+**Expected Outcome**:
+- Workflow completes.
+- Prediction persisted with OPC degraded confidence/comments.
+
+#### 3.2.3 PI Web API partial write error
+**Summary**: Partial PI acknowledgement is treated as degraded success.
+
+**Expected Outcome**:
+- Workflow completes.
+- Prediction persisted with PI error confidence and descriptive comment.
+
+#### 3.2.4 OPC session / channel error (confidence 14)
+**Summary**: Tier-1 `BadSessionIdInvalid` (or equivalent session error) degrades the prediction without failing the workflow.
+
+**Sources**:
+- Mock: `e2e/test_predictions_batch_format_export.py::test_scenario_3_2_4_opc_session_bad_mock`
+- Real server: `e2e/test_opc_real_server.py::test_scenario_3_2_4_opc_session_bad_real_server` (`@pytest.mark.opc`)
+
+**Expected Outcome**:
+- Workflow completes.
+- `prediction_confidence` is 14.
+- Comments contain `OPC UA session/channel error: BadSessionIdInvalid`.
+
+#### 3.2.5 OPC write blocked during reconnect (confidence 14)
+**Summary**: While reconnect holds the repository connection lock, writes fail fast with `reconnect_in_progress`.
+
+**Sources**:
+- Mock: `e2e/test_predictions_batch_format_export.py::test_scenario_3_2_5_opc_reconnect_in_progress_mock`
+- Real server: `e2e/test_opc_real_server.py::test_scenario_3_2_5_opc_write_blocked_during_reconnect_real_server` (`@pytest.mark.opc`)
+
+**Expected Outcome**:
+- Workflow completes.
+- `prediction_confidence` is 14.
+- Comments contain `OPC UA reconnect in progress`.
+
+### 3.3.1 Combined Optional Outputs (PI + OPC)
+**Summary**: Both external output channels are enabled together.
+
+**Description**:
+- PI Web API and OPC configs are both present.
+- Output mutation order matters for final persisted payload.
+
+**Expected Outcome**:
+- PI write executes before OPC write in workflow sequence.
+- Final Postgres payload reflects any confidence/comment updates.
+- OPC metrics are emitted when tag writes return response times.
 
 ---
 
-#### Scenario 1.2.3: Invalid Datetime Column Specification
+## 4. MinIO Offload Scenarios
-**Description**: Datetime column specified doesn't exist in query results
+Source: `e2e/test_minio_offload.py`
 
-**Input**:
+### 4.1.1 Forced offload to MinIO
-- `datetime_columns: ['nonexistent_column']`
+**Summary**: Very low threshold forces parquet upload.
-- Query results don't have this column
 
-**Expected Behavior**:
+**Description**:
-- `load_custom_query` may raise KeyError or warning
+- Payload is offloaded (`object_key` present, inline data absent/empty).
-- Depending on implementation, workflow may fail or continue
+- Object is present in MinIO under `prediction_datasets/...`.
-- Error notification sent
+- Retrieval reconstructs the dataframe.
 
-**Assertions**:
+### 4.1.2 Full workflow with offloaded load payload
-- Error raised or warning logged
+**Summary**: Offload path works during full `predictions_batch` execution.
-- Workflow behavior depends on error handling policy
+
+**Expected Outcome**:
+- Workflow completes.
+- Prediction row is persisted.
+
+### 4.2.1 Inline payload below threshold
+**Summary**: Data remains inline when threshold is not exceeded.
+
+**Expected Outcome**:
+- Payload stores inline `data`.
+- `object_key` is `None`.
+- Downstream persistence behavior matches offload scenario semantics.
 
 ---
 
-## 2. Prediction Process - Child Workflow Scenarios
+## 5. Drift Workflow Scenarios
+Source: `e2e/test_drift.py`
 
-### 2.1 Input gate Early Exit Scenarios
+The drift suite drives the **real** `sientia_model.analytics.drift_analysis.DriftAnalysis`
+analyzer (no stubs / mocks). Each scenario exercises the full pipeline:
 
-#### Scenario 2.1.1: Input Gate Triggers CONTINUE
+```
-**Description**: Input gate determines data should use previous prediction
+laborious_data (Postgres) -> load_custom_query
+                          -> calculate_drift (DriftAnalysis univariate + multivariate)
+                          -> export_data_to_postgres (sientia_data.drift_metrics)
+```
 
-**Input**:
+The `mlflow_repository_stub` provides the reference-data CSV via
-- Data that should continue with input data as prediction
+`download_artifacts`, and tests assert postgres rows in
-- `input_filters` configured with `POLICY: 'CONTINUE'`
+`sientia_data.drift_metrics` against this canonical schema:
-- `path_priority` includes CONTINUE
 
-**Expected Behavior**:
+`id, model_id, feature, method, value, alert, chunk_index, chunk_start_date, chunk_end_date, accurate, timestamp, created_at`.
-- `input_gate` returns `path_flag='CONTINUE'`
-- `path_flag_handler` calls export workflow with input data directly
-- MLFlow transform and predict skipped
-- Data exported as-is
 
-**Assertions**:
+Tests assert behavioral / structural properties (column presence, NOT NULL
-- `input_gate` called
+constraints, business-key invariants like uniform `timestamp` and stamped
-- MLFlow operations NOT called
+`model_id`) rather than exact numeric drift scores, since those depend on
-- Export workflow called with original data
+the real analyzer implementation and the synthetic data fed in.
-- Workflow completes
 
+### 5.1 Happy paths
 
-#### Scenario 2.1.2: Input Gate Triggers STOP
+#### D.1.1 Full pipeline persists all columns with reference data
-**Description**: Input data quality gate fails with STOP policy
+**Summary**: 10 minutes of target data are inserted; a 10-row reference CSV
+is configured via the MLflow stub. The `DriftAnalysis` runs end-to-end.
 
-**Input**:
+**Expected Outcome**:
-- Data with EMPTY_DATA or other critical issues
+- One row per `(chunk_index, feature, method)` plus a `multivariate` block
-- `input_filters` configured with `POLICY: 'STOP'`
+  per chunk is persisted.
+- Every column in the DDL is populated; `feature` is the only nullable column
+  per the new schema.
+- `accurate=True` for every row (reference path).
+- All three default univariate methods reach the analyzer.
+- `model_id` is stamped as `text` and uniform across rows.
+- `timestamp` equals `max(target_data.timestamp)` and is uniform across rows.
+- `chunk_start_date` / `chunk_end_date` are persisted as ISO text and ordered.
+- `p_value` is dropped before persistence.
 
-**Expected Behavior**:
+#### D.1.2 30% fallback when reference data is unavailable
-- `input_gate` returns `path_flag='STOP'`
+**Summary**: MLflow alias resolution is forced to fail so
-- `path_flag_handler` detects STOP
+`get_reference_data` returns `None`; `calculate_drift` falls back to the
-- Workflow returns early without calling MLFlow
+first 30% of target rows as reference.
-- No prediction exported
 
-**Assertions**:
+**Expected Outcome**:
-- `input_gate` called
+- All persisted rows carry `accurate=False`.
-- `path_flag_handler` returns True (early exit)
+- A `MODEL_METRICS_REFERENCE_DATA_WARNING` notification is emitted to MongoDB.
-- MLFlow transform NOT called
-- Export workflow NOT called
-- Workflow completes without error
 
+### 5.2 Failure paths
 
-#### Scenario 2.1.3: Input Gate Triggers REPEAT
+#### D.3.1 Empty target data short-circuits the workflow
-**Description**: Input gate determines data should repeat last prediction
+**Summary**: `load_custom_query` returns no rows.
 
-**Input**:
+**Expected Outcome**:
-- Data with quality issues that require using previous prediction
+- The workflow returns early and writes nothing to `sientia_data.drift_metrics`.
-- `input_filters` configured with `POLICY: 'REPEAT'`
-- `path_priority` includes REPEAT
 
-**Expected Behavior**:
+### 5.3 Configuration paths
-- `input_gate` returns `path_flag='REPEAT'`
-- `path_flag_handler` calls `repeat_last_prediction` activity
-- MLFlow transform and predict skipped
-- Last prediction repeated and exported
 
-**Assertions**:
+#### D.4.2 Invalid `chunk_period` raises ValueError
-- `input_gate` called
+**Summary**: Anything other than `min` / `s` is rejected by `calculate_drift`.
-- MLFlow operations NOT called
+
-- `repeat_last_prediction` activity called
+**Expected Outcome**:
-- Workflow completes
+- The workflow surfaces the `ValueError` ("Invalid chunk period: ...").
+- No rows are persisted.
+
+#### D.4.3 `chunk_period='s'` preserves seconds in `chunk_start_date`
+**Summary**: Target data spans two minutes with samples at second-30
+boundaries; the activity is configured with `chunk_period='s'`.
+
+**Expected Outcome**:
+- At least one persisted `chunk_start_date` carries `seconds=30`, proving
+  that the analyzer chunked at sub-minute granularity and the ISO-text
+  serialization preserved the boundary.
 
 ---
 
-### 2.2 Transform gate Early Exit Scenarios
+## 6. Simple Metrics Workflow Scenarios
+Source: `e2e/test_simple_metrics.py`
 
-#### Scenario 2.2.1: Transform Gate Triggers CONTINUE
+Validates `sientia_data.simple_metrics` columns:
-**Description**: Transform response gate determines data should continue despite issues
+`id, model_id, metric, value, timestamp, data_size, interval_minutes, created_at`.
+Note: ``timestamp`` is now nullable per the new DDL and ``model_id`` is ``text``.
 
-**Input**:
+### 6.1 Happy paths
-- Valid input data
-- Transform response has quality issues but policy is CONTINUE
-- `mlflow_transform_filters` configured with `POLICY: 'CONTINUE'`
-- `path_priority` includes CONTINUE
 
-**Expected Behavior**:
+#### S.1.1 rmse/mse/mae/r2 happy path
-- `request_transform` succeeds
+**Summary**: Prediction/target pairs are inserted; the activity computes all
-- `mlflow_response_gate` for transform returns `path_flag='CONTINUE'`
+four metrics with closed-form expected values.
-- `path_flag_handler` calls export workflow with transform data
-- MLFlow predict skipped
-- Transform data exported as-is
 
-**Assertions**:
+**Expected Outcome**:
-- Transform completed
+- One row per metric is persisted; all columns populated.
-- `mlflow_response_gate` called for transform
+- `data_size` matches the joined row count and `interval_minutes=60`.
-- MLFlow predict NOT called
+
-- Export workflow called with transform data
+#### S.1.2 Subset metrics
-- Workflow completes
+**Summary**: Requesting `metrics=['rmse']` writes only the rmse row.
+
+### 6.2 Edge cases
+
+#### S.2.1 Zero-variance target returns r2=0
+**Summary**: When all targets are equal, `ss_tot=0`; the activity must guard
+against division by zero and return `r2=0`.
+
+### 6.3 Failure paths
+
+#### S.3.1 No overlapping data short-circuits persistence
+**Summary**: With no `laborious_data` rows for the configured target variable
+the workflow exits before `calculate_simple_metrics` and writes nothing.
 
 ---
 
-#### Scenario 2.2.2: Transform Gate Triggers STOP
+## 7. Minimal Retrain Workflow Scenarios
-**Description**: Transform response validation fails with STOP policy
+Source: `e2e/test_minimal_retrain.py`
 
-**Input**:
+The MLflow registry is fully mocked (no real artifacts in test container).
-- Valid input data
+Validates `sientia_data.log_retrain` columns:
-- Transform response has critical errors
+`mlflow_experiment_id, mlflow_run_id, model_id, model_name, status, timestamp, version`.
-- `mlflow_transform_filters` configured with `POLICY: 'STOP'`
+Note: the new DDL drops the legacy ``id`` and ``created_at`` columns,
+``mlflow_experiment_id`` is now ``int8`` and ``model_id`` is ``text``.
 
-**Expected Behavior**:
+### 7.1 Happy path
-- `request_transform` succeeds but response invalid
-- `mlflow_response_gate` for transform returns `path_flag='STOP'`
-- Workflow exits without calling predict or export
 
-**Assertions**:
+#### MR.1.1 Successful retrain + promotion
-- Transform completed but validation failed
+**Summary**: Training data loads via MinIO offload, `wrapper.retrain` succeeds,
-- `mlflow_response_gate` called for transform
+the new version is promoted to the `production` alias.
-- MLFlow predict NOT called
+
-- Export workflow NOT called
+**Expected Outcome**:
-- Workflow completes without error
+- Report row has success status, `version='7'`, `mlflow_run_id='retrain-run-id'`,
+  `mlflow_experiment_id=4242` (`int8`).
+- `mlflow.log_artifact` is called with the input CSV.
+- `promote_to_alias` is called once with the resolved version and alias.
+
+### 7.2 Failure paths
+
+#### MR.2.1 Wrapper retrain raises
+**Summary**: `wrapper.retrain` raises `RuntimeError`. The activity returns
+`success=False`, `update_production_model` is NOT invoked.
+
+**Expected Outcome**:
+- Report row carries the error message and `version`/`mlflow_*` columns are NULL.
+
+#### MR.2.2 Missing `model_config.target`
+**Summary**: Empty model config short-circuits before any MLflow call.
+
+**Expected Outcome**:
+- Report row carries the explicit guard message.
+- `get_cached_model` is never invoked.
+
+#### MR.3.1 No training data
+**Summary**: The training query returns no rows; the workflow does not
+persist any report row. The current code raises plain `ValueError` from the
+workflow function, which Temporal treats as a workflow-task failure (see
+`CODE_ISSUES.md` issue MR-1).
 
 ---
 
-#### Scenario 2.2.3: Transform Gate Triggers REPEAT
+## Input Contract Reference
-**Description**: Transform response gate determines data should repeat last prediction
 
-**Input**:
+Common scenario input fields:
-- Valid input data
+- `schedule_name`
-- Transform response has quality issues that require using previous prediction
+- `model_name`
-- `mlflow_transform_filters` configured with `POLICY: 'REPEAT'`
+- `model_id`
-- `path_priority` includes REPEAT
+- `query`
+- `schema`
+- `table_name`
+- `transform_table_name`
+- `input_filters`
+- `mlflow_transform_filters`
+- `mlflow_predict_filters`
+- `path_priority` (default order: `STOP`, `CONTINUE`, `REPEAT`)
+- `save_transform`
+- `prediction_store_policy`
+- `model_config.target`
+- `datetime_columns` (when query returns temporal fields)
 
-**Expected Behavior**:
+Optional outputs:
-- `request_transform` succeeds but response has issues
+- `opc_output_config`
-- `mlflow_response_gate` for transform returns `path_flag='REPEAT'`
+- `pi_web_api_output_config`
-- `path_flag_handler` calls `repeat_last_prediction` activity
-- MLFlow predict skipped
-- Last prediction repeated and exported
-
-**Assertions**:
-- Transform completed but validation triggered REPEAT
-- `mlflow_response_gate` called for transform
-- MLFlow predict NOT called
-- `repeat_last_prediction` activity called
-- Workflow completes
-
----
-
-### 2.3 Predict gate Early Exit Scenarios
-
-#### Scenario 2.3.1: Predict Gate Triggers CONTINUE
-**Description**: Predict response gate determines data should continue despite issues
-
-**Input**:
-- Valid input and transform data
-- Predict response has quality issues but policy is CONTINUE
-- `mlflow_predict_filters` configured with `POLICY: 'CONTINUE'`
-- `path_priority` includes CONTINUE
-
-**Expected Behavior**:
-- `request_predict` succeeds
-- `mlflow_response_gate` for predict returns `path_flag='CONTINUE'`
-- `path_flag_handler` calls export workflow with predict data
-- Prediction exported despite quality issues
-
-**Assertions**:
-- Transform and predict completed
-- `mlflow_response_gate` called for predict
-- Export workflow called with predict data
-- Workflow completes
-
----
-
-#### Scenario 2.3.2: Predict Gate Triggers STOP
-**Description**: Prediction validation fails with STOP policy
-
-**Input**:
-- Valid input and transform
-- Predict response has critical errors
-- `mlflow_predict_filters` configured with `POLICY: 'STOP'`
-
-**Expected Behavior**:
-- `request_predict` succeeds but response invalid
-- `mlflow_response_gate` for predict returns `path_flag='STOP'`
-- Workflow exits without export
-
-**Assertions**:
-- Transform completed
-- Predict completed but validation failed
-- Export workflow NOT called
-- Workflow completes without error
-
----
-
-#### Scenario 2.3.3: Predict Gate Triggers REPEAT
-**Description**: Predict response gate determines data should repeat last prediction
-
-**Input**:
-- Valid input and transform data
-- Predict response has quality issues that require using previous prediction
-- `mlflow_predict_filters` configured with `POLICY: 'REPEAT'`
-- `path_priority` includes REPEAT
-
-**Expected Behavior**:
-- `request_predict` succeeds but response has issues
-- `mlflow_response_gate` for predict returns `path_flag='REPEAT'`
-- `path_flag_handler` calls `repeat_last_prediction` activity
-- Last prediction repeated and exported
-
-**Assertions**:
-- Transform and predict completed but validation triggered REPEAT
-- `mlflow_response_gate` called for predict
-- `repeat_last_prediction` activity called
-- Export workflow NOT called with current prediction
-- Workflow completes
-
----
-
-## 3. Format and Export Prediction - Child Workflow Scenarios
-
-### 3.1 Success Scenarios
-
-#### Scenario 3.1.1: Default Prediction Export
-**Description**: Error prediction path creates default prediction
-
-**Input**:
-- `path_flag: 'ERROR'` or other non-None value (not STOP/CONTINUE/REPEAT)
-- `comment` provided with error details
-
-**Expected Behavior**:
-- `format_default_prediction` called instead of `format_prediction`
-- Default prediction created with error metadata
-- Exported to PostgreSQL only
-- Transformed data NOT processed
-- Metrics written
-
-**Assertions**:
-- `format_default_prediction` called
-- `format_prediction` NOT called
-- `format_transformed_data` NOT called
-- One PostgreSQL export only
-- Default values in prediction data
-- Comment included
-
----
-
-#### Scenario 3.1.2: Export with OPC only
-**Description**: Export to PostgreSQL and OPC server only (no PI Web API)
-
-**Input**:
-- `path_flag: None`
-- `opc_output_config` configured with valid OPC settings
-- `pi_web_api_output_config: None` or `{}`
-
-**Expected Behavior**:
-- Normal formatting
-- PostgreSQL export executed
-- OPC export executed
-- PI Web API activity skipped
-- Metrics written with OPC metrics
-
-**Assertions**:
-- PI Web API activity NOT called
-- OPC activity called
-- PostgreSQL export called
-- Metrics written with `opc_metrics` populated
-
----
-
-#### Scenario 3.1.3: Export with PI Web API only
-**Description**: Export to PostgreSQL and PI Web API only (no OPC)
-
-**Input**:
-- `path_flag: None`
-- `pi_web_api_output_config` configured with valid PI Web API settings
-- `opc_output_config: None` or `{}`
-
-**Expected Behavior**:
-- Normal formatting
-- PostgreSQL export executed
-- PI Web API export executed
-- OPC activity skipped
-- Metrics written without OPC metrics
-
-**Assertions**:
-- OPC activity NOT called
-- PI Web API activity called
-- PostgreSQL export called
-- Metrics written with empty `opc_metrics`
-
----
-
-#### Scenario 3.1.4: Export Without Optional Outputs
-**Description**: Export only to PostgreSQL (no OPC or PI Web API)
-
-**Input**:
-- `path_flag: None`
-- `opc_output_config: None` or `{}`
-- `pi_web_api_output_config: None` or `{}`
-
-**Expected Behavior**:
-- Normal formatting
-- Only PostgreSQL export executed
-- OPC and PI Web API activities skipped
-- Metrics written without OPC metrics
-
-**Assertions**:
-- PI Web API activity NOT called
-- OPC activity NOT called
-- PostgreSQL export called
-- Metrics written with empty `opc_metrics`
-
----
-
-#### Scenario 3.1.5: Export Without Transformed Data
-**Description**: Only prediction exported, no transform table
-
-**Input**:
-- `path_flag: None`
-- `transformed_data: None` or `save_transform: False`
-- `opc_output_config: None` or `{}`
-- `pi_web_api_output_config: None` or `{}`
-
-**Expected Behavior**:
-- Only prediction formatted and exported
-- Transform export skipped
-- Single PostgreSQL write
-
-**Assertions**:
-- `format_transformed_data` NOT called
-- One PostgreSQL export
-- Transform table remains empty
-
----
-
-### 3.2 Error Scenarios
-
-These paths do **not** rely on Temporal activity retries for export failures: the write activities run once, errors are handled inside the activity, and the **workflow completes successfully** with degraded metadata on the persisted prediction (`prediction_confidence` and `comments`).
-
-#### Scenario 3.2.1: PI Web API Write Error
-**Description**: PI Web API export fails
-
-**Input**:
-- Valid prediction
-- PI Web API service unavailable or invalid config
-
-**Expected Behavior**:
-- `write_pi_web_api_data` surfaces the failure (exception handled in the activity layer)
-- Notification may be sent
-- Workflow **completes** (does not fail)
-- Prediction row is still written to PostgreSQL with error confidence **13** and a comment describing the PI error
-- Subsequent steps (e.g. OPC, Postgres) still run per workflow order with the updated prediction payload
-
-**Assertions**:
-- PI Web API error notification sent (when applicable)
-- Workflow completes
-- PostgreSQL contains the prediction with `prediction_confidence` 13 and expected `comments`
-
----
-
-#### Scenario 3.2.2: OPC Write Error
-**Description**: OPC server write fails
-
-**Input**:
-- Valid prediction
-- OPC server unavailable or invalid configuration
-
-**Expected Behavior**:
-- `write_opc_data` reports failure without aborting the workflow
-- Notification may be sent
-- Workflow **completes** (does not fail)
-- Prediction row is written to PostgreSQL with OPC error confidence **12** and a comment indicating OPC write issues
-
-**Assertions**:
-- OPC error notification sent (when applicable)
-- Workflow completes
-- PostgreSQL contains the prediction with `prediction_confidence` 12 and expected `comments`
-
----
-
-#### Scenario 3.2.3: PI Web API Partial Write Error
-**Description**: Two prediction tags attempt to be written to PI Web API, but only one succeeds
-
-**Input**:
-- Valid prediction
-- Two prediction tags configured
-- PI Web API returns partial success (one tag succeeds, one fails)
-
-**Expected Behavior**:
-- `write_pi_web_api_data` processes response
-- `process_pi_web_api_response` detects partial failure
-- Error confidence set (13)
-- Notification sent for failed tag
-- Workflow completes with error confidence (single activity attempt; no retry loop)
-
-**Assertions**:
-- One tag written successfully
-- One tag failed
-- Error confidence set in prediction
-- Error notification sent
-- Workflow completes
-
----

e2e/test_child_workflows_e2e.py

@@ -26,7 +26,7 @@ async def test_format_and_export_prediction_default_path_e2e(
     client = temporal_test_env.client
     model_id = 401
     with postgres_engine.begin() as conn:
-        conn.execute(text(f'DELETE FROM predictions_schema.predictions WHERE model_id = {model_id}'))
+        conn.execute(text(f'DELETE FROM sientia_data.predictions WHERE model_id = {model_id}'))
 
     metadata = {
         'metadata': {
@@ -44,7 +44,7 @@ async def test_format_and_export_prediction_default_path_e2e(
         'timestamp': '2024-01-01 12:00:00+00:00',
         'model_id': model_id,
         'model_name': 'test_model',
-        'schema': 'predictions_schema',
+        'schema': 'sientia_data',
         'table_name': 'predictions',
         'transform_table_name': 'transformed_data',
         'comment': 'e2e child workflow default path',
@@ -64,7 +64,7 @@ async def test_format_and_export_prediction_default_path_e2e(
         row = conn.execute(
             text(
                 f'SELECT prediction, prediction_confidence, prediction_status, comments '
-                f'FROM predictions_schema.predictions WHERE model_id = {model_id}'
+                f'FROM sientia_data.predictions WHERE model_id = {model_id}'
             )
         ).fetchone()
     assert row is not None

e2e/test_drift.py

@@ -0,0 +1,600 @@
+"""
+End-to-end tests for the Drift workflow.
+
+The drift suite drives the **real** ``sientia_model.analytics.drift_analysis.DriftAnalysis``
+analyzer (no mocking). Each scenario exercises the full pipeline:
+
+    laborious_data (Postgres)
+        -> load_custom_query
+        -> calculate_drift (DriftAnalysis univariate + multivariate)
+        -> export_data_to_postgres (sientia_data.drift_metrics)
+
+Coverage focus:
+
+- Happy path persists every column required by ``sientia_data.drift_metrics``
+  with a valid reference dataset downloaded from MLflow.
+- 30% fallback path activates when the MLflow reference is unavailable and
+  emits the ``MODEL_METRICS_REFERENCE_DATA_WARNING`` notification.
+- Empty target data short-circuits the workflow without persisting anything.
+- Invalid ``chunk_period`` is rejected by ``calculate_drift``.
+- ``chunk_period='s'`` preserves second-level precision in
+  ``chunk_start_date``.
+
+Tests assert behavioral / structural properties (column presence, NOT NULL
+constraints, business-key invariants) rather than exact numeric values, since
+those depend on the real analyzer implementation and synthetic data.
+"""
+
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+import pandas as pd
+import pytest
+from sqlalchemy import text
+from temporalio.testing import WorkflowEnvironment
+from temporalio.worker import Worker
+
+from e2e.helpers import (
+    insert_target_data_for_drift,
+    load_scenario_input,
+    make_workflow_id,
+    start_and_await_workflow,
+)
+from laborious.activities.activities import Activities
+from laborious.activities.model_metrics import ModelMetrics
+from laborious.workflows.drift import Drift
+from sientia_model.analytics.drift_analysis import DriftAnalysis
+
+# Drift columns persisted on every row in ``sientia_data.drift_metrics`` —
+# mirrors the production DDL.
+EXPECTED_DRIFT_COLUMNS = [
+    'id',
+    'model_id',
+    'feature',
+    'method',
+    'value',
+    'alert',
+    'chunk_index',
+    'chunk_start_date',
+    'chunk_end_date',
+    'accurate',
+    'timestamp',
+    'created_at',
+]
+
+# Columns the DDL marks as NOT NULL. ``feature`` and ``timestamp`` are
+# nullable in the production schema (multivariate rows do not bind to a
+# single feature; ``timestamp`` is allowed to be empty when upstream data has
+# no usable instant).
+NON_NULL_DRIFT_COLUMNS = {
+    'id',
+    'model_id',
+    'method',
+    'value',
+    'alert',
+    'chunk_index',
+    'chunk_start_date',
+    'chunk_end_date',
+    'accurate',
+    'created_at',
+}
+
+DEFAULT_DRIFT_METHODS = ['kolmogorov_smirnov', 'jensen_shannon', 'wasserstein']
+
+
+def _chunk_dataframe_skip_empty_groups(
+    self: DriftAnalysis,
+    df: pd.DataFrame,
+    timestamp_col: str,
+    chunk_period: str,
+) -> list[tuple[int, pd.DataFrame]]:
+    """
+    Same as ``DriftAnalysis._chunk_dataframe`` but omit empty time buckets.
+
+    ``pd.Grouper(freq='s')`` yields every second between min and max timestamp;
+    empty buckets still appear in the groupby iterator and produce invalid
+    drift rows (e.g. NaT timestamps) that ``calculate_drift`` later filters out
+    entirely. Production fix belongs in ``sientia_model``; this shim keeps the
+    e2e honest about second-level chunk boundaries with sparse samples.
+    """
+    grouped = df.groupby(pd.Grouper(key=timestamp_col, freq=chunk_period), dropna=True)
+    chunks: list[tuple[int, pd.DataFrame]] = []
+    idx = 0
+    for _, chunk in grouped:
+        if chunk.empty:
+            continue
+        chunks.append((idx, chunk.copy()))
+        idx += 1
+    return chunks
+
+
+def _drift_input(model_id: int, **overrides) -> dict:
+    """Load the base drift scenario JSON and apply ad-hoc overrides."""
+    input_data = load_scenario_input('drift_base.json', model_id=model_id)
+    input_data.update(overrides)
+    return input_data
+
+
+def _recent_minute_timestamps(count: int, offset_minutes: int = 6) -> list[str]:
+    """
+    Build ``count`` consecutive UTC minute timestamps placed in the recent past.
+
+    The Drift workflow filters target rows with ``timestamp > NOW() - INTERVAL``,
+    so timestamps must be recent for tests to retrieve any data. Snapping to
+    minute precision keeps the helper deterministic regardless of clock skew.
+
+    Args:
+     - count (int): How many consecutive minute timestamps to generate.
+     - offset_minutes (int): Minutes ago for the EARLIEST generated timestamp.
+
+    Return:
+     list[str]: ISO strings with ``+0000`` offset, one per minute.
+    """
+    base = datetime.now(timezone.utc).replace(second=0, microsecond=0) - timedelta(
+        minutes=offset_minutes
+    )
+    return [
+        (base + timedelta(minutes=i)).strftime('%Y-%m-%d %H:%M:%S%z') for i in range(count)
+    ]
+
+
+def _configure_reference_csv(mlflow_repository_stub, reference_rows: pd.DataFrame) -> None:
+    """
+    Wire ``mlflow_repository_stub`` so ``get_reference_data`` returns
+    ``reference_rows`` by writing them to ``dst_path/retrain_input.csv``.
+
+    Args:
+     - mlflow_repository_stub: External MLflow repository fixture.
+     - reference_rows (pd.DataFrame): Rows to expose as the production reference.
+    """
+
+    def _download(run_id: str, artifact_path: str, dst_path: str, metadata=None):
+        target = Path(dst_path) / artifact_path
+        target.parent.mkdir(parents=True, exist_ok=True)
+        reference_rows.to_csv(target, index=False)
+
+    mlflow_repository_stub._client.get_model_version_by_alias.return_value = MagicMock(
+        run_id='fake-reference-run'
+    )
+    file_info = MagicMock()
+    file_info.path = 'retrain_input.csv'
+    mlflow_repository_stub._client.list_artifacts.return_value = [file_info]
+    mlflow_repository_stub.download_artifacts.side_effect = _download
+
+
+def _force_reference_unavailable(mlflow_repository_stub) -> None:
+    """Make ``get_reference_data`` return ``None`` by failing alias resolution."""
+    mlflow_repository_stub._client.get_model_version_by_alias.side_effect = Exception(
+        'no production alias registered'
+    )
+
+
+def _select_drift_rows(postgres_engine, model_id: int) -> list[dict]:
+    """Read every persisted drift row for ``model_id`` ordered by chunk/feature/method."""
+    with postgres_engine.connect() as conn:
+        rows = (
+            conn.execute(
+                text(
+                    'SELECT * FROM sientia_data.drift_metrics '
+                    'WHERE model_id = :m '
+                    'ORDER BY chunk_index, feature, method'
+                ),
+                {'m': str(model_id)},
+            )
+            .mappings()
+            .all()
+        )
+    return [dict(row) for row in rows]
+
+
+def _assert_required_columns_populated(rows: list[dict]) -> None:
+    """Validate column presence and NOT NULL constraints on every row."""
+    assert rows, 'expected at least one drift row to be persisted'
+    seen_columns = set(rows[0].keys())
+    for column in EXPECTED_DRIFT_COLUMNS:
+        assert column in seen_columns, f'Missing drift column in postgres: {column}'
+    for row in rows:
+        for column in NON_NULL_DRIFT_COLUMNS:
+            assert row[column] is not None, f"Column '{column}' is NULL in {row}"
+        assert 'p_value' not in row, 'p_value must not be persisted to drift_metrics'
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_drift_happy_path_persists_all_columns_with_reference_data(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_drift: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    mlflow_repository_stub,
+):
+    """
+    Scenario D.1.1: Happy path with reference data downloaded from MLflow.
+
+    Drives the full pipeline against the real ``DriftAnalysis``. Asserts:
+
+    - One row is persisted per ``(chunk_index, feature, method)`` combination
+      plus the multivariate row block, with every column required by
+      ``sientia_data.drift_metrics`` populated.
+    - The three default univariate methods are forwarded to the analyzer.
+    - ``model_id`` and ``timestamp`` are stamped by the activity (not by the
+      analyzer); ``timestamp`` equals ``max(target_data.timestamp)`` and is
+      identical on every persisted row.
+    - ``chunk_start_date`` / ``chunk_end_date`` are persisted as ISO text so
+      the analyzer's nanosecond-precision boundaries survive the ``text``
+      column type.
+    - ``accurate=True`` because the reference dataset was available.
+    """
+    client = temporal_test_env.client
+    model_id = 411
+
+    target_timestamps = _recent_minute_timestamps(count=10)
+    insert_target_data_for_drift(
+        postgres_engine,
+        model_id=model_id,
+        timestamps=target_timestamps,
+        variables_values={
+            'sensor_1': [10.0 + i * 0.1 for i in range(10)],
+            'sensor_2': [20.0 + i * 0.5 for i in range(10)],
+        },
+    )
+
+    reference_df = pd.DataFrame(
+        {
+            'timestamp': [
+                f'2023-12-31 11:{minute:02d}:00+00:00' for minute in range(10)
+            ],
+            'sensor_1': [9.0 + i * 0.05 for i in range(10)],
+            'sensor_2': [18.0 + i * 0.25 for i in range(10)],
+        }
+    )
+    _configure_reference_csv(mlflow_repository_stub, reference_df)
+
+    input_data = _drift_input(model_id)
+    await start_and_await_workflow(
+        client, Drift.run, input_data, make_workflow_id('test-drift-happy-path')
+    )
+
+    rows = _select_drift_rows(postgres_engine, model_id)
+
+    exported_csv_path = '/tmp/test_drift_happy_path_exported.csv'
+    pd.DataFrame(rows).to_csv(exported_csv_path, index=False)
+    print(
+        f'\n[test_drift_happy_path] Exported drift dataframe '
+        f'({len(rows)} rows) -> {exported_csv_path}'
+    )
+
+    _assert_required_columns_populated(rows)
+
+    # The activity drops the target column from the feature list, so only
+    # ``sensor_2`` participates in univariate analysis (``sensor_1`` is the
+    # configured target). Multivariate produces one row per chunk regardless.
+    univariate_rows = [r for r in rows if r['feature'] != 'multivariate']
+    multivariate_rows = [r for r in rows if r['feature'] == 'multivariate']
+    assert univariate_rows, 'expected univariate drift rows for non-target features'
+    assert multivariate_rows, 'expected one multivariate drift row per chunk'
+
+    # All three default methods must reach the analyzer.
+    assert {r['method'] for r in univariate_rows} == set(DEFAULT_DRIFT_METHODS)
+    assert all(r['method'] == 'multivariate' for r in multivariate_rows)
+    assert {r['feature'] for r in univariate_rows} == {'sensor_2'}
+
+    # ``timestamp`` is stamped uniformly with ``max(target_data.timestamp)``.
+    expected_timestamp = pd.to_datetime(max(target_timestamps), utc=True)
+    persisted_timestamps = {pd.to_datetime(r['timestamp'], utc=True) for r in rows}
+    assert len(persisted_timestamps) == 1, (
+        'timestamp must be uniform across all drift rows '
+        f'(got {len(persisted_timestamps)} distinct values)'
+    )
+    assert pd.Timestamp(persisted_timestamps.pop()) == expected_timestamp, (
+        'timestamp must equal max(target_data.timestamp)'
+    )
+
+    # ``model_id`` is stamped by ``calculate_drift`` (not produced by the analyzer).
+    assert all(r['model_id'] == str(model_id) for r in rows), (
+        'model_id must be stamped on every drift row'
+    )
+
+    # Reference path → accurate=True.
+    assert all(r['accurate'] is True for r in rows), (
+        'reference path should mark all rows as accurate'
+    )
+
+    # ISO text serialization preserves ordering between start/end of each chunk.
+    for row in rows:
+        assert 'T' in row['chunk_start_date'], (
+            f"chunk_start_date should be ISO text, got {row['chunk_start_date']!r}"
+        )
+        assert 'T' in row['chunk_end_date'], (
+            f"chunk_end_date should be ISO text, got {row['chunk_end_date']!r}"
+        )
+        assert row['chunk_start_date'] <= row['chunk_end_date'], (
+            f'chunk_start_date must precede chunk_end_date '
+            f"(start={row['chunk_start_date']}, end={row['chunk_end_date']})"
+        )
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_drift_uses_30pct_fallback_when_reference_unavailable(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_drift: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    mlflow_repository_stub,
+    notification_inserts,
+):
+    """
+    Scenario D.1.2: ``get_reference_data`` returns ``None`` (production alias
+    missing), so ``calculate_drift`` falls back to using the first 30% of
+    target rows as reference. Persisted rows must report ``accurate=False``
+    and a ``MODEL_METRICS_REFERENCE_DATA_WARNING`` notification must be
+    emitted to mongo.
+    """
+    client = temporal_test_env.client
+    model_id = 412
+
+    target_timestamps = _recent_minute_timestamps(count=10)
+    insert_target_data_for_drift(
+        postgres_engine,
+        model_id=model_id,
+        timestamps=target_timestamps,
+        variables_values={
+            'sensor_1': [10.0 + i * 0.1 for i in range(10)],
+            'sensor_2': [20.0 + i * 0.5 for i in range(10)],
+        },
+    )
+
+    _force_reference_unavailable(mlflow_repository_stub)
+
+    input_data = _drift_input(model_id)
+    await start_and_await_workflow(
+        client, Drift.run, input_data, make_workflow_id('test-drift-fallback')
+    )
+
+    rows = _select_drift_rows(postgres_engine, model_id)
+    _assert_required_columns_populated(rows)
+
+    assert all(r['accurate'] is False for r in rows), (
+        'fallback path must mark all rows as inaccurate'
+    )
+
+    fallback_warnings = [
+        call
+        for call in notification_inserts.call_args_list
+        if call.args
+        and isinstance(call.args[0], dict)
+        and call.args[0].get('notification_id') == 'MODEL_METRICS_REFERENCE_DATA_WARNING'
+    ]
+    assert len(fallback_warnings) >= 1, 'expected reference fallback warning notification'
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_drift_empty_target_data_short_circuits_workflow(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_drift: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    mlflow_repository_stub,
+):
+    """
+    Scenario D.3.1: When ``load_custom_query`` returns no rows the workflow
+    must return early without invoking the analyzer or writing any drift rows.
+    """
+    client = temporal_test_env.client
+    model_id = 431
+
+    with postgres_engine.begin() as conn:
+        conn.execute(text(f'DELETE FROM sientia_data.laborious_data WHERE model_id = {model_id}'))
+
+    _force_reference_unavailable(mlflow_repository_stub)
+
+    input_data = _drift_input(model_id)
+    await start_and_await_workflow(
+        client, Drift.run, input_data, make_workflow_id('test-drift-empty-target')
+    )
+
+    with postgres_engine.connect() as conn:
+        count = conn.execute(
+            text('SELECT COUNT(*) FROM sientia_data.drift_metrics WHERE model_id = :m'),
+            {'m': str(model_id)},
+        ).scalar()
+    assert count == 0
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_drift_invalid_chunk_period_raises_value_error(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_drift: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    mlflow_repository_stub,
+):
+    """
+    Scenario D.4.2: ``calculate_drift`` validates ``chunk_period`` and rejects
+    anything other than ``min`` / ``s``. The workflow must surface the
+    ``ValueError`` and persist nothing.
+    """
+    client = temporal_test_env.client
+    model_id = 442
+
+    target_timestamps = _recent_minute_timestamps(count=5)
+    insert_target_data_for_drift(
+        postgres_engine,
+        model_id=model_id,
+        timestamps=target_timestamps,
+        variables_values={
+            'sensor_1': [1.0, 2.0, 3.0, 4.0, 5.0],
+            'sensor_2': [10.0, 20.0, 30.0, 40.0, 50.0],
+        },
+    )
+    _force_reference_unavailable(mlflow_repository_stub)
+
+    input_data = _drift_input(model_id, chunk_period='hour')
+
+    with pytest.raises(Exception) as excinfo:
+        await start_and_await_workflow(
+            client,
+            Drift.run,
+            input_data,
+            make_workflow_id('test-drift-bad-chunk-period'),
+        )
+
+    # Temporal wraps the activity ValueError in WorkflowFailureError; the
+    # message may live on ``.message`` or ``str(exc)`` depending on the SDK
+    # error class, so walk the cause chain looking for the guard text.
+    cause_descriptions = []
+    current: BaseException | None = excinfo.value
+    while current is not None:
+        cause_descriptions.append(
+            getattr(current, 'message', None) or str(current) or repr(current)
+        )
+        current = current.__cause__
+    assert any('Invalid chunk period' in msg for msg in cause_descriptions), (
+        f'Expected ValueError about chunk period in chain, got: {cause_descriptions}'
+    )
+
+    with postgres_engine.connect() as conn:
+        count = conn.execute(
+            text('SELECT COUNT(*) FROM sientia_data.drift_metrics WHERE model_id = :m'),
+            {'m': str(model_id)},
+        ).scalar()
+    assert count == 0
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_drift_empty_merge_skips_export_without_insufficient_notification(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_drift: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    mlflow_repository_stub,
+):
+    """
+    Scenario D.4.3a: When the analyzer returns an empty merged frame (no metric rows),
+    ``calculate_drift`` yields ``[]``; the workflow skips export. Real insufficient-data
+    cases are signaled by ``DriftInsufficientDataError`` inside ``sientia_model``, not by
+    empty output alone.
+    """
+    client = temporal_test_env.client
+    model_id = 444
+
+    target_timestamps = _recent_minute_timestamps(count=5)
+    insert_target_data_for_drift(
+        postgres_engine,
+        model_id=model_id,
+        timestamps=target_timestamps,
+        variables_values={
+            'sensor_1': [1.0 + i * 0.1 for i in range(5)],
+            'sensor_2': [10.0 + i * 0.5 for i in range(5)],
+        },
+    )
+    _force_reference_unavailable(mlflow_repository_stub)
+
+    input_data = _drift_input(model_id, chunk_period='min')
+    empty_merge = pd.DataFrame(
+        columns=[
+            'timestamp',
+            'feature',
+            'method',
+            'value',
+            'alert',
+            'chunk_index',
+            'chunk_start_date',
+            'chunk_end_date',
+            'threshold',
+            'drift_type',
+        ]
+    )
+    with patch.object(ModelMetrics, 'get_drift_metrics', return_value=empty_merge):
+        await start_and_await_workflow(
+            client,
+            Drift.run,
+            input_data,
+            make_workflow_id('test-drift-empty-merge'),
+        )
+
+    with postgres_engine.connect() as conn:
+        count = conn.execute(
+            text('SELECT COUNT(*) FROM sientia_data.drift_metrics WHERE model_id = :m'),
+            {'m': str(model_id)},
+        ).scalar()
+    assert count == 0
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_drift_chunk_period_seconds_sufficient_data_preserves_seconds_in_chunk_start_date(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_drift: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    mlflow_repository_stub,
+):
+    """
+    Scenario D.4.3b: With enough sub-minute samples and ``chunk_period='s'``, drift rows
+    persist and ``chunk_start_date`` keeps second-level precision (incl. second=30).
+
+    ``DriftAnalysis._chunk_dataframe`` is patched to skip empty ``pd.Grouper(freq='s')``
+    buckets so sparse seconds between samples do not flood the pipeline with NaT rows;
+    the durable fix belongs in ``sientia_model``.
+    """
+    client = temporal_test_env.client
+    model_id = 443
+
+    base = datetime.now(timezone.utc).replace(second=0, microsecond=0) - timedelta(minutes=6)
+    target_timestamps = []
+    sensor_1_vals = []
+    sensor_2_vals = []
+    for minute_offset in range(6):
+        t0 = base + timedelta(minutes=minute_offset)
+        t1 = t0 + timedelta(seconds=30)
+        target_timestamps.append(t0.strftime('%Y-%m-%d %H:%M:%S%z'))
+        target_timestamps.append(t1.strftime('%Y-%m-%d %H:%M:%S%z'))
+        v0 = 1.0 + minute_offset * 0.1
+        v1 = v0 + 0.05
+        sensor_1_vals.extend([v0, v1])
+        sensor_2_vals.extend([10.0 + v0, 10.0 + v1])
+
+    insert_target_data_for_drift(
+        postgres_engine,
+        model_id=model_id,
+        timestamps=target_timestamps,
+        variables_values={
+            'sensor_1': sensor_1_vals,
+            'sensor_2': sensor_2_vals,
+        },
+    )
+    _force_reference_unavailable(mlflow_repository_stub)
+
+    input_data = _drift_input(model_id, chunk_period='s')
+    with patch.object(DriftAnalysis, '_chunk_dataframe', _chunk_dataframe_skip_empty_groups):
+        await start_and_await_workflow(
+            client,
+            Drift.run,
+            input_data,
+            make_workflow_id('test-drift-chunk-seconds-sufficient'),
+        )
+
+    with postgres_engine.connect() as conn:
+        rows = (
+            conn.execute(
+                text(
+                    'SELECT chunk_start_date FROM sientia_data.drift_metrics '
+                    'WHERE model_id = :m ORDER BY chunk_index'
+                ),
+                {'m': str(model_id)},
+            )
+            .mappings()
+            .all()
+        )
+
+    assert rows, 'expected at least one drift row to be persisted'
+    seconds_present = {pd.Timestamp(r['chunk_start_date']).second for r in rows}
+    assert 30 in seconds_present, (
+        f'expected at least one chunk_start_date with seconds=30, got {seconds_present}'
+    )

e2e/test_minimal_retrain.py

@@ -0,0 +1,430 @@
+"""
+End-to-end tests for the MinimalRetrain workflow.
+
+The MLflow registry is fully stubbed because no real artifacts exist in a
+test container; we only validate that the workflow:
+
+- Loads training data via ``load_query_with_minio_offload``.
+- Calls ``retrain_model`` with a payload pointing at MinIO.
+- Calls ``update_production_model`` only when retrain succeeds.
+- Persists ``sientia_data.log_retrain`` rows with all required columns;
+  success rows carry the new ``version`` / ``mlflow_run_id`` /
+  ``mlflow_experiment_id`` while failure rows leave them ``NULL``.
+
+The production DDL drops the legacy ``id`` / ``created_at`` columns and
+moves ``mlflow_experiment_id`` to ``int8`` and ``model_id`` to ``text``.
+The stubs used here therefore emit ``experiment_id`` as an integer to fit
+the new column type.
+"""
+
+from contextlib import contextmanager
+from datetime import datetime, timedelta, timezone
+from unittest.mock import MagicMock, patch
+
+import pytest
+import pandas as pd
+from sqlalchemy import text
+from temporalio.testing import WorkflowEnvironment
+from temporalio.worker import Worker
+
+from e2e.helpers import (
+    insert_target_data_for_drift,
+    load_scenario_input,
+    make_workflow_id,
+    start_and_await_workflow,
+)
+from laborious.activities.activities import Activities
+from laborious.workflows.minimal_retrain import MinimalRetrain
+from sientia_model.wrappers.sientia_model import SientiaModel
+
+# Columns defined by the production DDL for ``sientia_data.log_retrain``.
+# The legacy ``retrain_reports`` table had ``id`` and ``created_at``; the new
+# DDL drops both. ``mlflow_experiment_id`` is ``int8`` and ``model_id`` is
+# ``text``.
+EXPECTED_RETRAIN_REPORT_COLUMNS = [
+    'mlflow_experiment_id',
+    'mlflow_run_id',
+    'model_id',
+    'model_name',
+    'status',
+    'timestamp',
+    'version',
+]
+
+# Matches ``retrain_model`` return ``message`` when ``success`` is True (also written to ``log_retrain.status``).
+RETRAIN_ACTIVITY_SUCCESS_MESSAGE = 'Model retrained successfully.'
+
+
+class _FakeSientiaModelForMinimalRetrain(SientiaModel):
+    """
+    Fake SientiaModel that uses the real SientiaModel lifecycle to surface
+    index-alignment issues during ``retrain()``.
+
+    It intentionally performs strict alignment inside ``_retrain_model``:
+    ``y.loc[x.index]``.
+    """
+
+    def __init__(self, *, target: str = 'sensor_1'):
+        super().__init__(
+            model_type='FakeMinimalRetrain',
+            model_version='0.0.0',
+            model=object(),
+            transformer=object(),
+        )
+        self.target = target
+        self.model_is_fitted = True
+        self.force_retrain_error = False
+
+    def store_model(  # type: ignore[override]
+        self,
+        name: str,
+        signature=None,
+        pip_requirements=None,
+        code_path=None,
+    ) -> None:
+        # No-op: E2E tests validate workflow persistence, not real MLflow artifacts.
+        return None
+
+    def _predict(self, data: pd.DataFrame):
+        pred = pd.DataFrame({'prediction': [0.5] * len(data)}, index=data.index)
+        return pred, {}
+
+    def _transform(self, data: pd.DataFrame):
+        out = data.drop(columns=[self.target], errors='ignore').copy()
+        out.index = data.index
+        return out, {}
+
+    def _train_transformer(self, train_data: pd.DataFrame, val_data: pd.DataFrame) -> None:
+        return None
+
+    def _train_model(
+        self,
+        x: pd.DataFrame,
+        y: pd.DataFrame,
+        x_val: pd.DataFrame | None = None,
+        y_val: pd.DataFrame | None = None,
+    ) -> None:
+        return None
+
+    def _retrain_transformer(self, data: pd.DataFrame) -> None:
+        return None
+
+    def _retrain_model(self, x: pd.DataFrame, y: pd.DataFrame | None) -> None:
+        if self.force_retrain_error:
+            raise RuntimeError('training did not converge')
+        if y is None:
+            return
+        # Strict alignment on purpose to reproduce the production failure mode.
+        _ = y.loc[x.index]
+
+
+@pytest.fixture
+def mlflow_repository_stub():
+    """
+    Override the shared E2E fixture: return a real fake ``SientiaModel`` wrapper
+    instead of a MagicMock wrapper.
+    """
+    repo = MagicMock()
+    repo._client = MagicMock()
+
+    wrapper = _FakeSientiaModelForMinimalRetrain(target='sensor_1')
+    repo.get_cached_model = MagicMock(return_value=wrapper)
+    return repo
+
+
+def _retrain_input(model_id: int, **overrides) -> dict:
+    """Load and override the minimal-retrain base scenario."""
+    payload = load_scenario_input('minimal_retrain_base.json', model_id=model_id)
+    payload.update(overrides)
+    return payload
+
+
+def _seed_retrain_training_rows(postgres_engine, model_id: int) -> None:
+    """
+    Insert training rows in long format that pivot cleanly into
+    ``index=timestamp`` / ``columns={sensor_1, sensor_2}`` for ``retrain_model``.
+    """
+    target_timestamps = [
+        (datetime.now(timezone.utc).replace(second=0, microsecond=0) - timedelta(minutes=10 - i))
+        .strftime('%Y-%m-%d %H:%M:%S%z')
+        for i in range(5)
+    ]
+    insert_target_data_for_drift(
+        postgres_engine,
+        model_id=model_id,
+        timestamps=target_timestamps,
+        variables_values={
+            'sensor_1': [10.0, 11.0, 12.0, 13.0, 14.0],
+            'sensor_2': [20.0, 21.0, 22.0, 23.0, 24.0],
+        },
+    )
+
+
+def _configure_retrain_happy_path(mlflow_repository_stub) -> None:
+    """
+    Wire ``mlflow_repository_stub`` so retrain + update_production succeed.
+
+    Mocks (in order of consumption):
+
+    - ``_client.get_model_version_by_alias``: returns ``mv`` with a stable
+      ``run_id`` (used as ``source_run_id``).
+    - ``start_run``: returns a context manager yielding a ``run_info`` with
+      run/experiment ids.
+    - ``log_params``: inert.
+    - ``_client.search_model_versions``: returns one registry entry whose
+      ``version`` is promoted by ``update_production_model``.
+    - ``promote_to_alias``: inert success.
+    """
+    mv_src = MagicMock()
+    mv_src.run_id = 'source-run-id'
+
+    new_version = MagicMock()
+    new_version.version = '7'
+    new_version.run_id = 'retrain-run-id'
+
+    mlflow_repository_stub._client.get_model_version_by_alias.return_value = mv_src
+
+    @contextmanager
+    def fake_start_run(**kwargs):
+        run_info = MagicMock()
+        run_info.run_id = 'retrain-run-id'
+        # ``mlflow_experiment_id`` is ``int8`` in the new DDL, so we feed an
+        # integer-compatible id from the stubbed run info.
+        run_info.experiment_id = 4242
+        yield run_info
+
+    mlflow_repository_stub.start_run.side_effect = fake_start_run
+    mlflow_repository_stub.log_params = MagicMock(return_value=None)
+    mlflow_repository_stub._client.search_model_versions.return_value = [new_version]
+    mlflow_repository_stub.promote_to_alias = MagicMock(return_value=None)
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_minimal_retrain_happy_path_writes_success_report(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_minimal_retrain: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    mlflow_repository_stub,
+):
+    """
+    Scenario MR.1.1: Retrain succeeds. ``sientia_data.log_retrain`` must
+    contain a success row with version/mlflow_run_id/mlflow_experiment_id
+    populated and the registry must have been told to promote the new version
+    to the configured alias.
+    """
+    client = temporal_test_env.client
+    model_id = 711
+
+    _seed_retrain_training_rows(postgres_engine, model_id)
+    _configure_retrain_happy_path(mlflow_repository_stub)
+
+    input_data = _retrain_input(model_id)
+
+    with patch('laborious.activities.mlflow.mlflow.log_artifact') as log_artifact_mock:
+        await start_and_await_workflow(
+            client,
+            MinimalRetrain.run,
+            input_data,
+            make_workflow_id('test-retrain-happy'),
+        )
+
+    with postgres_engine.connect() as conn:
+        rows = (
+            conn.execute(
+                text(
+                    'SELECT * FROM sientia_data.log_retrain '
+                    'WHERE model_id = :m'
+                ),
+                {'m': str(model_id)},
+            )
+            .mappings()
+            .all()
+        )
+
+    assert len(rows) == 1
+    row = rows[0]
+    for column in EXPECTED_RETRAIN_REPORT_COLUMNS:
+        assert column in row, f'Missing log_retrain column: {column}'
+
+    assert row['status'] == RETRAIN_ACTIVITY_SUCCESS_MESSAGE, (
+        "Expected retrain_model to return success (experiment_response['success'] is True). "
+        'Persisted log_retrain.status is the activity message; when success is False the run '
+        'never reaches mlflow.log_artifact — diagnose the retrain failure from status below, '
+        'not from a skipped artifact upload. '
+        f"Got status={row['status']!r}, version={row.get('version')!r}, "
+        f"mlflow_run_id={row.get('mlflow_run_id')!r}."
+    )
+
+    assert log_artifact_mock.called, (
+        'After a successful retrain, retrain_model must call mlflow.log_artifact for the '
+        'input CSV inside start_run.'
+    )
+
+    # ``model_id`` is now ``text``; compare against the stringified id.
+    assert row['model_id'] == str(model_id)
+    assert row['model_name'] == 'test_model'
+    assert row['version'] == '7'
+    assert row['mlflow_run_id'] == 'retrain-run-id'
+    # ``mlflow_experiment_id`` is now ``int8``; assert the integer value
+    # provided by the stubbed run info.
+    assert row['mlflow_experiment_id'] == 4242
+    assert row['timestamp'] is not None
+
+    mlflow_repository_stub.promote_to_alias.assert_called_once()
+    promote_kwargs = mlflow_repository_stub.promote_to_alias.call_args.kwargs
+    assert promote_kwargs['model_name'] == 'test_model'
+    assert promote_kwargs['version'] == '7'
+    assert promote_kwargs['alias'] == 'production'
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_minimal_retrain_failure_writes_report_without_version_columns(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_minimal_retrain: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    mlflow_repository_stub,
+):
+    """
+    Scenario MR.2.1: ``wrapper.retrain`` raises. The activity must catch the
+    error, return ``success=False`` so ``update_production_model`` is skipped,
+    and ``format_retrain_report`` must produce a row with the error message
+    and NULL version columns.
+    """
+    client = temporal_test_env.client
+    model_id = 721
+
+    _seed_retrain_training_rows(postgres_engine, model_id)
+    _configure_retrain_happy_path(mlflow_repository_stub)
+    mlflow_repository_stub.get_cached_model.return_value.force_retrain_error = True
+
+    input_data = _retrain_input(model_id)
+
+    with patch('laborious.activities.mlflow.mlflow.log_artifact'):
+        await start_and_await_workflow(
+            client,
+            MinimalRetrain.run,
+            input_data,
+            make_workflow_id('test-retrain-failure'),
+        )
+
+    with postgres_engine.connect() as conn:
+        rows = (
+            conn.execute(
+                text(
+                    'SELECT * FROM sientia_data.log_retrain '
+                    'WHERE model_id = :m'
+                ),
+                {'m': str(model_id)},
+            )
+            .mappings()
+            .all()
+        )
+
+    assert len(rows) == 1
+    row = rows[0]
+    assert row['model_id'] == str(model_id)
+    assert row['model_name'] == 'test_model'
+    assert 'training did not converge' in row['status']
+    assert row['version'] is None
+    assert row['mlflow_run_id'] is None
+    assert row['mlflow_experiment_id'] is None
+
+    mlflow_repository_stub.promote_to_alias.assert_not_called()
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_minimal_retrain_missing_target_writes_failure_report(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_minimal_retrain: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    mlflow_repository_stub,
+):
+    """
+    Scenario MR.2.2: ``model_config`` does not declare ``target``. The retrain
+    activity must short-circuit before any MLflow call and the report row must
+    carry the explicit guard message.
+    """
+    client = temporal_test_env.client
+    model_id = 722
+
+    _seed_retrain_training_rows(postgres_engine, model_id)
+    _configure_retrain_happy_path(mlflow_repository_stub)
+
+    input_data = _retrain_input(model_id, model_config={})
+
+    with patch('laborious.activities.mlflow.mlflow.log_artifact'):
+        await start_and_await_workflow(
+            client,
+            MinimalRetrain.run,
+            input_data,
+            make_workflow_id('test-retrain-missing-target'),
+        )
+
+    with postgres_engine.connect() as conn:
+        row = (
+            conn.execute(
+                text(
+                    'SELECT * FROM sientia_data.log_retrain '
+                    'WHERE model_id = :m'
+                ),
+                {'m': str(model_id)},
+            )
+            .mappings()
+            .first()
+        )
+
+    assert row is not None
+    assert 'target' in row['status'].lower(), (
+        f"expected target-missing message, got status={row['status']!r}"
+    )
+    assert row['version'] is None
+    mlflow_repository_stub.get_cached_model.assert_not_called()
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_minimal_retrain_no_training_data_does_not_persist_report(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_minimal_retrain: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    mlflow_repository_stub,
+):
+    """
+    Scenario MR.3.1: When the training query returns no rows the workflow must
+    not persist any report row. The workflow currently raises plain
+    ``ValueError`` which Temporal treats as a workflow-task failure (causing
+    indefinite retries until the test environment times out), so the assertion
+    here is constrained to the persistence side-effect. See ``e2e/CODE_ISSUES.md``
+    issue MR-1 for the recommended ``ApplicationError`` fix.
+    """
+    client = temporal_test_env.client
+    model_id = 731
+
+    with postgres_engine.begin() as conn:
+        conn.execute(text(f'DELETE FROM sientia_data.laborious_data WHERE model_id = {model_id}'))
+
+    _configure_retrain_happy_path(mlflow_repository_stub)
+
+    input_data = _retrain_input(model_id)
+
+    with pytest.raises(Exception), patch('laborious.activities.mlflow.mlflow.log_artifact'):
+        await start_and_await_workflow(
+            client,
+            MinimalRetrain.run,
+            input_data,
+            make_workflow_id('test-retrain-no-data'),
+        )
+
+    with postgres_engine.connect() as conn:
+        count = conn.execute(
+            text('SELECT COUNT(*) FROM sientia_data.log_retrain WHERE model_id = :m'),
+            {'m': str(model_id)},
+        ).scalar()
+    assert count == 0

e2e/test_minio_offload.py

@@ -9,7 +9,12 @@ from sqlalchemy import text
 from temporalio.testing import WorkflowEnvironment
 from temporalio.worker import Worker
 
-from e2e.helpers import insert_sample_data, make_workflow_id, start_and_await_workflow
+from e2e.helpers import (
+    insert_sample_data,
+    load_scenario_input,
+    make_workflow_id,
+    start_and_await_workflow,
+)
 from laborious.activities.activities import Activities
 from laborious.utils.models import minio_dataframe_payload as mdp
 from laborious.workflows.predictions_batch import PredictionsBatch
@@ -29,33 +34,17 @@ async def test_load_query_with_minio_offload_writes_object_to_bucket(
     """
     model_id = 501
     with postgres_engine.begin() as conn:
-        conn.execute(text(f'DELETE FROM predictions_schema.laborious_data WHERE model_id = {model_id}'))
+        conn.execute(text(f'DELETE FROM sientia_data.laborious_data WHERE model_id = {model_id}'))
     insert_sample_data(postgres_engine, model_id, [1.0, 2.0])
 
-    metadata = {
+    scenario_input = load_scenario_input('minio_offload_load_query.json', model_id=model_id)
-        'metadata': {
+    metadata = {'metadata': scenario_input['metadata']}
-            'schedule_name': 'test-schedule',
-            'model_name': 'test_model',
-            'model_id': model_id,
-            'workflow_name': 'predictions_batch',
-        }
-    }
     with patch.object(mdp, 'OFFLOAD_THRESHOLD_BYTES', 1):
-        payload = await test_activities_real_minio.load_query_with_minio_offload(
+        payload = test_activities_real_minio.load_query_with_minio_offload(scenario_input)
-            {
-                **metadata,
-                'query': (
-                    'SELECT timestamp, variable, value, created_at '
-                    f'FROM predictions_schema.laborious_data WHERE model_id = {model_id}'
-                ),
-                'model_name': 'test_model',
-                'datetime_columns': ['timestamp', 'created_at'],
-            }
-        )
     assert payload.object_key, 'offloaded payload must reference a MinIO object'
     assert payload.data is None or payload.data == {}, 'large payloads should not inline tabular dict'
 
-    df = await payload.retrieve(test_activities_real_minio.minio_repository, metadata['metadata'])
+    df = payload.retrieve(test_activities_real_minio.minio_repository, metadata['metadata'])
     assert len(df) >= 1
 
     client = minio_container.get_client()
@@ -77,37 +66,12 @@ async def test_predictions_batch_with_minio_offload_path(
     """
     model_id = 502
     with postgres_engine.begin() as conn:
-        conn.execute(text(f'DELETE FROM predictions_schema.laborious_data WHERE model_id = {model_id}'))
+        conn.execute(text(f'DELETE FROM sientia_data.laborious_data WHERE model_id = {model_id}'))
-        conn.execute(text(f'DELETE FROM predictions_schema.predictions WHERE model_id = {model_id}'))
+        conn.execute(text(f'DELETE FROM sientia_data.predictions WHERE model_id = {model_id}'))
-        conn.execute(text(f'DELETE FROM predictions_schema.transformed_data WHERE model_id = {model_id}'))
+        conn.execute(text(f'DELETE FROM sientia_data.transformed_data WHERE model_id = {model_id}'))
     insert_sample_data(postgres_engine, model_id, [10.0, 20.0, 30.0])
 
-    input_data = {
+    input_data = load_scenario_input('minio_offload_workflow.json', model_id=model_id)
-        'schedule_name': 'test-schedule',
-        'model_name': 'test_model',
-        'model_id': model_id,
-        'query': (
-            'SELECT timestamp, variable, value, created_at '
-            f'FROM predictions_schema.laborious_data WHERE model_id = {model_id}'
-        ),
-        'schema': 'predictions_schema',
-        'table_name': 'predictions',
-        'transform_table_name': 'transformed_data',
-        'input_filters': {'EMPTY_DATA': {'POLICY': 'STOP', 'CONFIG': {}}},
-        'mlflow_transform_filters': {'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}}},
-        'mlflow_predict_filters': {'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}}},
-        'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
-        'opc_output_config': {},
-        'pi_web_api_output_config': {},
-        'save_transform': True,
-        'prediction_store_policy': 'lts:1',
-        'model_config': {
-            'retention_minutes': 0,
-            'transform_flavor': 'sklearn',
-            'predict_flavor': 'sklearn',
-        },
-        'datetime_columns': ['timestamp', 'created_at'],
-    }
 
     with patch.object(mdp, 'OFFLOAD_THRESHOLD_BYTES', 1):
         await start_and_await_workflow(
@@ -119,6 +83,26 @@ async def test_predictions_batch_with_minio_offload_path(
 
     with postgres_engine.connect() as conn:
         count = conn.execute(
-            text(f'SELECT COUNT(*) FROM predictions_schema.predictions WHERE model_id = {model_id}')
+            text(f'SELECT COUNT(*) FROM sientia_data.predictions WHERE model_id = {model_id}')
         ).scalar()
     assert count == 1
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_load_query_with_inline_payload_when_below_threshold(
+    postgres_engine,
+    test_activities_real_minio: Activities,
+):
+    """Scenario 4.2.1: payload stays inline when threshold is high enough."""
+    model_id = 503
+    with postgres_engine.begin() as conn:
+        conn.execute(text(f'DELETE FROM sientia_data.laborious_data WHERE model_id = {model_id}'))
+    insert_sample_data(postgres_engine, model_id, [1.0, 2.0])
+
+    scenario_input = load_scenario_input('minio_offload_load_query.json', model_id=model_id)
+    with patch.object(mdp, 'OFFLOAD_THRESHOLD_BYTES', 10**9):
+        payload = test_activities_real_minio.load_query_with_minio_offload(scenario_input)
+
+    assert payload.object_key is None
+    assert payload.data is not None

e2e/test_opc_real_server.py

@@ -0,0 +1,201 @@
+"""
+E2E tests for OPC export using an in-process asyncua server and real OpcRepository.
+
+Covers scenarios 3.1.2, 3.2.2, 3.2.4, and 3.2.5 from e2e/scenarios.md.
+Mock-based OPC tests remain in test_predictions_batch_format_export.py.
+"""
+
+import asyncio
+import threading
+import time
+
+import pytest
+from temporalio.testing import WorkflowEnvironment
+from temporalio.worker import Worker
+
+from e2e.helpers import assert_prediction, insert_sample_data, make_workflow_id, start_and_await_workflow
+from e2e.opc_test_server import UNKNOWN_NODE_ID, OpcE2ETestServer, build_opc_output_config
+from e2e.test_predictions_batch_format_export import get_base_input_data
+from laborious.activities.activities import Activities
+from laborious.activities.opc import OPC_RECONNECT_IN_PROGRESS_COMMENT
+from laborious.utils.repository.opc_repository import OpcRepository
+from laborious.workflows.predictions_batch import PredictionsBatch
+
+
+def _slow_reconnect_under_lock(repo: OpcRepository, hold_seconds: float = 0.75) -> None:
+    """
+    Hold the connection lock briefly so concurrent writes see reconnect_in_progress.
+
+    Args:
+        repo (OpcRepository): Connected repository.
+        hold_seconds (float): Time to keep the lock before reconnecting.
+    """
+    with repo._connection_lock:
+        time.sleep(hold_seconds)
+        repo._reconnect_locked()
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+@pytest.mark.opc
+async def test_scenario_3_1_2_export_with_opc_only_real_server(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_real_opc: Worker,
+    test_activities_real_opc: Activities,
+    opc_e2e_server: OpcE2ETestServer,
+    postgres_engine,
+):
+    """
+    Scenario 3.1.2 (real OPC): connect, write prediction and confidence, verify server values.
+    """
+    client = temporal_test_env.client
+    model_id = 412
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+
+    input_data = get_base_input_data(model_id)
+    input_data['opc_output_config'] = build_opc_output_config(opc_e2e_server.node_ids)
+    input_data['pi_web_api_output_config'] = None
+
+    await start_and_await_workflow(
+        client,
+        PredictionsBatch.run,
+        input_data,
+        make_workflow_id('test-opc-real-happy'),
+    )
+
+    test_activities_real_opc.pi_web_api_client.write_value.assert_not_called()
+    assert await opc_e2e_server.read_prediction() == pytest.approx(0.5)
+    assert await opc_e2e_server.read_confidence() == pytest.approx(0.0)
+    assert_prediction(postgres_engine, model_id)
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+@pytest.mark.opc
+async def test_scenario_3_2_2_opc_write_error_real_server(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_real_opc: Worker,
+    test_activities_real_opc: Activities,
+    opc_e2e_server: OpcE2ETestServer,
+    postgres_engine,
+):
+    """
+    Scenario 3.2.2 (real OPC): unknown NodeId yields generic write failure (confidence 12).
+    """
+    client = temporal_test_env.client
+    model_id = 422
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+
+    node_ids = opc_e2e_server.node_ids
+    input_data = get_base_input_data(model_id)
+    input_data['opc_output_config'] = build_opc_output_config(
+        node_ids,
+        prediction_tag=UNKNOWN_NODE_ID,
+        confidence_tag=UNKNOWN_NODE_ID,
+    )
+    input_data['pi_web_api_output_config'] = None
+
+    await start_and_await_workflow(
+        client,
+        PredictionsBatch.run,
+        input_data,
+        make_workflow_id('test-opc-real-bad-node'),
+    )
+
+    assert_prediction(
+        postgres_engine,
+        model_id,
+        prediction_confidence=12,
+        comments='Some data could not be written to OPC servers',
+    )
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+@pytest.mark.opc
+async def test_scenario_3_2_4_opc_session_bad_real_server(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_real_opc: Worker,
+    test_activities_real_opc: Activities,
+    opc_e2e_server: OpcE2ETestServer,
+    postgres_engine,
+):
+    """
+    Scenario 3.2.4 (real OPC): server PreWrite fault injects BadSessionIdInvalid (confidence 14).
+    """
+    client = temporal_test_env.client
+    model_id = 424
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+
+    opc_e2e_server.set_session_bad_on_write(True)
+    try:
+        input_data = get_base_input_data(model_id)
+        input_data['opc_output_config'] = build_opc_output_config(
+            opc_e2e_server.node_ids,
+            prediction_only=True,
+        )
+        input_data['pi_web_api_output_config'] = None
+
+        await start_and_await_workflow(
+            client,
+            PredictionsBatch.run,
+            input_data,
+            make_workflow_id('test-opc-real-session-bad'),
+        )
+    finally:
+        opc_e2e_server.set_session_bad_on_write(False)
+
+    assert_prediction(
+        postgres_engine,
+        model_id,
+        prediction_confidence=14,
+        comments_contains='OPC UA session/channel error: BadSessionIdInvalid',
+    )
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+@pytest.mark.opc
+async def test_scenario_3_2_5_opc_write_blocked_during_reconnect_real_server(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_real_opc: Worker,
+    test_activities_real_opc: Activities,
+    opc_e2e_server: OpcE2ETestServer,
+    postgres_engine,
+):
+    """
+    Scenario 3.2.5 (real OPC): writes rejected while reconnect holds the connection lock.
+    """
+    client = temporal_test_env.client
+    model_id = 425
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+
+    repo = test_activities_real_opc.opc_repository['1']
+    repo._session_ready.clear()
+    reconnect_thread = threading.Thread(
+        target=_slow_reconnect_under_lock,
+        args=(repo,),
+        daemon=True,
+    )
+    reconnect_thread.start()
+
+    input_data = get_base_input_data(model_id)
+    input_data['opc_output_config'] = build_opc_output_config(opc_e2e_server.node_ids)
+    input_data['pi_web_api_output_config'] = None
+
+    try:
+        await start_and_await_workflow(
+            client,
+            PredictionsBatch.run,
+            input_data,
+            make_workflow_id('test-opc-real-reconnect-block'),
+        )
+    finally:
+        reconnect_thread.join(timeout=5.0)
+
+    assert_prediction(
+        postgres_engine,
+        model_id,
+        prediction_confidence=14,
+        comments_contains=OPC_RECONNECT_IN_PROGRESS_COMMENT,
+    )

e2e/test_predictions_batch_format_export.py

@@ -4,7 +4,7 @@ End-to-end tests for PredictionsBatch workflow - Format and Export scenarios.
 
 from decimal import Decimal
 from typing import Any, cast
-from unittest.mock import ANY, AsyncMock, call
+from unittest.mock import call
 
 import pytest
 from sientia_do.notifications.models import NotificationLevel
@@ -12,48 +12,18 @@ from sqlalchemy import text
 from temporalio.testing import WorkflowEnvironment
 from temporalio.worker import Worker
 
-from e2e.helpers import assert_prediction, insert_sample_data, make_workflow_id, start_and_await_workflow
+from e2e.helpers import (
+    assert_prediction,
+    insert_sample_data,
+    load_scenario_input,
+    make_workflow_id,
+    start_and_await_workflow,
+)
 from laborious.activities.activities import Activities
 from laborious.workflows.predictions_batch import PredictionsBatch
 
-base_input_data = {
-    'schedule_name': 'test-schedule',
-    'model_name': 'test_model',
-    'model_id': 301,
-    'query': 'SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = 301',
-    'schema': 'predictions_schema',
-    'table_name': 'predictions',
-    'transform_table_name': 'transformed_data',
-    'input_filters': {
-        'EMPTY_DATA': {'POLICY': 'STOP', 'CONFIG': {}},
-    },
-    'mlflow_transform_filters': {
-        'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}},
-    },
-    'mlflow_predict_filters': {
-        'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}},
-    },
-    'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
-    'opc_output_config': {},
-    'pi_web_api_output_config': {},
-    'save_transform': True,
-    'prediction_store_policy': 'lts:1',
-    'model_config': {
-        'retention_minutes': 0,
-        'transform_flavor': 'sklearn',
-        'predict_flavor': 'sklearn',
-    },
-    'datetime_columns': ['timestamp', 'created_at'],
-}
-
-base_query = "SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = {model_id}"
-
 def get_base_input_data(model_id):
-    return {
+    return load_scenario_input('format_export_base.json', model_id=model_id)
-        **base_input_data,
-        'model_id': model_id,
-        'query': base_query.format(model_id=model_id),
-    }
 
 
 @pytest.mark.asyncio
@@ -79,8 +49,8 @@ async def test_scenario_3_1_1_default_prediction_export(
     model_id = 311
 
     with postgres_engine.begin() as conn:
-        conn.execute(text(f"DELETE FROM predictions_schema.predictions WHERE model_id = {model_id}"))
+        conn.execute(text(f"DELETE FROM sientia_data.predictions WHERE model_id = {model_id}"))
-        conn.execute(text(f"DELETE FROM predictions_schema.transformed_data WHERE model_id = {model_id}"))
+        conn.execute(text(f"DELETE FROM sientia_data.transformed_data WHERE model_id = {model_id}"))
     insert_sample_data(postgres_engine, model_id, ['NULL', 78.2])
 
     input_data = get_base_input_data(model_id)
@@ -153,7 +123,6 @@ async def test_scenario_3_1_1_default_prediction_export(
                 'addr_1',
                 0,
                 'float',
-                ANY,
                 {
                     'model_id': 311,
                     'model_name': 'test_model',
@@ -165,7 +134,6 @@ async def test_scenario_3_1_1_default_prediction_export(
                 'addr_2',
                 2,
                 'float',
-                ANY,
                 {
                     'model_id': 311,
                     'model_name': 'test_model',
@@ -178,7 +146,7 @@ async def test_scenario_3_1_1_default_prediction_export(
 
     with postgres_engine.connect() as conn:
         tf_count = conn.execute(
-            text(f"SELECT COUNT(*) FROM predictions_schema.transformed_data WHERE model_id = {model_id}")
+            text(f"SELECT COUNT(*) FROM sientia_data.transformed_data WHERE model_id = {model_id}")
         ).scalar()
         assert tf_count == 0, 'transform export must be skipped when path_flag is set'
 
@@ -252,20 +220,28 @@ async def test_scenario_3_1_2_export_with_opc_only(
     opc_write_data = cast(Any, test_activities.opc_repository['1'].write_data)
     opc_write_data.assert_has_calls(
         [
-            call('addr_1', 0.5, 'float', ANY,
+            call(
+                'addr_1',
+                0.5,
+                'float',
                 {
                     'model_id': 312,
                     'model_name': 'test_model',
                     'schedule_name': 'test-schedule',
                     'workflow_name': 'predictions_batch',
-            }),
+                },
-            call('addr_2', 0, 'float', ANY,
+            ),
+            call(
+                'addr_2',
+                0,
+                'float',
                 {
                     'model_id': 312,
                     'model_name': 'test_model',
                     'schedule_name': 'test-schedule',
                     'workflow_name': 'predictions_batch',
-            }),
+                },
+            ),
         ]
     )
 
@@ -437,7 +413,7 @@ async def test_scenario_3_1_5_export_without_transformed_data(
 
     insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
     with postgres_engine.begin() as conn:
-        conn.execute(text(f"DELETE FROM predictions_schema.transformed_data WHERE model_id = {model_id}"))
+        conn.execute(text(f"DELETE FROM sientia_data.transformed_data WHERE model_id = {model_id}"))
 
     input_data = get_base_input_data(model_id)
     input_data['save_transform'] = False  # Don't save transformed data
@@ -500,26 +476,34 @@ async def test_scenario_3_1_5_export_without_transformed_data(
     opc_write_data = cast(Any, test_activities.opc_repository['1'].write_data)
     opc_write_data.assert_has_calls(
         [
-            call('addr_1', 0.5, 'float', ANY,
+            call(
+                'addr_1',
+                0.5,
+                'float',
                 {
                     'model_id': 315,
                     'model_name': 'test_model',
                     'schedule_name': 'test-schedule',
                     'workflow_name': 'predictions_batch',
-            }),
+                },
-            call('addr_2', 0, 'float', ANY,
+            ),
+            call(
+                'addr_2',
+                0,
+                'float',
                 {
                     'model_id': 315,
                     'model_name': 'test_model',
                     'schedule_name': 'test-schedule',
                     'workflow_name': 'predictions_batch',
-            }),
+                },
+            ),
         ]
     )
 
     with postgres_engine.connect() as conn:
         result_query = conn.execute(
-            text(f"SELECT COUNT(*) FROM predictions_schema.transformed_data WHERE model_id = {model_id}")
+            text(f"SELECT COUNT(*) FROM sientia_data.transformed_data WHERE model_id = {model_id}")
         )
         count = result_query.scalar()
         assert count == 0, f"Expected transform table to be empty, but found {count} records"
@@ -648,6 +632,109 @@ async def test_scenario_3_2_2_opc_write_error(
     )
 
 
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_3_2_4_opc_session_bad_mock(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 3.2.4 (mock): Tier-1 session error maps to confidence 14.
+    """
+    client = temporal_test_env.client
+    model_id = 324
+
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+
+    opc_write_data = cast(Any, test_activities.opc_repository['1'].write_data)
+    opc_write_data.return_value = (
+        False,
+        {
+            'opc_error_kind': 'session_bad',
+            'opc_status': 'BadSessionIdInvalid',
+            'notification_id': 'OPC_WRITE_DATA_ERROR_1',
+            'message': 'OPC session invalid',
+            'block': 'opc_repository',
+            'level': NotificationLevel.ERROR,
+            'attachment_content': 'BadSessionIdInvalid',
+        },
+    )
+
+    input_data = get_base_input_data(model_id)
+    input_data['opc_output_config'] = {
+        '1': {
+            'prediction_tags': {'addr_1': {'data_type': 'float'}},
+            'confidence_tags': {'addr_2': {'data_type': 'float'}},
+        }
+    }
+    input_data['pi_web_api_output_config'] = None
+
+    await start_and_await_workflow(
+        client, PredictionsBatch.run, input_data, make_workflow_id('test-opc-session-bad-mock')
+    )
+
+    assert_prediction(
+        postgres_engine,
+        model_id,
+        prediction_confidence=14,
+        comments_contains='OPC UA session/channel error: BadSessionIdInvalid',
+    )
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_3_2_5_opc_reconnect_in_progress_mock(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 3.2.5 (mock): reconnect_in_progress maps to confidence 14.
+    """
+    client = temporal_test_env.client
+    model_id = 325
+
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+
+    opc_write_data = cast(Any, test_activities.opc_repository['1'].write_data)
+    opc_write_data.return_value = (
+        False,
+        {
+            'opc_error_kind': 'reconnect_in_progress',
+            'notification_id': 'OPC_WRITE_DATA_ERROR_1',
+            'message': 'OPC reconnect in progress',
+            'block': 'opc_repository',
+            'level': NotificationLevel.ERROR,
+            'attachment_content': 'reconnect',
+        },
+    )
+
+    input_data = get_base_input_data(model_id)
+    input_data['opc_output_config'] = {
+        '1': {
+            'prediction_tags': {'addr_1': {'data_type': 'float'}},
+            'confidence_tags': {'addr_2': {'data_type': 'float'}},
+        }
+    }
+    input_data['pi_web_api_output_config'] = None
+
+    await start_and_await_workflow(
+        client,
+        PredictionsBatch.run,
+        input_data,
+        make_workflow_id('test-opc-reconnect-mock'),
+    )
+
+    assert_prediction(
+        postgres_engine,
+        model_id,
+        prediction_confidence=14,
+        comments_contains='OPC UA reconnect in progress',
+    )
+
 
 @pytest.mark.asyncio
 @pytest.mark.integration
@@ -669,8 +756,8 @@ async def test_scenario_3_2_3_pi_web_api_partial_write_error(
 
     insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
 
-    test_activities.pi_web_api_client.write_value = AsyncMock(
+    test_activities.pi_web_api_client.set_side_effect(
-        side_effect=[
+        [
             # Prediction batch: two web_ids requested, only one acknowledged.
             [{'WebId': 'web_id_1', 'Errors': []}],
             # Confidence write succeeds.
@@ -709,3 +796,41 @@ async def test_scenario_3_2_3_pi_web_api_partial_write_error(
         comments="The number of written tags does not match the number of tag names: Expected ['tag_1', 'tag_3'] tags, but ['tag_1'] tags were written.",
     )
 
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_3_3_1_combined_pi_and_opc_outputs(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 3.3.1: PI and OPC enabled together.
+    """
+    client = temporal_test_env.client
+    model_id = 333
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+
+    input_data = get_base_input_data(model_id)
+    input_data['pi_web_api_output_config'] = {
+        'endpoint': 'test_endpoint',
+        'prediction_tags': {'tag_1': 'web_id_1'},
+        'confidence_tags': {'tag_2': 'web_id_2'},
+    }
+    input_data['opc_output_config'] = {
+        '1': {
+            'prediction_tags': {'addr_1': {'data_type': 'float'}},
+            'confidence_tags': {'addr_2': {'data_type': 'float'}},
+        }
+    }
+
+    await start_and_await_workflow(
+        client, PredictionsBatch.run, input_data, make_workflow_id('test-pi-opc-combined')
+    )
+
+    assert test_activities.pi_web_api_client.write_value.call_count == 2
+    opc_write_data = cast(Any, test_activities.opc_repository['1'].write_data)
+    assert opc_write_data.call_count == 2
+    assert_prediction(postgres_engine, model_id)
+    

e2e/test_predictions_batch_main_workflow.py

@@ -10,7 +10,7 @@ from temporalio.worker import Worker
 
 import pytest
 
-from e2e.helpers import make_workflow_id, start_and_await_workflow
+from e2e.helpers import load_scenario_input, make_workflow_id, start_and_await_workflow
 from laborious.activities.activities import Activities
 from laborious.workflows.predictions_batch import PredictionsBatch
 
@@ -27,9 +27,9 @@ async def test_scenario_1_1_1_happy_path_complete_success(
     client = temporal_test_env.client
 
     with postgres_engine.begin() as conn:
-        conn.execute(text('DELETE FROM predictions_schema.laborious_data WHERE model_id = 123'))
+        conn.execute(text('DELETE FROM sientia_data.laborious_data WHERE model_id = 123'))
         insert_sql = """
-            INSERT INTO predictions_schema.laborious_data (model_id, variable, value, timestamp, created_at)
+            INSERT INTO sientia_data.laborious_data (model_id, variable, value, timestamp, created_at)
             VALUES
                 (123, 'sensor_1', 23.5, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00'),
                 (123, 'sensor_2', 78.2, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00'),
@@ -37,43 +37,7 @@ async def test_scenario_1_1_1_happy_path_complete_success(
         """
         conn.execute(text(insert_sql))
 
-    input_data = {
+    input_data = load_scenario_input('main_happy_path.json', model_id=123)
-        'metadata': {
-            'metadata': {
-                'model_id': 123,
-                'model_name': 'test_model',
-                'schedule_name': 'test-schedule',
-                'workflow_name': 'predictions_batch',
-            }
-        },
-        'schedule_name': 'test-schedule',
-        'model_name': 'test_model',
-        'model_id': 123,
-        'query': 'SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = 123',
-        'schema': 'predictions_schema',
-        'table_name': 'predictions',
-        'transform_table_name': 'transformed_data',
-        'input_filters': {
-            'EMPTY_DATA': {'POLICY': 'STOP', 'CONFIG': {}},
-        },
-        'mlflow_transform_filters': {
-            'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}},
-        },
-        'mlflow_predict_filters': {
-            'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}},
-        },
-        'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
-        'opc_output_config': {},
-        'pi_web_api_output_config': {},
-        'save_transform': True,
-        'prediction_store_policy': 'lts:1',
-        'model_config': {
-            'retention_minutes': 0,
-            'transform_flavor': 'sklearn',
-            'predict_flavor': 'sklearn',
-        },
-        'datetime_columns': ['timestamp', 'created_at'],
-    }
 
     await start_and_await_workflow(
         client,
@@ -82,7 +46,7 @@ async def test_scenario_1_1_1_happy_path_complete_success(
         make_workflow_id('test-predictions-batch'),
     )
 
-    schema_name = 'predictions_schema'
+    schema_name = 'sientia_data'
     with postgres_engine.connect() as conn:
         result_query = conn.execute(
             text(
@@ -126,42 +90,7 @@ async def test_scenario_1_2_1_sql_query_execution_error(
     """Invalid SQL: workflow may complete with early exit; no prediction rows."""
     client = temporal_test_env.client
 
-    input_data = {
+    input_data = load_scenario_input('main_sql_error.json', model_id=128)
-        'metadata': {
-            'metadata': {
-                'model_id': 128,
-                'model_name': 'test_model',
-                'schedule_name': 'test-schedule',
-                'workflow_name': 'predictions_batch',
-            }
-        },
-        'schedule_name': 'test-schedule',
-        'model_name': 'test_model',
-        'model_id': 128,
-        'query': 'SELECT * FROM nonexistent_table WHERE invalid_syntax =',
-        'schema': 'predictions_schema',
-        'table_name': 'predictions',
-        'transform_table_name': 'transformed_data',
-        'input_filters': {
-            'EMPTY_DATA': {'POLICY': 'STOP', 'CONFIG': {}},
-        },
-        'mlflow_transform_filters': {
-            'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}},
-        },
-        'mlflow_predict_filters': {
-            'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}},
-        },
-        'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
-        'opc_output_config': {},
-        'pi_web_api_output_config': {},
-        'save_transform': True,
-        'prediction_store_policy': 'lts:1',
-        'model_config': {
-            'retention_minutes': 0,
-            'transform_flavor': 'sklearn',
-            'predict_flavor': 'sklearn',
-        },
-    }
 
     await start_and_await_workflow(
         client,
@@ -172,7 +101,7 @@ async def test_scenario_1_2_1_sql_query_execution_error(
 
     with postgres_engine.connect() as conn:
         count = conn.execute(
-            text('SELECT COUNT(*) FROM predictions_schema.predictions WHERE model_id = 128')
+            text('SELECT COUNT(*) FROM sientia_data.predictions WHERE model_id = 128')
         ).scalar()
     assert count == 0
 
@@ -188,22 +117,7 @@ async def test_scenario_1_2_2_missing_required_parameters(
     """Missing query: workflow does not produce predictions and is terminated explicitly."""
     client = temporal_test_env.client
 
-    input_data = {
+    input_data = load_scenario_input('main_missing_required.json', model_id=129)
-        'metadata': {
-            'metadata': {
-                'model_id': 129,
-                'model_name': 'test_model',
-                'schedule_name': 'test-schedule',
-                'workflow_name': 'predictions_batch',
-            }
-        },
-        'schedule_name': 'test-schedule',
-        'model_name': 'test_model',
-        'model_id': 129,
-        'schema': 'predictions_schema',
-        'table_name': 'predictions',
-        'transform_table_name': 'transformed_data',
-    }
 
     handle = await client.start_workflow(
         PredictionsBatch.run,
@@ -217,7 +131,7 @@ async def test_scenario_1_2_2_missing_required_parameters(
 
     with postgres_engine.connect() as conn:
         count = conn.execute(
-            text('SELECT COUNT(*) FROM predictions_schema.predictions WHERE model_id = 129')
+            text('SELECT COUNT(*) FROM sientia_data.predictions WHERE model_id = 129')
         ).scalar()
     assert count == 0
 
@@ -236,53 +150,17 @@ async def test_scenario_1_2_3_invalid_datetime_column_specification(
     client = temporal_test_env.client
 
     with postgres_engine.begin() as conn:
-        conn.execute(text('DELETE FROM predictions_schema.laborious_data WHERE model_id = 130'))
+        conn.execute(text('DELETE FROM sientia_data.laborious_data WHERE model_id = 130'))
         conn.execute(
             text(
                 """
-            INSERT INTO predictions_schema.laborious_data (model_id, variable, value, timestamp, created_at)
+            INSERT INTO sientia_data.laborious_data (model_id, variable, value, timestamp, created_at)
             VALUES (130, 'sensor_1', 23.5, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00')
         """
             )
         )
 
-    input_data = {
+    input_data = load_scenario_input('main_invalid_datetime.json', model_id=130)
-        'metadata': {
-            'metadata': {
-                'model_id': 130,
-                'model_name': 'test_model',
-                'schedule_name': 'test-schedule',
-                'workflow_name': 'predictions_batch',
-            }
-        },
-        'schedule_name': 'test-schedule',
-        'model_name': 'test_model',
-        'model_id': 130,
-        'query': 'SELECT timestamp, variable, value FROM predictions_schema.laborious_data WHERE model_id = 130',
-        'schema': 'predictions_schema',
-        'table_name': 'predictions',
-        'transform_table_name': 'transformed_data',
-        'input_filters': {
-            'EMPTY_DATA': {'POLICY': 'STOP', 'CONFIG': {}},
-        },
-        'mlflow_transform_filters': {
-            'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}},
-        },
-        'mlflow_predict_filters': {
-            'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}},
-        },
-        'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
-        'opc_output_config': {},
-        'pi_web_api_output_config': {},
-        'save_transform': True,
-        'prediction_store_policy': 'lts:1',
-        'model_config': {
-            'retention_minutes': 0,
-            'transform_flavor': 'sklearn',
-            'predict_flavor': 'sklearn',
-        },
-        'datetime_columns': ['nonexistent_column'],
-    }
 
     handle = await client.start_workflow(
         PredictionsBatch.run,
@@ -296,7 +174,7 @@ async def test_scenario_1_2_3_invalid_datetime_column_specification(
 
     with postgres_engine.connect() as conn:
         count = conn.execute(
-            text('SELECT COUNT(*) FROM predictions_schema.predictions WHERE model_id = 130')
+            text('SELECT COUNT(*) FROM sientia_data.predictions WHERE model_id = 130')
         ).scalar()
     assert count == 0
 

e2e/test_predictions_batch_prediction_process.py

@@ -14,90 +14,54 @@ from temporalio.worker import Worker
 
 from e2e.helpers import (
     assert_continue,
+    assert_postgres_unique_violation_in_chain,
+    assert_prediction,
+    assert_prediction_row_count,
     assert_repeat,
     assert_stop,
     insert_sample_data,
+    insert_sample_prediction,
+    load_scenario_input,
     make_workflow_id,
     start_and_await_workflow,
 )
 from laborious.activities.activities import Activities
+from laborious.utils.models import minio_dataframe_payload as minio_payload_module
 from laborious.workflows.predictions_batch import PredictionsBatch
 
-base_input_data = {
+DISTINCT_BATCH_TIMESTAMP = '2024-01-01 13:00:00+00:00'
-    'schedule_name': 'test-schedule',
+HISTORY_TIMESTAMP = '2024-01-01 12:00:00+00:00'
-    'model_name': 'test_model',
-    'model_id': 201,
-    'query': 'SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = 201',
-    'schema': 'predictions_schema',
-    'table_name': 'predictions',
-    'transform_table_name': 'transformed_data',
-    'input_filters': {
-        'SPECIFIC_VARIABLES_NULL_VALUES': {
-            'POLICY': 'CONTINUE',
-            'CONFIG': {'variables': ['sensor_1']},
-        },
-    },
-    'mlflow_transform_filters': {
-        'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}},
-    },
-    'mlflow_predict_filters': {
-        'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}},
-    },
-    'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
-    'opc_output_config': {},
-    'pi_web_api_output_config': {},
-    'save_transform': True,
-    'prediction_store_policy': 'lts:1',
-    'model_config': {
-        'retention_minutes': 0,
-        'transform_flavor': 'sklearn',
-        'predict_flavor': 'sklearn',
-    },
-    'datetime_columns': ['timestamp', 'created_at'],
-}
-
-base_query = 'SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = {model_id}'
 
 
 def get_base_input_data(model_id):
-    return {
+    return load_scenario_input('prediction_process_base.json', model_id=model_id)
-        **base_input_data,
+
-        'model_id': model_id,
+
-        'query': base_query.format(model_id=model_id),
+@pytest.fixture
+def bad_data_model(mlflow_repository_stub):
+    mlflow_repository_stub.stub_wrapper.transform = MagicMock(
+        side_effect=Exception('Bad data model')
+    )
+    return mlflow_repository_stub.stub_wrapper
+
+
+@pytest.fixture
+def bad_predict_model(mlflow_repository_stub):
+    wrapper = mlflow_repository_stub.stub_wrapper
+
+    def _good_transform(data):
+        result = pd.DataFrame(
+            {
+                'feature_1': [0.234] * len(data),
+                'feature_2': [0.783] * len(data),
             }
+        )
+        result.index = data.index
+        return result, {}
 
-
+    wrapper.transform.side_effect = _good_transform
-def insert_sample_prediction(postgres_engine, model_id):
+    wrapper.predict = MagicMock(side_effect=Exception('Bad predict model'))
-    with postgres_engine.begin() as conn:
+    return wrapper
-        conn.execute(text(f'DELETE FROM predictions_schema.predictions WHERE model_id = {model_id}'))
-        insert_sql = f"""
-            INSERT INTO predictions_schema.predictions (model_id, timestamp, prediction, prediction_confidence, prediction_status, comments, response_time)
-            VALUES
-               ({model_id}, '2024-01-01 12:00:00+00:00', 10, 0, 'Good', '', 0.1)
-        """
-        conn.execute(text(insert_sql))
-    return (model_id, Decimal(10), Decimal(0), 'Good')
-
-
-@pytest.fixture
-def bad_data_model(patch_mlflow):
-    model = MagicMock(predict=MagicMock(side_effect=Exception('Bad data model')))
-    patch_mlflow.sklearn.load_model = MagicMock(return_value=model)
-    return model
-
-
-@pytest.fixture
-def bad_predict_model(patch_mlflow, mock_mlflow_models):
-    model = MagicMock(predict=MagicMock(side_effect=Exception('Bad predict model')))
-
-    def mock_sklearn_load_model(model_uri):
-        if 'data_model' in model_uri or 'transform' in model_uri.lower():
-            return mock_mlflow_models['transform_model']
-        return model
-
-    patch_mlflow.sklearn = MagicMock()
-    patch_mlflow.sklearn.load_model = MagicMock(side_effect=mock_sklearn_load_model)
-    return model
 
 
 @pytest.mark.asyncio
@@ -107,7 +71,7 @@ async def test_scenario_2_1_1_input_gate_triggers_continue(
     temporal_worker: Worker,
     test_activities: Activities,
     postgres_engine,
-    mock_mlflow_models,
+    mlflow_repository_stub,
 ):
     """Input gate CONTINUE: export default prediction; MLflow transform/predict not used."""
     client = temporal_test_env.client
@@ -118,8 +82,8 @@ async def test_scenario_2_1_1_input_gate_triggers_continue(
         client, PredictionsBatch.run, input_data, make_workflow_id('test-continue-policy')
     )
     assert_continue(postgres_engine, model_id)
-    mock_mlflow_models['transform_model'].predict.assert_not_called()
+    mlflow_repository_stub.stub_wrapper.transform.assert_not_called()
-    mock_mlflow_models['predict_model'].predict.assert_not_called()
+    mlflow_repository_stub.stub_wrapper.predict.assert_not_called()
 
 
 @pytest.mark.asyncio
@@ -129,7 +93,7 @@ async def test_scenario_2_1_2_input_gate_triggers_stop(
     temporal_worker: Worker,
     test_activities: Activities,
     postgres_engine,
-    mock_mlflow_models,
+    mlflow_repository_stub,
 ):
     """Input gate STOP: no export, no MLflow."""
     client = temporal_test_env.client
@@ -141,30 +105,60 @@ async def test_scenario_2_1_2_input_gate_triggers_stop(
         client, PredictionsBatch.run, input_data, make_workflow_id('test-input-stop')
     )
     assert_stop(postgres_engine, model_id)
-    mock_mlflow_models['transform_model'].predict.assert_not_called()
+    mlflow_repository_stub.stub_wrapper.transform.assert_not_called()
 
 
 @pytest.mark.asyncio
 @pytest.mark.integration
-async def test_scenario_2_1_3_input_gate_triggers_repeat(
+async def test_scenario_2_1_3_input_gate_repeat_batch_timestamp_equals_history_fails(
     temporal_test_env: WorkflowEnvironment,
     temporal_worker: Worker,
     test_activities: Activities,
     postgres_engine,
-    mock_mlflow_models,
+    mlflow_repository_stub,
 ):
-    """Input gate REPEAT with existing history."""
+    """
+    REPEAT uses ``last_timestamp`` from the batch payload as the new row's ``timestamp``.
+    When it equals the only historical prediction row, Postgres rejects the duplicate key.
+    """
     client = temporal_test_env.client
     model_id = 213
-    insert_sample_data(postgres_engine, model_id, ['NULL', 78.2])
+    insert_sample_data(postgres_engine, model_id, ['NULL', 78.2], data_timestamp=HISTORY_TIMESTAMP)
-    data = insert_sample_prediction(postgres_engine, model_id)
+    insert_sample_prediction(postgres_engine, model_id, prediction_timestamp=HISTORY_TIMESTAMP)
+    input_data = get_base_input_data(model_id)
+    input_data['input_filters']['SPECIFIC_VARIABLES_NULL_VALUES']['POLICY'] = 'REPEAT'
+    with pytest.raises(Exception) as excinfo:
+        await start_and_await_workflow(
+            client, PredictionsBatch.run, input_data, make_workflow_id('test-input-repeat-collision')
+        )
+    assert_postgres_unique_violation_in_chain(excinfo.value)
+    assert_prediction_row_count(postgres_engine, model_id, 1)
+    mlflow_repository_stub.stub_wrapper.transform.assert_not_called()
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_1_3_input_gate_repeat_distinct_batch_timestamp_inserts_second_row(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    mlflow_repository_stub,
+):
+    """REPEAT succeeds when batch ``last_timestamp`` differs from the historical prediction row."""
+    client = temporal_test_env.client
+    model_id = 2131
+    insert_sample_data(
+        postgres_engine, model_id, ['NULL', 78.2], data_timestamp=DISTINCT_BATCH_TIMESTAMP
+    )
+    data = insert_sample_prediction(postgres_engine, model_id, prediction_timestamp=HISTORY_TIMESTAMP)
     input_data = get_base_input_data(model_id)
     input_data['input_filters']['SPECIFIC_VARIABLES_NULL_VALUES']['POLICY'] = 'REPEAT'
     await start_and_await_workflow(
-        client, PredictionsBatch.run, input_data, make_workflow_id('test-input-repeat')
+        client, PredictionsBatch.run, input_data, make_workflow_id('test-input-repeat-ok')
     )
     assert_repeat(postgres_engine, model_id, data)
-    mock_mlflow_models['transform_model'].predict.assert_not_called()
+    mlflow_repository_stub.stub_wrapper.transform.assert_not_called()
 
 
 @pytest.mark.asyncio
@@ -180,7 +174,7 @@ async def test_scenario_2_1_4_input_gate_repeat_without_prior_prediction(
     model_id = 214
     insert_sample_data(postgres_engine, model_id, ['NULL', 78.2])
     with postgres_engine.begin() as conn:
-        conn.execute(text(f'DELETE FROM predictions_schema.predictions WHERE model_id = {model_id}'))
+        conn.execute(text(f'DELETE FROM sientia_data.predictions WHERE model_id = {model_id}'))
     input_data = get_base_input_data(model_id)
     input_data['input_filters']['SPECIFIC_VARIABLES_NULL_VALUES']['POLICY'] = 'REPEAT'
     await start_and_await_workflow(
@@ -222,7 +216,7 @@ async def test_scenario_2_2_2_transform_gate_triggers_stop(
     test_activities: Activities,
     postgres_engine,
     bad_data_model,
-    mock_mlflow_models,
+    mlflow_repository_stub,
 ):
     client = temporal_test_env.client
     model_id = 222
@@ -233,12 +227,12 @@ async def test_scenario_2_2_2_transform_gate_triggers_stop(
         client, PredictionsBatch.run, input_data, make_workflow_id('test-transform-stop')
     )
     assert_stop(postgres_engine, model_id)
-    mock_mlflow_models['predict_model'].predict.assert_not_called()
+    mlflow_repository_stub.stub_wrapper.predict.assert_not_called()
 
 
 @pytest.mark.asyncio
 @pytest.mark.integration
-async def test_scenario_2_2_3_transform_gate_triggers_repeat(
+async def test_scenario_2_2_3_transform_gate_repeat_batch_timestamp_equals_history_fails(
     temporal_test_env: WorkflowEnvironment,
     temporal_worker: Worker,
     test_activities: Activities,
@@ -247,12 +241,37 @@ async def test_scenario_2_2_3_transform_gate_triggers_repeat(
 ):
     client = temporal_test_env.client
     model_id = 223
-    insert_sample_data(postgres_engine, model_id, [60.0, 78.2])
+    insert_sample_data(postgres_engine, model_id, [60.0, 78.2], data_timestamp=HISTORY_TIMESTAMP)
-    data = insert_sample_prediction(postgres_engine, model_id)
+    insert_sample_prediction(postgres_engine, model_id, prediction_timestamp=HISTORY_TIMESTAMP)
+    input_data = get_base_input_data(model_id)
+    input_data['mlflow_transform_filters']['API_ERROR']['POLICY'] = 'REPEAT'
+    with pytest.raises(Exception) as excinfo:
+        await start_and_await_workflow(
+            client, PredictionsBatch.run, input_data, make_workflow_id('test-transform-repeat-collision')
+        )
+    assert_postgres_unique_violation_in_chain(excinfo.value)
+    assert_prediction_row_count(postgres_engine, model_id, 1)
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_2_3_transform_gate_repeat_distinct_batch_timestamp_inserts_second_row(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    bad_data_model,
+):
+    client = temporal_test_env.client
+    model_id = 2231
+    insert_sample_data(
+        postgres_engine, model_id, [60.0, 78.2], data_timestamp=DISTINCT_BATCH_TIMESTAMP
+    )
+    data = insert_sample_prediction(postgres_engine, model_id, prediction_timestamp=HISTORY_TIMESTAMP)
     input_data = get_base_input_data(model_id)
     input_data['mlflow_transform_filters']['API_ERROR']['POLICY'] = 'REPEAT'
     await start_and_await_workflow(
-        client, PredictionsBatch.run, input_data, make_workflow_id('test-transform-repeat')
+        client, PredictionsBatch.run, input_data, make_workflow_id('test-transform-repeat-ok')
     )
     assert_repeat(postgres_engine, model_id, data)
 
@@ -264,19 +283,20 @@ async def test_scenario_2_2_4_transform_content_gate_nan_values_stop(
     temporal_worker: Worker,
     test_activities: Activities,
     postgres_engine,
-    mock_mlflow_models,
+    mlflow_repository_stub,
 ):
     """mlflow_content_gate triggers STOP when transform output is all NaN (NAN_VALUES filter)."""
     client = temporal_test_env.client
     model_id = 224
 
     def all_nan_transform(data):
-        num_rows = max(len(data), 1) if hasattr(data, '__len__') else 1
+        result = pd.DataFrame(
-        result = pd.DataFrame({'feature_1': [np.nan] * num_rows, 'feature_2': [np.nan] * num_rows})
+            {'feature_1': [np.nan] * len(data), 'feature_2': [np.nan] * len(data)}
+        )
         result.index = data.index
-        return result
+        return result, {}
 
-    mock_mlflow_models['transform_model'].predict = MagicMock(side_effect=all_nan_transform)
+    mlflow_repository_stub.stub_wrapper.transform = MagicMock(side_effect=all_nan_transform)
 
     insert_sample_data(postgres_engine, model_id, [60.0, 78.2])
     input_data = get_base_input_data(model_id)
@@ -288,7 +308,7 @@ async def test_scenario_2_2_4_transform_content_gate_nan_values_stop(
         client, PredictionsBatch.run, input_data, make_workflow_id('test-transform-content-stop')
     )
     assert_stop(postgres_engine, model_id)
-    mock_mlflow_models['predict_model'].predict.assert_not_called()
+    mlflow_repository_stub.stub_wrapper.predict.assert_not_called()
 
 
 @pytest.mark.asyncio
@@ -338,7 +358,7 @@ async def test_scenario_2_3_2_predict_gate_triggers_stop(
 
 @pytest.mark.asyncio
 @pytest.mark.integration
-async def test_scenario_2_3_3_predict_gate_triggers_repeat(
+async def test_scenario_2_3_3_predict_gate_repeat_batch_timestamp_equals_history_fails(
     temporal_test_env: WorkflowEnvironment,
     temporal_worker: Worker,
     test_activities: Activities,
@@ -347,13 +367,39 @@ async def test_scenario_2_3_3_predict_gate_triggers_repeat(
 ):
     client = temporal_test_env.client
     model_id = 233
-    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2], data_timestamp=HISTORY_TIMESTAMP)
-    data = insert_sample_prediction(postgres_engine, model_id)
+    insert_sample_prediction(postgres_engine, model_id, prediction_timestamp=HISTORY_TIMESTAMP)
+    input_data = get_base_input_data(model_id)
+    input_data['mlflow_predict_filters']['API_ERROR']['POLICY'] = 'REPEAT'
+    input_data['path_priority'] = ['REPEAT', 'STOP', 'CONTINUE']
+    with pytest.raises(Exception) as excinfo:
+        await start_and_await_workflow(
+            client, PredictionsBatch.run, input_data, make_workflow_id('test-predict-repeat-collision')
+        )
+    assert_postgres_unique_violation_in_chain(excinfo.value)
+    assert_prediction_row_count(postgres_engine, model_id, 1)
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_3_3_predict_gate_repeat_distinct_batch_timestamp_inserts_second_row(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    bad_predict_model,
+):
+    client = temporal_test_env.client
+    model_id = 2331
+    insert_sample_data(
+        postgres_engine, model_id, [23.5, 78.2], data_timestamp=DISTINCT_BATCH_TIMESTAMP
+    )
+    data = insert_sample_prediction(postgres_engine, model_id, prediction_timestamp=HISTORY_TIMESTAMP)
     input_data = get_base_input_data(model_id)
     input_data['mlflow_predict_filters']['API_ERROR']['POLICY'] = 'REPEAT'
     input_data['path_priority'] = ['REPEAT', 'STOP', 'CONTINUE']
     await start_and_await_workflow(
-        client, PredictionsBatch.run, input_data, make_workflow_id('test-predict-repeat')
+        client, PredictionsBatch.run, input_data, make_workflow_id('test-predict-repeat-ok')
     )
     assert_repeat(postgres_engine, model_id, data)
 
@@ -370,10 +416,73 @@ async def test_scenario_2_4_1_input_empty_data_stop(
     client = temporal_test_env.client
     model_id = 241
     with postgres_engine.begin() as conn:
-        conn.execute(text(f'DELETE FROM predictions_schema.laborious_data WHERE model_id = {model_id}'))
+        conn.execute(text(f'DELETE FROM sientia_data.laborious_data WHERE model_id = {model_id}'))
     input_data = get_base_input_data(model_id)
     input_data['input_filters'] = {'EMPTY_DATA': {'POLICY': 'STOP', 'CONFIG': {}}}
     await start_and_await_workflow(
         client, PredictionsBatch.run, input_data, make_workflow_id('test-empty-data-stop')
     )
     assert_stop(postgres_engine, model_id)
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_4_1_priority_conflict_resolution(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    bad_data_model,
+):
+    """Conflicting filter outputs must honor configured path_priority order."""
+    client = temporal_test_env.client
+    model_id = 242
+    insert_sample_data(postgres_engine, model_id, [60.0, 78.2])
+    input_data = get_base_input_data(model_id)
+    input_data['mlflow_transform_filters'] = {'API_ERROR': {'POLICY': 'CONTINUE', 'CONFIG': {}}}
+    input_data['mlflow_predict_filters'] = {'API_ERROR': {'POLICY': 'STOP', 'CONFIG': {}}}
+    input_data['path_priority'] = ['STOP', 'CONTINUE', 'REPEAT']
+
+    await start_and_await_workflow(
+        client, PredictionsBatch.run, input_data, make_workflow_id('test-priority-conflict')
+    )
+    assert_continue(
+        postgres_engine=postgres_engine,
+        model_id=model_id,
+        prediction_confidence=Decimal(10),
+        comments='Unknown MLFlow API error',
+    )
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_e2e_request_predict_inline_minio_payload_with_datetimeindex(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    mlflow_repository_stub,
+):
+    """
+    High offload threshold forces inline tabular dicts; ``DatetimeIndex`` must serialize as JSON
+    (string index keys via ``MinioDataFramePayload.from_dataframe``) so ``request_predict`` completes.
+    """
+    client = temporal_test_env.client
+    model_id = 252
+    with postgres_engine.begin() as conn:
+        conn.execute(text(f'DELETE FROM sientia_data.laborious_data WHERE model_id = {model_id}'))
+        conn.execute(text(f'DELETE FROM sientia_data.predictions WHERE model_id = {model_id}'))
+        conn.execute(text(f'DELETE FROM sientia_data.transformed_data WHERE model_id = {model_id}'))
+    insert_sample_data(postgres_engine, model_id, [60.0, 78.2])
+    input_data = get_base_input_data(model_id)
+
+    with patch.object(minio_payload_module, 'OFFLOAD_THRESHOLD_BYTES', 10**9):
+        await start_and_await_workflow(
+            client,
+            PredictionsBatch.run,
+            input_data,
+            make_workflow_id('test-predict-inline-json-datetimeindex'),
+        )
+
+    assert_prediction(postgres_engine, model_id, prediction=0.5, prediction_confidence=0)
+    mlflow_repository_stub.stub_wrapper.predict.assert_called()

e2e/test_simple_metrics.py

@@ -0,0 +1,333 @@
+"""
+End-to-end tests for the SimpleMetrics workflow.
+
+Coverage focus:
+
+- Happy path computes rmse/mse/mae/r2 from predictions joined against ``laborious_data``
+  and persists rows to ``sientia_data.simple_metrics`` with all required columns.
+- Subset metric selection (only rmse) writes exactly the requested rows.
+- Zero-variance target produces ``r2=0`` per division-by-zero guard.
+- Empty join (no overlapping data) short-circuits without persisting anything.
+"""
+
+from datetime import datetime, timedelta, timezone
+from decimal import Decimal
+
+import math
+import pytest
+from sqlalchemy import text
+from temporalio.testing import WorkflowEnvironment
+from temporalio.worker import Worker
+
+from e2e.helpers import load_scenario_input, make_workflow_id, start_and_await_workflow
+from laborious.activities.activities import Activities
+from laborious.workflows.simple_metrics import SimpleMetrics
+
+# Columns defined by the production DDL for ``sientia_data.simple_metrics``.
+EXPECTED_SIMPLE_METRICS_COLUMNS = [
+    'id',
+    'model_id',
+    'metric',
+    'value',
+    'timestamp',
+    'data_size',
+    'interval_minutes',
+    'created_at',
+]
+
+# ``timestamp`` is now nullable per the new DDL (production code may write it
+# null when the upstream data has no usable instant); skip the non-null check
+# for it while still validating presence.
+NULLABLE_SIMPLE_METRICS_COLUMNS = {'timestamp'}
+
+
+def _simple_metrics_input(model_id: int, **overrides) -> dict:
+    """Load and override the simple-metrics base scenario."""
+    payload = load_scenario_input('simple_metrics_base.json', model_id=model_id)
+    payload.update(overrides)
+    return payload
+
+
+def _seed_predictions_and_targets(
+    postgres_engine,
+    model_id: int,
+    pairs: list[tuple[float, float]],
+    target_name: str = 'sensor_target',
+    offset_minutes: int = 6,
+) -> list[str]:
+    """
+    Insert matching prediction/target rows used by the SimpleMetrics SQL JOIN.
+
+    For each ``(prediction, target)`` pair we write a row in ``predictions`` and
+    a matching row in ``laborious_data`` with ``variable=target_name`` so the
+    inner join in the workflow query yields one row per pair.
+
+    Args:
+     - postgres_engine: SQLAlchemy engine bound to the test container.
+     - model_id: Model id stamped on every row.
+     - pairs: ``(prediction, target)`` pairs, one per minute.
+     - target_name: Variable name in ``laborious_data`` representing the target.
+     - offset_minutes: Earliest row sits this many minutes ago so timestamps fall
+       inside the workflow's recent-data window.
+
+    Return:
+     List of timestamp strings written for the inserted rows.
+    """
+    base = datetime.now(timezone.utc).replace(second=0, microsecond=0) - timedelta(
+        minutes=offset_minutes
+    )
+    timestamps = [
+        (base + timedelta(minutes=i)).strftime('%Y-%m-%d %H:%M:%S%z')
+        for i in range(len(pairs))
+    ]
+
+    prediction_rows = []
+    target_rows = []
+    for index, (prediction, target_value) in enumerate(pairs):
+        ts = timestamps[index]
+        prediction_rows.append(
+            f"({model_id}, {prediction}, 0, 0, 'Good', '{ts}', '{ts}')"
+        )
+        target_rows.append(
+            f"({model_id}, '{target_name}', {target_value}, '{ts}', '{ts}')"
+        )
+
+    with postgres_engine.begin() as conn:
+        conn.execute(text(f'DELETE FROM sientia_data.predictions WHERE model_id = {model_id}'))
+        conn.execute(text(f'DELETE FROM sientia_data.laborious_data WHERE model_id = {model_id}'))
+        # The SimpleMetrics SQL JOIN only filters ``predictions.model_id``; it does
+        # NOT filter ``laborious_data.model_id`` (see ``e2e/CODE_ISSUES.md`` issue
+        # SM-1). Without this cross-model cleanup, a previous test's target rows
+        # under the same variable name would join into this test's predictions
+        # whenever timestamps happened to overlap.
+        conn.execute(
+            text(
+                "DELETE FROM sientia_data.laborious_data "
+                "WHERE variable IN (:sensor_default, :target_name) "
+                "AND timestamp >= NOW() - INTERVAL '120 minutes'"
+            ),
+            {'sensor_default': 'sensor_target', 'target_name': target_name},
+        )
+        if prediction_rows:
+            conn.execute(
+                text(
+                    'INSERT INTO sientia_data.predictions '
+                    '(model_id, prediction, prediction_confidence, response_time, '
+                    'prediction_status, "timestamp", created_at) VALUES '
+                    + ', '.join(prediction_rows)
+                )
+            )
+            conn.execute(
+                text(
+                    'INSERT INTO sientia_data.laborious_data '
+                    '(model_id, variable, value, "timestamp", created_at) VALUES '
+                    + ', '.join(target_rows)
+                )
+            )
+    return timestamps
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_simple_metrics_happy_path_persists_all_metrics_and_columns(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_simple_metrics: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario S.1.1: rmse/mse/mae/r2 are calculated from a deterministic
+    prediction/target pair set and written one row per metric. Every column
+    expected by ``sientia_data.simple_metrics`` must be populated (except the
+    nullable ``timestamp`` column) and the numerical values must match
+    closed-form expectations.
+    """
+    client = temporal_test_env.client
+    model_id = 511
+
+    pairs = [
+        (1.0, 2.0),
+        (2.0, 4.0),
+        (3.0, 5.0),
+        (4.0, 9.0),
+        (5.0, 12.0),
+    ]
+    diffs = [target - prediction for prediction, target in pairs]
+    n = len(diffs)
+    expected_rmse = math.sqrt(sum(d * d for d in diffs) / n)
+    expected_mse = sum(d * d for d in diffs) / n
+    expected_mae = sum(abs(d) for d in diffs) / n
+    target_mean = sum(t for _, t in pairs) / n
+    ss_res = sum((target - prediction) ** 2 for prediction, target in pairs)
+    ss_tot = sum((t - target_mean) ** 2 for _, t in pairs)
+    expected_r2 = 1.0 - (ss_res / ss_tot)
+
+    _seed_predictions_and_targets(postgres_engine, model_id=model_id, pairs=pairs)
+
+    input_data = _simple_metrics_input(model_id)
+    await start_and_await_workflow(
+        client,
+        SimpleMetrics.run,
+        input_data,
+        make_workflow_id('test-simple-metrics-happy'),
+    )
+
+    with postgres_engine.connect() as conn:
+        rows = (
+            conn.execute(
+                text(
+                    'SELECT * FROM sientia_data.simple_metrics '
+                    'WHERE model_id = :m ORDER BY metric'
+                ),
+                {'m': str(model_id)},
+            )
+            .mappings()
+            .all()
+        )
+
+    assert len(rows) == 4, f'Expected 4 metric rows, got {len(rows)}'
+    for column in EXPECTED_SIMPLE_METRICS_COLUMNS:
+        assert column in rows[0], f'Missing simple_metrics column: {column}'
+    for row in rows:
+        for column in EXPECTED_SIMPLE_METRICS_COLUMNS:
+            if column in NULLABLE_SIMPLE_METRICS_COLUMNS:
+                continue
+            assert row[column] is not None, f"Column '{column}' is NULL in {dict(row)}"
+
+    by_metric = {row['metric']: row for row in rows}
+    assert set(by_metric) == {'rmse', 'mse', 'mae', 'r2'}
+
+    def _decimal_close(actual, expected, places: int = 6) -> bool:
+        return abs(float(actual) - expected) < 10 ** (-places)
+
+    assert _decimal_close(by_metric['rmse']['value'], expected_rmse)
+    assert _decimal_close(by_metric['mse']['value'], expected_mse)
+    assert _decimal_close(by_metric['mae']['value'], expected_mae)
+    assert _decimal_close(by_metric['r2']['value'], expected_r2)
+
+    assert all(row['data_size'] == n for row in rows), 'data_size must equal target row count'
+    assert all(row['interval_minutes'] == 60 for row in rows)
+    # ``model_id`` is now ``text`` in the new DDL, so we compare with the
+    # stringified test id rather than the numeric value.
+    assert all(row['model_id'] == str(model_id) for row in rows)
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_simple_metrics_subset_metrics_writes_only_requested_rows(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_simple_metrics: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario S.1.2: Requesting ``metrics=['rmse']`` must persist exactly one row
+    with metric ``rmse`` and skip mse/mae/r2.
+    """
+    client = temporal_test_env.client
+    model_id = 512
+
+    pairs = [(1.0, 2.0), (2.0, 4.0), (3.0, 6.0)]
+    _seed_predictions_and_targets(postgres_engine, model_id=model_id, pairs=pairs)
+
+    input_data = _simple_metrics_input(model_id, metrics=['rmse'])
+    await start_and_await_workflow(
+        client,
+        SimpleMetrics.run,
+        input_data,
+        make_workflow_id('test-simple-metrics-subset'),
+    )
+
+    with postgres_engine.connect() as conn:
+        metrics = [
+            r[0]
+            for r in conn.execute(
+                text(
+                    'SELECT metric FROM sientia_data.simple_metrics '
+                    'WHERE model_id = :m'
+                ),
+                {'m': str(model_id)},
+            ).all()
+        ]
+    assert metrics == ['rmse']
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_simple_metrics_zero_variance_target_returns_zero_r2(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_simple_metrics: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario S.2.1: When the target column has zero variance the activity must
+    return ``r2 = 0`` (division-by-zero guard) and still persist all four metrics.
+    """
+    client = temporal_test_env.client
+    model_id = 521
+
+    pairs = [(0.0, 5.0), (1.0, 5.0), (2.0, 5.0), (3.0, 5.0)]
+    _seed_predictions_and_targets(postgres_engine, model_id=model_id, pairs=pairs)
+
+    input_data = _simple_metrics_input(model_id)
+    await start_and_await_workflow(
+        client,
+        SimpleMetrics.run,
+        input_data,
+        make_workflow_id('test-simple-metrics-zero-variance'),
+    )
+
+    with postgres_engine.connect() as conn:
+        r2_value = conn.execute(
+            text(
+                "SELECT value FROM sientia_data.simple_metrics "
+                "WHERE model_id = :m AND metric = 'r2'"
+            ),
+            {'m': str(model_id)},
+        ).scalar()
+    assert r2_value is not None
+    assert Decimal(str(r2_value)) == Decimal('0'), f'expected r2=0, got {r2_value!r}'
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_simple_metrics_no_overlapping_data_short_circuits(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker_simple_metrics: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario S.3.1: When the join produces no rows (no matching laborious_data
+    row for the configured ``target``), the workflow returns early without
+    invoking ``calculate_simple_metrics`` and writes nothing.
+    """
+    client = temporal_test_env.client
+    model_id = 531
+
+    # Insert predictions but no matching target rows for the configured variable.
+    _seed_predictions_and_targets(
+        postgres_engine,
+        model_id=model_id,
+        pairs=[(1.0, 1.0)],
+        target_name='wrong_variable_name',
+    )
+
+    input_data = _simple_metrics_input(model_id)
+    await start_and_await_workflow(
+        client,
+        SimpleMetrics.run,
+        input_data,
+        make_workflow_id('test-simple-metrics-empty-join'),
+    )
+
+    with postgres_engine.connect() as conn:
+        count = conn.execute(
+            text(
+                'SELECT COUNT(*) FROM sientia_data.simple_metrics '
+                'WHERE model_id = :m'
+            ),
+            {'m': str(model_id)},
+        ).scalar()
+    assert count == 0, 'Empty target data must short-circuit and skip persistence'

git-requirements-mapping.txt

@@ -1,2 +1,2 @@
-git+ssh://git@github.com/Aignosi/sientia-dataops-library.git:sientia-do
+git+ssh://git@github.com/Aignosi/sientia-dataops-library.git:sientia_do
-git+ssh://git@github.com/Aignosi/sientia-mlops-library.git:sientia
+git+ssh://git@github.com/Aignosi/sientia-model-library.git:sientia_model

input_sample.json

@@ -0,0 +1,143 @@
+{
+  "models": [
+    {
+      "id": "1001",
+      "name": "test-runtime",
+      "active": false,
+      "model_config": {
+        "alias": "production",
+        "retention_minutes": 60,
+        "target": "Square"
+      }
+    }
+  ],
+  "pipelines": [
+    {
+      "schedule_name": "laborious-test-runtime",
+      "model_id": "1001",
+      "workflow_type": "predictions_batch",
+      "frequency": "60s",
+      "max_retry_policy": 1,
+      "query": "select * from sientia_data.laborious_data where model_id = 1 and \"timestamp\" > NOW() - INTERVAL '5 minutes' order by \"timestamp\" desc limit 30;",
+      "retention_time": 60,
+      "write_tags": [],
+      "input_filters": [
+        {
+          "filter_name": "EMPTY_DATA",
+          "policy": "STOP"
+        },
+        {
+          "filter_name": "SPECIFIC_VARIABLES_NULL_VALUES",
+          "policy": "CONTINUE",
+          "config": {
+            "variables": [
+              "Counter"
+            ]
+          }
+        }
+      ],
+      "mlflow_transform_filters": [
+        {
+          "filter_name": "API_ERROR",
+          "policy": "REPEAT"
+        },
+        {
+          "filter_name": "NAN_VALUES",
+          "policy": "STOP"
+        }
+      ],
+      "mlflow_predict_filters": [
+        {
+          "filter_name": "API_ERROR",
+          "policy": "CONTINUE"
+        }
+      ],
+      "path_priority": [
+        "STOP",
+        "CONTINUE",
+        "REPEAT"
+      ],
+      "active": true,
+      "updated_at": {
+        "$date": "2026-05-07T23:35:01.600Z"
+      },
+      "save_transform": false,
+      "pi_web_api_output_config": {},
+      "datetime_columns": [
+        "timestamp",
+        "created_at"
+      ]
+    },
+    {
+      "schedule_name": "minimal-retrain-test-runtime",
+      "model_id": "1001",
+      "model_name": "test-runtime",
+      "workflow_type": "minimal_retrain",
+      "frequency": "1h",
+      "max_retry_policy": 1,
+      "query": "select * from sientia_data.laborious_data where model_id = 1 and \"timestamp\" > NOW() - INTERVAL '60 minutes' order by \"timestamp\" desc;",
+      "schema": "sientia_data",
+      "table_name": "log_retrain",
+      "datetime_columns": ["timestamp", "created_at"],
+      "model_config": {
+        "target": "Square"
+      },
+      "active": true,
+      "updated_at": {
+        "$date": "2026-05-07T23:35:01.600Z"
+      }
+    },
+    {
+      "schedule_name": "drift-test-runtime",
+      "model_id": "1001",
+      "model_name": "test-runtime",
+      "workflow_type": "drift",
+      "frequency": "5m",
+      "offset": "2m",
+      "max_retry_policy": 1,
+      "execution_timeout_seconds": 300,
+      "task_timeout_seconds": 300,
+      "interval": 5,
+      "drift_metrics": [
+        "kolmogorov_smirnov",
+        "jensen_shannon",
+        "wasserstein"
+      ],
+      "chunk_period": "min",
+      "schema": "sientia_data",
+      "source_table_name": "laborious_data",
+      "target_table_name": "drift_metrics",
+      "model_config": {
+        "target": "Square"
+      },
+      "active": true,
+      "updated_at": {
+        "$date": "2026-05-07T23:35:01.600Z"
+      }
+    },
+    {
+      "schedule_name": "simple-metrics-test-runtime",
+      "model_id": "1001",
+      "model_name": "test-runtime",
+      "workflow_type": "simple_metrics",
+      "frequency": "5m",
+      "offset": "2m",
+      "max_retry_policy": 1,
+      "execution_timeout_seconds": 300,
+      "task_timeout_seconds": 300,
+      "interval_minutes": 5,
+      "metrics": ["rmse", "mse", "mae", "r2"],
+      "schema": "sientia_data",
+      "predictions_table_name": "predictions",
+      "data_table_name": "laborious_data",
+      "target_table_name": "simple_metrics",
+      "model_config": {
+        "target": "Square"
+      },
+      "active": true,
+      "updated_at": {
+        "$date": "2026-05-18T23:35:01.600Z"
+      }
+    }
+  ]
+}

inter_arrival.py

@@ -0,0 +1,25 @@
+# %%
+
+# Load logs.txt
+with open('logs.txt', 'r') as file:
+    lines = file.readlines()
+
+# %%
+import re
+# Grep "inter-arrival_s=number" with regex
+intervals = []
+for line in lines:
+    match = re.search(r'inter-arrival_s=([0-9.]+)', line)
+    if match:
+        intervals.append(float(match.group(1)))
+# %%
+
+print(intervals)
+# %%
+import matplotlib.pyplot as plt
+plt.plot(intervals)
+plt.ylabel('Inter-arrival time (s)')
+plt.xlabel('Sample')
+plt.title('Inter-arrival time distribution')
+plt.show()
+# %%

laborious-temporal-plugin-store-migration-plan.md

@@ -0,0 +1,305 @@
+---
+tags:
+  - engineering
+  - sientia
+  - runtime-system
+  - laborious-temporal
+  - plugin-store
+  - migration-plan
+created: 2026-03-02
+modified: 2026-03-02
+created_by: Vitor Pimentel
+modified_by: Vitor Pimentel
+status: draft
+---
+
+# Sientia Laborious Temporal — PluginStore & Wrapper Migration Plan
+
+> Migration plan for evolving `sientia-dataops-laborious_temporal` from direct MLflow model loading to a runtime-aware architecture that uses Sientia model wrappers (`SientiaModel`) via their public methods, aligned with the runtime strategy.
+
+## Summary
+
+1. [[#Objectives and Scope|Objectives and Scope]] — What this migration must achieve
+2. [[#Existing State Overview (laborious_temporal)|Existing State Overview]] — Current responsibilities and coupling points
+3. [[#Requirements Mapping|Requirements Mapping]] — Functional and non-functional requirements
+4. [[#Target Architecture|Target Architecture]] — Desired runtime and model interaction architecture
+5. [[#Implementation Plan|Implementation Plan]] — Phased, detailed changes to apply
+6. [[#Testing Strategy|Testing Strategy]] — How to validate the new behavior
+7. [[#Rollout and Migration Strategy|Rollout and Migration Strategy]] — How to safely roll out the changes
+8. [[#Related Documents|Related Documents]] — Cross-links to supporting documents
+
+---
+
+## Objectives and Scope
+
+This migration focuses on the `sientia-dataops-laborious_temporal` application and aims to:
+
+- Keep the **runtime-aware deployment model** consistent with the rest of the runtime system (Helm + `RUNTIME` env var, runtime installation via PluginStore).
+- Ensure that **all interactions with models use the public methods of the Sientia wrapper** (`SientiaModel`):
+  - Use `SientiaModel.train(...)` and `retrain(...)` for training and retraining flows.
+  - Use `SientiaModel.predict(...)` and `SientiaModel.transform(...)` for inference and preprocessing.
+- **Use the shared MLflow repository** (`SientiaMLflowRepository`) for all MLflow operations (load, runs, artifacts, promotion, production lookup, metadata logging); do not implement these in Laborious.
+
+Out of scope:
+
+- Replacing MLflow as the tracking and registry backend.
+- Redesigning Temporal workflows (queues, retry policies) beyond what is required for the new model interaction style.
+
+---
+
+## Existing State Overview (laborious_temporal)
+
+Key components in `sientia-dataops-laborious_temporal`:
+
+- **MLflow activities** (`laborious/activities/mlflow.py`)
+  - `MLFlow` class exposes Temporal activities for:
+    - `request_transform` — loads transformation models from MLflow and applies them to input data.
+    - `request_predict` — loads predictive models from MLflow and generates predictions.
+    - `retrain_model` — orchestrates retraining using historical data stored in MinIO and MLflow registry.
+    - `update_production_model` — promotes new versions to production.
+    - `get_reference_data` — fetches evaluation/reference datasets from model artifacts.
+  - These activities delegate ML-specific work to `MLFlowRepository`.
+
+- **MLflow repository** (`laborious/utils/repository/model_repository.py`)
+  - `MLFlowRepository` encapsulates the interaction with MLflow:
+    - Model discovery and run resolution (`get_model_run_id`, `get_model_uri`, `get_experiment`, etc.).
+    - Artifact download and loading for both transformer and prediction models.
+    - Model caching and retention (`get_model`, `get_cached_operation`).
+    - Transformation and prediction entry points:
+      - `transform(...)` wraps `get_cached_operation(..., operation='transform')`.
+      - `predict(...)` wraps `get_cached_operation(..., operation='predict')`.
+    - Retraining orchestration (`fit_models`, `create_new_experiment`, `retrain_model`, `update_production_model`).
+  - Today:
+    - Models are loaded via MLflow flavors: sklearn, pyfunc, pytorch.
+    - When `flavor == 'pyfunc'` and `load_wrapper=True`, the repository loads a wrapper via:
+      - `raw_model = mlflow.pyfunc.load_model(artifact_path)`
+      - `model = raw_model._model_impl.python_model`
+    - Production models are resolved using **stages** in the Model Registry (for example, selecting the latest version in stage `Production`); **aliases such as `@production` are not used yet**, and models are registered explicitly as part of the current retrain/promotion flows.
+    - The wrapper’s `_model_impl` class does not extend `SientiaModel`.
+    - All this MLflow-specific logic is local to `laborious_temporal` and partially duplicated in `sientia-dataops-model-manager`, which motivates the extraction of a shared MLflow repository in `sientia-dataops-library` (see `mlflow-shared-repository-migration-plan`).
+
+**MLflow:** All MLflow ops → [[mlflow-shared-repository-migration-plan|shared repository]]. Laborious uses the interface; `SientiaModel` lifecycle is in sientia-model-library.
+
+### Current vs Target — High-level Flow
+
+```mermaid
+flowchart LR
+    subgraph current [Current State — Laborious Temporal]
+        direction TB
+        TemporalWorker["Temporal Worker"]
+        MlflowActivities["MLFlow Activities\nrequest_transform / request_predict / retrain_model"]
+        MLFlowRepositoryNode["MLFlowRepository"]
+        MLflowRegistry["MLflow Tracking + Registry"]
+        RawModel["Loaded Model\n(sklearn / pyfunc / pytorch)"]
+
+        TemporalWorker -->|"start workflow\n(Temporal)"| MlflowActivities
+        MlflowActivities -->|"call transform()/predict()/retrain_model()"| MLFlowRepositoryNode
+        MLFlowRepositoryNode -->|"search_model_versions()\ncurrent_stage == 'Production'"| MLflowRegistry
+        MLFlowRepositoryNode -->|"mlflow.*.load_model(model_uri)"| RawModel
+        MLFlowRepositoryNode -->|"raw_model.predict(data)\nraw_model.fit(data)"| RawModel
+    end
+
+    subgraph target [Target State — Laborious Temporal]
+        direction TB
+        TemporalWorker2["Temporal Worker"]
+        MlflowActivities2["MLFlow Activities\n(same APIs)"]
+        MLFlowRepositoryNode2["MLFlowRepository\n(wrapper-aware)"]
+        MLflowRegistry2["MLflow Tracking + Registry\n(aliases enabled)"]
+        SientiaWrapperNode["Wrapper Instance\n(extends SientiaModel)"]
+
+        TemporalWorker2 -->|"start workflow\n(Temporal)"| MlflowActivities2
+        MlflowActivities2 -->|"call transform()/predict()/retrain_model()"| MLFlowRepositoryNode2
+        MLFlowRepositoryNode2 -->|"get_model_version_by_alias('production')\n& models:/name@production"| MLflowRegistry2
+        MLFlowRepositoryNode2 -->|"mlflow.pyfunc.load_model(...)"| SientiaWrapperNode
+        MLFlowRepositoryNode2 -->|"wrapper.transform(...)\nwrapper.predict(...)\nwrapper.train()/retrain()"| SientiaWrapperNode
+        SientiaWrapperNode -->|"store_model(...)\n(auto-register + update alias)"| MLflowRegistry2
+    end
+```
+
+### Current vs Target — Retrain Hot Path (Code Sketch)
+
+Current retrain flow inside `MLFlowRepository.fit_models` / `retrain_model` (simplified):
+
+```python
+data_model, _ = await self.download_model(
+    model_name=model_name,
+    metadata=metadata,
+    model_type="transform",
+    flavor=transform_flavor,
+    load_wrapper=(transform_flavor == "pyfunc"),
+)
+
+prediction_model, _ = await self.download_model(
+    model_name=model_name,
+    metadata=metadata,
+    model_type="predict",
+    flavor=predict_flavor,
+    load_wrapper=(predict_flavor == "pyfunc"),
+)
+
+treated_data_candidate = data_model.fit(data)          # or data_model.predict(data)
+...
+prediction_model.fit(retrain_dataset)                  # direct fit on underlying model
+```
+
+Target retrain flow when using wrappers that extend `SientiaModel`:
+
+```python
+wrapper, _ = await self.download_model(
+    model_name=model_name,
+    metadata=metadata,
+    # model_type="predict",              # wrapper owns both transformer + model stack
+    # flavor="pyfunc", flavor will always be pyfunc, this parameter will be removed
+    # load_wrapper=True, wrapper will always be loaded from _model_impl
+)
+
+# First-time training or full retrain using public API
+wrapper.train(
+    train_data=train_df,               # features + target
+    val_data=val_df,
+    target=target_name,
+)
+
+# Incremental retrain (when appropriate)
+wrapper.retrain(full_retrain_df)
+
+transformed_df, trans_meta = wrapper.transform(raw_df)
+pred_df, pred_meta = wrapper.predict({}, transformed_df, params={})
+```
+
+---
+
+## Requirements Mapping
+
+### Functional Requirements
+
+| ID    | Requirement                                        | Description                                                                                      | Impacted Areas                                         |
+| ----- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------ |
+| FR-01 | Runtime detection and installation                 | Align worker startup with `RUNTIME` and runtime installation via PluginStore                     | Worker                                                 |
+| FR-02 | Wrapper‑based training using public API            | Retraining must call the public `retrain(...)` method of `SientiaModel`                          | `fit_models`, `retrain_model` paths                    |
+| FR-03 | Wrapper‑based inference using public API           | Inference must call `predict(...)` and `transform(...)` on the wrapper; obtain wrappers via `SientiaMLflowRepository` | Activities, shared repository                          |
+| FR-04 | Use shared MLflow repository                       | Delegate all MLflow operations (load, runs, artifacts, promotion, production lookup) to `SientiaMLflowRepository`; do not implement in Laborious | [[mlflow-shared-repository-migration-plan]]             |
+| FR-05 | Model configuration for training/retraining        | `model_config` must define `target` and `retention_minutes`; all models are pyfunc + wrapper (no flavor selection) | `model_config` structures                              |
+
+### Non-Functional Requirements
+
+| ID     | Requirement                      | Description                                                                                                              | Impacted Areas                          |
+| ------ | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------- |
+| NFR-01 | Consistent public API usage      | All wrapper interactions must go through public `SientiaModel` methods; metadata logging is handled by the shared repository | Activities                              |
+| NFR-02 | Fail fast when wrappers unavailable | Where wrappers are not yet available, raise an explicit error requesting model update                                     | Shared repository, config               |
+| NFR-03 | Testability                      | Enable unit tests to validate wrapper-based flows and integration with shared repository                                  | `tests/laborious`                       |
+| NFR-04 | Operational safety               | Retraining and promotion behavior must remain auditable and robust                                                       | Retrain & promotion flows               |
+
+---
+
+## Target Architecture
+
+### Wrapper-centric model interactions
+
+The target state for `laborious_temporal` is:
+
+- All training and retraining logic goes through the wrapper’s public methods: `train(...)` and `retrain(...)`.
+- Inference and transformation use `transform(df)` and `predict(context, df, params)`.
+- All MLflow operations (load, runs, artifacts, promotion, production lookup) go through `SientiaMLflowRepository` (see [[mlflow-shared-repository-migration-plan]]).
+
+### Use of Shared MLflow Repository
+
+Laborious obtains wrappers and performs all MLflow operations via `SientiaMLflowRepository`. The shared repository (see [[mlflow-shared-repository-migration-plan]]) owns: alias-based URIs, pyfunc loading, wrapper extraction, promotion, runs, artifacts, and metadata logging. Laborious activities call `repo.load_wrapper(...)` and then the wrapper’s public methods (`transform`, `predict`, `train`, `retrain`); they do not implement MLflow logic.
+
+---
+
+## Implementation Plan
+
+### Phase 0 — Design and Configuration Alignment
+
+- **P0-01**: Catalog model types and flavors used by `laborious_temporal`:
+  - For each active model:
+    - Flavor will be always pyfunc
+    - Whether a `_model_impl` wrapper is already present and extends `SientiaModel`.
+- **P0-02**: Define configuration fields in `model_config` for wrapper usage:
+  - Example:
+    - `target` field for training/retraining (already partially present).
+    - `retention_minutes` field for model retention in minutes (unchanged).
+
+### Phase 1 — Inference via SientiaModel Public API (using shared repository)
+
+- **P1-01**: Replace `download_model` with `SientiaMLflowRepository.load_wrapper(...)`; remove local MLflow loading logic.
+- **P1-02**: Update `get_cached_operation` to call `wrapper.transform(data)` and `wrapper.predict({}, data)`; unpack returned metadata; metadata logging is handled by the shared repository.
+- **P1-03**: Ensure activities (`request_transform`, `request_predict`) remain unchanged externally (inputs/outputs unchanged).
+
+### Phase 2 — Retraining via SientiaModel.retrain
+
+- **P2-01**: Refactor `fit_models` to call `wrapper.retrain(data)` and `wrapper.train(...)`; use `SientiaMLflowRepository` for runs, metrics, artifacts, and promotion (no local MLflow logic).
+
+### Phase 3 — MLflow Logging and Promotion (via shared repository)
+
+- **P3-01**: In `create_new_experiment`, use the wrapper’s `store_model(...)` for model artifacts; use `SientiaMLflowRepository` for runs, metrics, and any additional MLflow operations.
+- **P3-02**: Use `SientiaMLflowRepository.promote_to_alias(...)` for production promotion; model registration is auto-handled by wrappers.
+
+### Phase 4 — Runtime Alignment
+
+- **P4-01**:`laborious_temporal` is also deployed via the runtime-aware Helm chart:
+  - Read `RUNTIME` env var.
+  - Install runtime via PluginStore before starting Temporal workers.
+- **P4-02**: Standardize worker queue name as `{project_name}-{runtime}-queue`.
+- **P4-03**: Fix quality pipelines to a single runtime (to be decided).
+
+---
+
+## Testing Strategy
+
+- **T1 — Unit tests**
+  - Add tests in `tests/laborious/utils/repository/test_model_repository.py` to cover:
+    - Wrapper-based `get_cached_operation` for both `transform` and `predict` (using shared repository).
+    - Wrapper-based `fit_models` and `retrain_model` paths calling `train` and `retrain` respectively.
+  - Code coverage must be 100%.
+- **T2 — Integration tests**
+  - Use existing end-to-end tests under `e2e/`:
+    - Configure a model with `SientiaModel` wrapper (loaded via shared repository).
+    - Run full prediction and retrain workflows; compare predictions, retrain outcomes, and MLflow artifacts.
+
+---
+
+## Rollout and Migration Strategy
+
+- **R1 — Create models with the new architecture**
+  - Update or create models in the modeling pipeline so they:
+    - Use wrappers that extend `SientiaModel`.
+    - Correctly implement `train`, `retrain`, `predict`, `transform`, and `store_model`.
+    - Produce structured metadata in `transform_meta` and `pred_meta`.
+  - Publish these models to a test store (or dedicated branch/experiment) for initial validation.
+
+- **R2 — Provision runtimes for the new models**
+  - Configure and install dedicated runtimes for the new models:
+    - Ensure runtime dependencies (Python and system libraries) are available via PluginStore/runtime installer.
+    - Validate that each runtime can:
+      - Load the wrapper through MLflow.
+      - Execute `transform` and `predict` end-to-end on sample data.
+
+- **R3 — Run new models in real workflows**
+  - Integrate the new wrapper-based models into real Laborious workflows, initially in non-critical environments:
+    - Route only a subset of flows or entities to the new models.
+    - Monitor logs (including metadata), business metrics, and retraining/promotion behavior.
+  - Promote these models to production using aliases (`@production`) in MLflow 3+.
+
+- **R4 — Migrate remaining models progressively**
+  - Define migration waves by model family:
+    - For each existing model:
+      - Create or adapt a `SientiaModel` wrapper.
+      - Provision the corresponding runtime.
+      - Execute the test cycle (T1–T2) from the previous section.
+      - Update aliases so production traffic uses the new wrapper.
+  - After all models are migrated:
+    - Remove legacy stage-based paths (`Production`) and non-wrapper models.
+    - Simplify the codebase to assume wrappers + aliases only.
+
+---
+
+## Related Documents
+
+- [[mlflow-shared-repository-migration-plan|MLflow Shared Repository Migration Plan]] — Concepts implemented in the common interface (production lookup, wrapper loading, promotion, metadata logging, etc.)
+- [[model-manager-plugin-store-migration-plan|Model Manager PluginStore Migration Plan]]
+- [[analytics-implementation-plan|Runtime Analytics Helm Implementation Plan]]
+- [[analytics|Runtime Analytics Architecture and Analysis]]
+- [[../model-plugin-system/06-end-to-end-flow|Model Plugin System — End-to-End Flow]]
+

laborious/activities/activities.py

@@ -6,7 +6,9 @@ with workflow.unsafe.imports_passed_through():
 
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
     from sientia_do.observability.logger import Logger
-    from sientia_do.repository.minio_repository import MinioRepository
+    from sientia_do.repository.minio_repository_sync import MinioRepository
+    from sientia_model.model_repository.mlflow_repository import SientiaMLflowRepository
+    from sientia_model.model_repository.plugin_store import PluginStore
 
     from laborious.activities.api import API
     from laborious.activities.gates import Gates
@@ -14,65 +16,78 @@ with workflow.unsafe.imports_passed_through():
     from laborious.activities.model_metrics import ModelMetrics
     from laborious.activities.opc import OPC
     from laborious.activities.storage import Storage
+    from laborious.utils.connectors_config import build_mlflow_config
 
 
 class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API):
     """
-    Main activities orchestrator for the Laborious system.
+    Central orchestrator for all Temporal activities used by Laborious workflows.
 
-    This class combines functionality from multiple activity classes to provide
+    Composes Storage (Postgres + MinIO offload), MLFlow (wrapper-based inference and retrain
-    a unified interface for all workflow operations. It manages database connections,
+    via ``SientiaMLflowRepository``), Gates (data quality and ML response filters), OPC exports,
-    MLFlow model interactions, data quality validation, and OPC server communications.
+    drift/simple metrics, and PI Web API writes. The worker constructs one ``Activities`` instance
+    per process and registers its callables on multiple workers bound to different task queues.
 
-    The class implements multiple inheritance to combine specialized functionality:
+    MLflow connectivity: unless ``mlflow_repository`` is injected (tests only), this class builds
-    - Storage: Database operations and data persistence
+    ``SientiaMLflowRepository`` from ``build_mlflow_config()`` so tracking credentials and URL
-    - MLFlow: Model inference and transformation operations
+    stay aligned with the rest of Laborious env-based configuration.
-    - Gates: Data quality validation and filtering mechanisms
-    - OPC: Real-time data export to OPC servers
-    - ModelMetrics: Model performance metrics and drift detection
-    - API: PI Web API export operations for industrial systems
 
     Attributes:
-        postgres_config (dict): PostgreSQL connection configuration
+        Inherits and exposes behaviour from mixins; the MLFlow mixin holds ``mlflow_repository``
-        mlflow_config (dict): MLFlow server configuration
+        and ``plugin_store`` after ``__init__``.
-        opc_config (dict): OPC server configuration
-        pi_web_api_config (dict): PI Web API server configuration
-        logger (Logger): Logging and observability instance
-        notification_handler (NotificationHandler): Notification management instance
     """
 
     def __init__(
         self,
         postgres_config: dict[str, Any],
-        mlflow_config: dict[str, Any],
+        plugin_store: PluginStore,
         minio_config: dict[str, Any],
         opc_config: dict[str, Any],
         pi_web_api_config: dict[str, Any],
         logger: Logger,
         notification_handler: NotificationHandler,
+        metrics_controller: MetricsController | None = None,
+        mlflow_repository: SientiaMLflowRepository | None = None,
     ):
         """
-        Initialize the Activities orchestrator with all required configurations.
+        Wire Postgres, MinIO, MLflow, OPC, gates, metrics, and PI Web API into a single object.
 
-        This constructor initializes all parent classes with their respective
+        A single ``MetricsController`` instance is created (or reused) and passed to MinIO,
-        configurations and sets up the foundation for all activity operations.
+        MLflow repository, and all mixins so Prometheus and SDK metrics stay consistent.
 
         Args:
-            postgres_config: PostgreSQL connection configuration dictionary
+         - postgres_config: Host, port, credentials, db name, and pool bounds for Storage.
-                Required keys: host, port, user, password, dbname, min_connections, max_connections
+         - plugin_store: ``PluginStore`` instance; the worker must call ``install_runtime`` before
-            mlflow_config: MLFlow server configuration dictionary
+           activities run so wrapper code is importable.
-                Required keys: host, port, username, password
+         - minio_config: Endpoint, keys, bucket, retention, and TLS flag for object storage payloads.
-            opc_config: OPC server configuration dictionary
+         - opc_config: Map of OPC server id to connection settings for ``OPC`` mixin.
-                Can contain multiple server configurations
+         - pi_web_api_config: Base URL and auth for ``API`` mixin.
-            pi_web_api_config: PI Web API server configuration dictionary
+         - logger: Structured logger used across all activities.
-                Required keys: base_url, auth_type, auth_token
+         - notification_handler: Handler for alerts and persisted notifications.
-            logger: Logger instance for observability and debugging
+         - metrics_controller: Optional shared controller; if ``None``, a new one is created.
-            notification_handler: Notification handler for alerts and monitoring
+         - mlflow_repository: Optional ``SientiaMLflowRepository`` for unit/e2e tests; in production
+           leave unset so the repository is built from environment via ``build_mlflow_config()``.
 
         Raises:
-            Exception: If any parent class initialization fails
+            Exception: If any parent ``__init__`` fails (e.g. invalid config keys).
+
+        Return:
+            None
         """
-        metrics_controller = MetricsController(logger=logger)
+
+        mc = metrics_controller or MetricsController(logger=logger)
+
+        # Production path: one shared MLflow client for all model registry / tracking calls.
+        if mlflow_repository is None:
+            mlflow_cfg = build_mlflow_config()
+            mlflow_repository = SientiaMLflowRepository(
+                host=mlflow_cfg['url'],
+                username=mlflow_cfg['username'],
+                password=mlflow_cfg['password'],
+                logger=logger,
+                notification_handler=notification_handler,
+                metrics_controller=mc,
+            )
 
         minio_repository = MinioRepository(
             endpoint=minio_config['endpoint_url'],
@@ -81,11 +96,10 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API):
             bucket=minio_config['default_bucket'],
             logger=logger,
             notification_handler=notification_handler,
-            metrics_controller=metrics_controller,
+            metrics_controller=mc,
             secure=minio_config['secure'],
         )
 
-        # Initialize parent classes
         Storage.__init__(
             self,
             host=postgres_config['host'],
@@ -99,19 +113,17 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API):
             minio_repository=minio_repository,
             logger=logger,
             notification_handler=notification_handler,
-            metrics_controller=metrics_controller,
+            metrics_controller=mc,
         )
 
         MLFlow.__init__(
             self,
-            mlflow_host=mlflow_config['host'],
+            mlflow_repository=mlflow_repository,
-            mlflow_port=mlflow_config['port'],
+            plugin_store=plugin_store,
-            mlflow_username=mlflow_config['username'],
-            mlflow_password=mlflow_config['password'],
             minio_repository=minio_repository,
             logger=logger,
             notification_handler=notification_handler,
-            metrics_controller=metrics_controller,
+            metrics_controller=mc,
         )
 
         Gates.__init__(
@@ -119,7 +131,7 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API):
             minio_repository=minio_repository,
             logger=logger,
             notification_handler=notification_handler,
-            metrics_controller=metrics_controller,
+            metrics_controller=mc,
         )
 
         OPC.__init__(
@@ -127,14 +139,14 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API):
             opc_servers=opc_config,
             logger=logger,
             notification_handler=notification_handler,
-            metrics_controller=metrics_controller,
+            metrics_controller=mc,
         )
 
         ModelMetrics.__init__(
             self,
             logger=logger,
             notification_handler=notification_handler,
-            metrics_controller=metrics_controller,
+            metrics_controller=mc,
         )
 
         API.__init__(
@@ -144,26 +156,22 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API):
             auth_token=pi_web_api_config['auth_token'],
             logger=logger,
             notification_handler=notification_handler,
-            metrics_controller=metrics_controller,
+            metrics_controller=mc,
         )
 
-    async def shutdown(self):
+    def shutdown(self) -> None:
         """
-        Gracefully shutdown all activities and clean up resources.
+        Close database pools, sync clients, and OPC sessions in a defined order.
 
-        This method ensures proper cleanup of all resources including:
+        Should be invoked on worker exit so connection pools and OPC sessions are released
-        - PostgreSQL connection pools
+        cleanly before process termination.
-        - OPC server connections
-        - PI Web API client connections
-        - MLFlow model repositories
-        - Any other resources that need explicit cleanup
 
-        The method should be called before the application terminates to ensure
+        Return:
-        proper resource cleanup and prevent resource leaks.
+            None
         """
         Storage.close(self)
         MLFlow.close(self)
         Gates.close(self)
-        await OPC.close(self)
+        OPC.close(self)
         ModelMetrics.close(self)
         API.close(self)

laborious/activities/api.py

@@ -11,7 +11,7 @@ with workflow.unsafe.imports_passed_through():
     from sientia_do.observability.logger import Logger
     from sientia_do.observability.metrics_controller import MetricsController
     from sientia_do.observability.sientia_monitoring import SientiaMonitoring
-    from sientia_do.repository.pi_web_api_client import PIWebAPIClient
+    from sientia_do.repository.pi_web_api_client_sync import PIWebAPIClient
 
     from laborious import metrics
 
@@ -105,7 +105,7 @@ class API(SientiaMonitoring):
         self.pi_web_api_client.close()
         SientiaMonitoring.shutdown(self)
 
-    async def process_pi_web_api_response(
+    def process_pi_web_api_response(
         self,
         response_data: list[dict[str, Any]],
         tags: dict[str, str],
@@ -156,7 +156,7 @@ class API(SientiaMonitoring):
                 self.error(
                     f'Error writing tag {tag_name}:{web_id} to PI Web API: {errors}', metadata
                 )
-                await self.emit_metric(
+                self.emit_metric_sync(
                     metric_object=metrics.PI_WEB_API_PREDICTION_WRITTEN_ERROR_COUNT,
                     tags={
                         **core_labels,
@@ -165,7 +165,7 @@ class API(SientiaMonitoring):
                 )
                 confidence = PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
             else:
-                await self.emit_metric(
+                self.emit_metric_sync(
                     metric_object=metrics.PI_WEB_API_PREDICTION_WRITTEN_COUNT,
                     tags={
                         **core_labels,
@@ -182,7 +182,7 @@ class API(SientiaMonitoring):
                 metadata,
             )
 
-            await self.send_notification_async(
+            self.send_notification(
                 metadata=metadata,
                 notification_id='WRITE_PI_WEB_API_PREDICTION_ERROR',
                 message=f'The number of written tags does not match the number of tag names: Expected {tag_names} tags, but {written_tags} tags were written.\nResponse:\n {json.dumps(response_data, indent=4)}\nTags:\n {json.dumps(tags, indent=4)}',
@@ -194,7 +194,7 @@ class API(SientiaMonitoring):
         return confidence, message
 
     @activity.defn(name='write_pi_web_api_data')
-    async def write_pi_web_api_data(self, input_data: dict[str, Any]) -> dict[Any, Any]:
+    def write_pi_web_api_data(self, input_data: dict[str, Any]) -> dict[Any, Any]:
         """
         Write prediction and confidence data to PI Web API.
 
@@ -232,7 +232,7 @@ class API(SientiaMonitoring):
         confidence_value = data.head(1)['prediction_confidence'].values[0]
 
         try:
-            prediction_response = await self.pi_web_api_client.write_value(
+            prediction_response = self.pi_web_api_client.write_value(
                 web_ids=prediction_tags,
                 value={
                     'Timestamp': data.head(1)['timestamp'].values[0],
@@ -241,7 +241,7 @@ class API(SientiaMonitoring):
                 metadata=metadata,
             )
 
-            confidence, message = await self.process_pi_web_api_response(
+            confidence, message = self.process_pi_web_api_response(
                 response_data=prediction_response,
                 tags=raw_prediction_tags,
                 core_labels=core_labels,
@@ -258,7 +258,7 @@ class API(SientiaMonitoring):
 
         except Exception as e:
             trace = traceback.format_exc()
-            await self.send_notification_async(
+            self.send_notification(
                 metadata=metadata,
                 notification_id='WRITE_PI_WEB_API_PREDICTION_ERROR',
                 message=f'Error writing prediction data to PI Web API: {e}\n Tags: {raw_prediction_tags}',
@@ -275,7 +275,7 @@ class API(SientiaMonitoring):
             return data.to_dict()
 
         try:
-            confidence_response = await self.pi_web_api_client.write_value(
+            confidence_response = self.pi_web_api_client.write_value(
                 web_ids=confidence_tags,
                 value={
                     'Timestamp': data.head(1)['timestamp'].values[0],
@@ -284,7 +284,7 @@ class API(SientiaMonitoring):
                 metadata=metadata,
             )
 
-            await self.process_pi_web_api_response(
+            self.process_pi_web_api_response(
                 response_data=confidence_response,
                 tags=raw_confidence_tags,
                 core_labels=core_labels,
@@ -293,7 +293,7 @@ class API(SientiaMonitoring):
 
         except Exception as e:
             trace = traceback.format_exc()
-            await self.send_notification_async(
+            self.send_notification(
                 metadata=metadata,
                 notification_id='WRITE_PI_WEB_API_CONFIDENCE_ERROR',
                 message=f'Error writing confidence data to PI Web API: {e}\n Tags: {raw_confidence_tags}',

laborious/activities/gates.py

@@ -1,8 +1,5 @@
-from sientia_do.repository.minio_repository import MinioRepository
 from temporalio import activity, workflow
 
-from laborious.utils.repository.minio_manager import MinioManager
-
 with workflow.unsafe.imports_passed_through():
     import traceback
     from collections.abc import Callable, Mapping
@@ -13,6 +10,8 @@ with workflow.unsafe.imports_passed_through():
     from sientia_do.notifications.models import NotificationLevel
     from sientia_do.observability.logger import Logger
     from sientia_do.observability.metrics_controller import MetricsController
+    from sientia_do.observability.sientia_monitoring import SientiaMonitoring
+    from sientia_do.repository.minio_repository_sync import MinioRepository
     from sientia_do.utils.formatters import create_sample_dict
 
     from laborious import metrics
@@ -66,7 +65,7 @@ mlflow_content_path_confidence: Mapping[str, int] = {
 }
 
 
-class Gates(MinioManager):
+class Gates(SientiaMonitoring):
     """
     Data quality gates and filtering activities for the Laborious system.
 
@@ -106,8 +105,12 @@ class Gates(MinioManager):
         Raises:
             Exception: If BaseActivity initialization fails
         """
-        MinioManager.__init__(
+        self.minio_repository = minio_repository
-            self, minio_repository, logger, notification_handler, metrics_controller
+        SientiaMonitoring.__init__(
+            self,
+            logger=logger,
+            notification_handler=notification_handler,
+            metrics_controller=metrics_controller,
         )
 
     def close(self) -> None:
@@ -115,7 +118,12 @@ class Gates(MinioManager):
         Close the gates activity and clean up resources.
         """
 
-        MinioManager.close(self)
+        if self.minio_repository is not None:
+            try:
+                self.minio_repository.close()
+            finally:
+                self.minio_repository = None
+        SientiaMonitoring.shutdown(self)
 
     def __del__(self):
         self.close()
@@ -155,7 +163,7 @@ class Gates(MinioManager):
         return policy, filter_config
 
     @activity.defn(name='input_gate')
-    async def input_gate(self, input_data: dict[str, Any]) -> tuple[str | None, int, str]:
+    def input_gate(self, input_data: dict[str, Any]) -> tuple[str | None, int, str]:
         """
         Apply input data quality filters and validation.
 
@@ -194,7 +202,7 @@ class Gates(MinioManager):
 
         filters = input_data['filters']
         payload = MinioDataFramePayload.from_dict(input_data['data'])
-        data = await payload.retrieve(self.minio_repository, metadata)
+        data = payload.retrieve(self.minio_repository, metadata)
         path_priority = input_data['path_priority']
 
         filter_output = []
@@ -214,7 +222,7 @@ class Gates(MinioManager):
                     filter_output.append(policy)
             except Exception as e:
                 trace = traceback.format_exc()
-                await self.send_notification_async(
+                self.send_notification(
                     metadata=metadata,
                     notification_id=f'INTPUT_GATE_ERROR__{fil}',
                     message=f'Error in filter {fil}:{config}: \n {e}',
@@ -235,7 +243,7 @@ class Gates(MinioManager):
         return None, 0, ''
 
     @activity.defn(name='mlflow_response_gate')
-    async def mlflow_response_gate(self, input_data: dict[str, Any]) -> tuple[str | None, int, str]:
+    def mlflow_response_gate(self, input_data: dict[str, Any]) -> tuple[str | None, int, str]:
         """
         Validate MLFlow API response quality and integrity.
 
@@ -279,7 +287,7 @@ class Gates(MinioManager):
         self.debug(f'Filters: {filters}', metadata)
 
         payload = MinioDataFramePayload.from_dict(raw_data)
-        data = await payload.retrieve(self.minio_repository, metadata)
+        data = payload.retrieve(self.minio_repository, metadata)
 
         gate_type = input_data['type']
         path_priority = input_data['path_priority']
@@ -298,7 +306,7 @@ class Gates(MinioManager):
                 if mlflow_response_filter_functions[fil](status, filter_config):
                     filter_output.append(policy)
                     comments.append(status.get('message', 'Unknown MLFlow API error'))
-                    await self.send_notification_async(
+                    self.send_notification(
                         metadata=metadata,
                         notification_id=f'{gate_type.upper()}_GATE_RESPONSE_FILTER__{fil}',
                         message=status.get('message', 'Unknown MLFlow API error'),
@@ -308,7 +316,7 @@ class Gates(MinioManager):
                     )
             except Exception as e:
                 trace = traceback.format_exc()
-                await self.send_notification_async(
+                self.send_notification(
                     metadata=metadata,
                     notification_id=f'MLFLOW_GATE_RESPONSE_FILTER__{fil}',
                     message=f'Error in filter {fil}:{config}: \n {e}',
@@ -329,7 +337,7 @@ class Gates(MinioManager):
         return None, 0, ''
 
     @activity.defn(name='mlflow_content_gate')
-    async def mlflow_content_gate(self, input_data: dict[str, Any]) -> tuple[str | None, int, str]:
+    def mlflow_content_gate(self, input_data: dict[str, Any]) -> tuple[str | None, int, str]:
         """
         Validate MLFlow prediction content quality and integrity.
 
@@ -368,7 +376,7 @@ class Gates(MinioManager):
         filters = input_data['filters']
 
         payload = MinioDataFramePayload.from_dict(input_data['data'])
-        data = await payload.retrieve(self.minio_repository, metadata)
+        data = payload.retrieve(self.minio_repository, metadata)
 
         gate_type = input_data['type']
         path_priority = input_data['path_priority']
@@ -385,7 +393,7 @@ class Gates(MinioManager):
             try:
                 if mlflow_content_filter_functions[fil](data, filter_config):
                     filter_output.append(policy)
-                    await self.send_notification_async(
+                    self.send_notification(
                         metadata=metadata,
                         notification_id=f'{gate_type.upper()}_GATE_CONTENT_FILTER__{fil}',
                         message=f'Data not passed the content filter {fil}:{config}',
@@ -395,7 +403,7 @@ class Gates(MinioManager):
                     )
             except Exception as e:
                 trace = traceback.format_exc()
-                await self.send_notification_async(
+                self.send_notification(
                     metadata=metadata,
                     notification_id=f'MLFLOW_GATE_CONTENT_FILTER__{fil}',
                     message=f'Error in filter {fil}:{config}: \n {e}',
@@ -471,7 +479,7 @@ class Gates(MinioManager):
         return policy_type, int(policy_value)
 
     @activity.defn(name='format_transformed_data')
-    async def format_transformed_data(self, input_data: dict[str, Any]) -> MinioDataFramePayload:
+    def format_transformed_data(self, input_data: dict[str, Any]) -> MinioDataFramePayload:
         """
         Format transformed data for storage and export operations.
 
@@ -507,7 +515,7 @@ class Gates(MinioManager):
         self.info('Formatting transformed data...', metadata)
 
         payload = MinioDataFramePayload.from_dict(input_data['data'])
-        data = await payload.retrieve(self.minio_repository, metadata)
+        data = payload.retrieve(self.minio_repository, metadata)
 
         data['timestamp'] = data.index
         data = data.reset_index(drop=True)
@@ -515,7 +523,7 @@ class Gates(MinioManager):
         data = data.melt(id_vars='timestamp', var_name='variable', value_name='value')
         data['model_id'] = model_id
 
-        return await MinioDataFramePayload.from_dataframe(
+        return MinioDataFramePayload.from_dataframe(
             dataframe=data,
             minio_repo=self.minio_repository,
             model_name=input_data['model_name'],
@@ -526,7 +534,7 @@ class Gates(MinioManager):
         )
 
     @activity.defn(name='format_prediction')
-    async def format_prediction(self, input_data: dict[str, Any]) -> dict:
+    def format_prediction(self, input_data: dict[str, Any]) -> dict:
         """
         Format prediction data according to configured storage policies.
 
@@ -558,7 +566,7 @@ class Gates(MinioManager):
         self.info('Formatting prediction...', metadata)
 
         payload = MinioDataFramePayload.from_dict(input_data['data'])
-        data = await payload.retrieve(self.minio_repository, metadata)
+        data = payload.retrieve(self.minio_repository, metadata)
 
         # Create timestamp column from index and reset index
         data['timestamp'] = data.index
@@ -607,7 +615,7 @@ class Gates(MinioManager):
         return data.to_dict()
 
     @activity.defn(name='format_default_prediction')
-    async def format_default_prediction(self, input_data: dict[str, Any]) -> dict:
+    def format_default_prediction(self, input_data: dict[str, Any]) -> dict:
         """
         Create and format default prediction data for error conditions.
 
@@ -652,7 +660,7 @@ class Gates(MinioManager):
         return data.to_dict()
 
     @activity.defn(name='format_retrain_report')
-    async def format_retrain_report(self, input_data: dict[str, Any]) -> dict:
+    def format_retrain_report(self, input_data: dict[str, Any]) -> dict:
         """
         Format retrain report data for storage and audit trail maintenance.
 
@@ -722,7 +730,7 @@ class Gates(MinioManager):
         return report.to_dict()
 
     @activity.defn(name='write_metrics')
-    async def write_metrics(self, input_data: dict[str, Any]):
+    def write_metrics(self, input_data: dict[str, Any]):
         """
         Write prediction performance metrics to Prometheus monitoring system.
 
@@ -759,19 +767,19 @@ class Gates(MinioManager):
             'model_name': metadata['model_name'],
             'workflow_name': metadata['workflow_name'],
         }
-        await self.emit_metric(
+        self.emit_metric_sync(
             metric_object=metrics.PREDICTIONS_WRITTEN_COUNT,
             tags=core_tags,
         )
 
-        await self.emit_metric(
+        self.emit_metric_sync(
             metric_object=metrics.PREDICTION_CONFIDENCE_MONITOR,
             method='set',
             tags=core_tags,
             value=prediction_confidence,
         )
 
-        await self.emit_metric(
+        self.emit_metric_sync(
             metric_object=metrics.PREDICTION_RESPONSE_TIME_MONITOR,
             method='observe',
             tags=core_tags,
@@ -781,7 +789,7 @@ class Gates(MinioManager):
         for server_id, tags in opc_metrics.items():
             for tag, response_time in tags.items():
                 if response_time is not None:
-                    await self.emit_metric(
+                    self.emit_metric_sync(
                         metric_object=metrics.PREDICTION_OPC_WRITING_RESPONSE_TIME_MONITOR,
                         method='observe',
                         tags={
@@ -792,7 +800,7 @@ class Gates(MinioManager):
                         value=response_time,
                     )
 
-                    await self.emit_metric(
+                    self.emit_metric_sync(
                         metric_object=metrics.PREDICTION_OPC_WRITING_COUNT,
                         tags={
                             **core_tags,

laborious/activities/mlflow.py

@@ -1,16 +1,23 @@
 from temporalio import activity, workflow
 
 with workflow.unsafe.imports_passed_through():
+    import tempfile
     import traceback
+    from datetime import datetime
+    from pathlib import Path
+    from shutil import rmtree
     from typing import Any
 
+    import mlflow
     import numpy as np
-    from pandas import to_datetime
+    import pandas as pd
+    from pandas import DataFrame, to_datetime
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
     from sientia_do.notifications.models import NotificationLevel
     from sientia_do.observability.logger import Logger
     from sientia_do.observability.metrics_controller import MetricsController
-    from sientia_do.repository.minio_repository import MinioRepository
+    from sientia_do.observability.sientia_monitoring import SientiaMonitoring
+    from sientia_do.repository.minio_repository_sync import MinioRepository
     from sientia_do.temporal.constants import (
         DATETIME_FORMAT,
         DATETIME_FORMAT_MS_WITH_TZ,
@@ -18,81 +25,83 @@ with workflow.unsafe.imports_passed_through():
         now,
     )
     from sientia_do.utils.formatters import create_sample_dict
+    from sientia_model.model_repository.mlflow_repository import SientiaMLflowRepository
+    from sientia_model.model_repository.plugin_store import PluginStore
 
     from laborious.utils.dataframe_debug import build_dataframe_debug_message
     from laborious.utils.models.minio_dataframe_payload import MinioDataFramePayload
-    from laborious.utils.repository.minio_manager import MinioManager
-    from laborious.utils.repository.model_repository import MLFlowRepository
 
 
-class MLFlow(MinioManager):
+class MLFlow(SientiaMonitoring):
     """
-    MLFlow integration activities for model inference operations.
+    Temporal activities that talk to MLflow through ``SientiaMLflowRepository`` and ``SientiaModel`` wrappers.
 
-    This class provides activities for interacting with MLFlow models, including
+    Models are resolved by registered name and the ``production`` alias (not by legacy stages or
-    data transformation and prediction operations. It handles authentication,
+    separate transform/predict flavors). ``get_cached_model`` loads or reuses a wrapper; inference
-    data preprocessing, and model management with configurable retention policies.
+    uses ``wrapper.transform`` / ``wrapper.predict``; retrain uses ``wrapper.retrain`` or
+    ``wrapper.train`` plus ``store_model`` and registry promotion via ``promote_to_alias``.
 
-    The class implements comprehensive error handling and logging for all
+    Large inputs and outputs flow through ``MinioDataFramePayload`` when workflows offload parquet
-    MLFlow operations, ensuring reliable model inference in production environments.
+    to MinIO. On failure, transform/predict still return a payload with ``success: False`` and
+    error details for downstream gates.
 
     Attributes:
-        mlflow_host (str): MLFlow server hostname
+        mlflow_repository: Client for tracking, registry, artifact download, and run lifecycle.
-        mlflow_port (int): MLFlow server port
+        plugin_store: Reference to the store (runtime is installed on the worker; reserved for
-        mlflow_username (str): MLFlow authentication username
+            future store-backed helpers).
-        mlflow_password (str): MLFlow authentication password
-        model_monitoring_repository (MLFlowRepository): Repository for MLFlow operations
     """
 
     _MAX_DEBUG_DATAFRAME_ROWS = 100
+    _DEFAULT_MODEL_ALIAS = 'production'
+    _REFERENCE_ARTIFACT_CANDIDATES = ('evaluation_data.csv', 'test_data.csv')
 
     def __init__(
         self,
-        mlflow_host: str,
+        mlflow_repository: SientiaMLflowRepository,
-        mlflow_port: int,
+        plugin_store: PluginStore,
-        mlflow_username: str,
-        mlflow_password: str,
         minio_repository: MinioRepository | None = None,
         logger: Logger | None = None,
         notification_handler: NotificationHandler | None = None,
         metrics_controller: MetricsController | None = None,
     ):
         """
-        Initialize MLFlow activities with server configuration.
+        Attach shared MLflow and MinIO clients used by all ML activities in this mixin.
 
         Args:
-            mlflow_host: MLFlow server hostname or IP address
+         - mlflow_repository: Repository built by ``Activities`` (or injected in tests).
-            mlflow_port: MLFlow server port number
+         - plugin_store: Plugin store instance from worker bootstrap.
-            mlflow_username: Username for MLFlow authentication
+         - minio_repository: MinIO client for ``MinioDataFramePayload`` upload/download.
-            mlflow_password: Password for MLFlow authentication
+         - logger: Structured logger.
-            logger: Logger instance for observability and debugging
+         - notification_handler: Notifications on hard failures where applicable.
-            notification_handler: Notification handler for alerts and monitoring
+         - metrics_controller: Shared metrics controller.
 
-        Raises:
+        Return:
-            Exception: If MLFlowRepository initialization fails
+            None
         """
-        MinioManager.__init__(
-            self, minio_repository, logger, notification_handler, metrics_controller
-        )
-        self.mlflow_host = mlflow_host
-        self.mlflow_port = mlflow_port
-        self.mlflow_username = mlflow_username
-        self.mlflow_password = mlflow_password
 
-        self.model_monitoring_repository = MLFlowRepository(
+        self.minio_repository = minio_repository
-            f'{mlflow_host}:{mlflow_port}',
+        SientiaMonitoring.__init__(
-            mlflow_username,
+            self,
-            mlflow_password,
+            logger=logger,
-            logger,
+            notification_handler=notification_handler,
-            notification_handler,
+            metrics_controller=metrics_controller,
-            metrics_controller,
         )
+        self.mlflow_repository = mlflow_repository
+        self.plugin_store = plugin_store
 
     def close(self) -> None:
         """
-        Close the MLFlow activity and clean up resources.
+        Release MinIO manager resources held by the mixin.
+
+        Return:
+            None
         """
-        MinioManager.close(self)
+        if self.minio_repository is not None:
+            try:
+                self.minio_repository.close()
+            finally:
+                self.minio_repository = None
+        SientiaMonitoring.shutdown(self)
 
     def __del__(self):
         self.close()
@@ -115,53 +124,133 @@ class MLFlow(MinioManager):
             metadata,
         )
 
-    @activity.defn(name='request_transform')
+    def _detect_and_parse_datetime_index(self, data: pd.DataFrame, metadata: dict) -> pd.DataFrame:
-    async def request_transform(self, input_data: dict[str, Any]) -> MinioDataFramePayload:
         """
-        Transform input data using MLFlow models.
+        Ensure the transform output index is homogeneous and encoded as ``DATETIME_FORMAT_WITH_TZ`` strings.
 
-        This activity processes input data through MLFlow model transformation,
+        Accepts an all-string index (validated against the format), or all-``datetime`` /
-        including data preprocessing, format conversion, and validation. It handles
+        ``Timestamp`` (naive timestamps are localized to UTC before formatting). Mixed element types
-        data deduplication, pivoting, and cleanup to ensure optimal model performance.
+        or unsupported types raise ``ValueError`` with a message logged at info level.
-
-        The transformation process includes:
-        1. Data deduplication based on variable and timestamp
-        2. Data pivoting for model input format
-        3. Null value handling and cleanup
-        4. MLFlow model transformation request
-        5. Response validation and logging
 
         Args:
-            input_data: Configuration and data for transformation
+         - data: DataFrame whose index carries the time dimension after transform.
-                Required keys:
+         - metadata: Workflow metadata for log correlation.
-                    - metadata (dict): Workflow execution metadata
-                    - data (dict): Input data for transformation
-                    - model_name (str): Name of the MLFlow model to use
-                    - model_retention (int): Model retention period in minutes
 
-        Returns:
+        Return:
-            dict: Transformed data from MLFlow model
+            ``pd.DataFrame``: Same frame with a normalized string index; empty frames are returned as-is.
+        """
+
+        if data.empty:
+            self.info('Data is empty, skipping datetime index detection and parsing', metadata)
+            return data
+
+        index = data.index
+        index_type = type(index[0])
+
+        self.info(f'Index type: {index_type}', metadata)
+
+        message = (
+            f'Index must be all timestamp like column. Valid formats are: pandas Timestamp, datetime, '
+            f'string in format {DATETIME_FORMAT_WITH_TZ}.'
+        )
+
+        if not all(isinstance(i, index_type) for i in index):
+            types = map(str, map(type, index))
+            raise ValueError(f'{message}. Elements are {",".join(types)}')
+
+        if index_type is str:
+            try:
+                pd.to_datetime(data.index, format=DATETIME_FORMAT_WITH_TZ)
+            except ValueError as e:
+                raise ValueError(f'{message}. Unable to parse given date format: {e}') from e
+
+        elif index_type is datetime or index_type is pd.Timestamp:
+            idx = data.index
+            if hasattr(idx, 'tz') and idx.tz is None:
+                data.index = idx.tz_localize('UTC')  # type: ignore[attr-defined]
+
+            data.index = data.index.strftime(DATETIME_FORMAT_WITH_TZ)  # type: ignore[attr-defined]
+        else:
+            raise ValueError(f'{message}. Got {index_type}.')
+
+        return data
+
+    def _resolve_model_version_for_run(self, run_id: str) -> str:
+        """
+        Map an MLflow ``run_id`` to the latest registered model version that produced that run.
+
+        ``search_model_versions`` may return multiple versions if the model was registered more than
+        once for the same run; the highest numeric ``version`` wins so promotion targets the newest
+        artifact set.
+
+        Args:
+         - run_id: Run UUID from ``retrain_model`` / experiment payload.
+
+        Return:
+            str: Registry version string acceptable by ``promote_to_alias``.
 
         Raises:
-            Exception: If transformation fails or MLFlow model is unavailable
+            ValueError: If the filter returns no versions (model not registered for this run).
+        """
+
+        versions = self.mlflow_repository._client.search_model_versions(
+            filter_string=f"run_id='{run_id}'"
+        )
+        if not versions:
+            raise ValueError(f'No registered model version found for run_id={run_id}')
+        latest = max(versions, key=lambda v: int(v.version))
+        return str(latest.version)
+
+    def _resolve_model_alias(self, model_config: dict[str, Any] | None = None) -> str:
+        """
+        Resolve which MLflow alias should be used for model lookup/promotion.
+
+        Args:
+         - model_config: Optional model configuration that may include ``alias``.
+
+        Return:
+            str: Alias name trimmed and normalized; defaults to ``production``.
+        """
+        if not model_config:
+            return self._DEFAULT_MODEL_ALIAS
+        alias = str(model_config.get('alias', self._DEFAULT_MODEL_ALIAS)).strip()
+        return alias or self._DEFAULT_MODEL_ALIAS
+
+    @activity.defn(name='request_transform')
+    def request_transform(self, input_data: dict[str, Any]) -> MinioDataFramePayload:
+        """
+        Pivot long-format sensor rows, load the production wrapper, and run ``wrapper.transform``.
+
+        Expected tabular shape after load: columns including ``variable``, ``timestamp``, ``value``,
+        and ``created_at`` for deduplication. Data are sorted by ``created_at``, de-duplicated per
+        ``(variable, timestamp)``, pivoted wide, then passed to the model. ``model_config`` may
+        include ``retention_minutes`` for wrapper cache TTL.
+
+        Args:
+         - input_data: Dict with ``metadata``, ``model_name``, ``data`` (``MinioDataFramePayload``
+           dict or inline dataframe dict), and optional ``model_config``.
+
+        Return:
+            ``MinioDataFramePayload`` with transformed frame and ``success: True``, or a payload
+            with ``success: False`` and exception details in ``status`` if transform fails.
         """
         metadata = input_data['metadata']
         self.info('Transforming data...', metadata)
 
         payload = MinioDataFramePayload.from_dict(input_data['data'])
-        data = await payload.retrieve(self.minio_repository, metadata)
+        data = payload.retrieve(self.minio_repository, metadata)
 
         model_name = input_data['model_name']
         model_config = input_data.get('model_config', {})
+        model_alias = self._resolve_model_alias(model_config)
 
         self._debug_dataframe('Raw input data:', data, metadata)
 
-        # Sort by created_at in descending order and keep first occurrence of each variable/timestamp pair
+        # Long → wide: keep newest row per (variable, timestamp), then pivot for the wrapper API.
         data = data.sort_values('created_at', ascending=False).drop_duplicates(
             subset=['variable', 'timestamp'], keep='first'
         )
 
-        # Pivot data for model input format
         data = data.pivot(index='timestamp', columns='variable', values='value')
         data.fillna(np.nan, inplace=True)
 
@@ -172,10 +261,25 @@ class MLFlow(MinioManager):
 
         self._debug_dataframe('Processed input data:', data, metadata)
 
-        # Request transformation from MLFlow model
+        try:
-        response_data = await self.model_monitoring_repository.transform(
+            wrapper = self.mlflow_repository.get_cached_model(
-            model_name, data, model_config, metadata
+                model_name=model_name,
+                alias=model_alias,
+                retention_minutes=model_config.get('retention_minutes', 0),
+                metadata=metadata,
             )
+            transformed_df, transform_meta = wrapper.transform(data)
+            if transform_meta:
+                self.info(f'Wrapper transform metadata: {transform_meta}', metadata)
+
+            transformed_df = self._detect_and_parse_datetime_index(transformed_df, metadata)
+
+            response_data: dict[str, Any] = {'success': True, 'content': transformed_df}
+        except Exception as e:
+            response_data = {
+                'success': False,
+                'content': {'message': str(e), 'traceback': traceback.format_exc()},
+            }
 
         self.debug(
             f'Transform raw response data: \n {create_sample_dict(response_data, max_items=5, max_depth=5)}',
@@ -190,7 +294,7 @@ class MLFlow(MinioManager):
         self.info('Data transformed successfully', metadata)
 
         if not response_data.get('success', False):
-            return await MinioDataFramePayload.from_dataframe(
+            return MinioDataFramePayload.from_dataframe(
                 dataframe=None,
                 minio_repo=self.minio_repository,
                 model_name=model_name,
@@ -201,7 +305,7 @@ class MLFlow(MinioManager):
                 logger=self.logger,
             )
 
-        return await MinioDataFramePayload.from_dataframe(
+        return MinioDataFramePayload.from_dataframe(
             dataframe=response_data['content'],
             minio_repo=self.minio_repository,
             model_name=model_name,
@@ -215,58 +319,78 @@ class MLFlow(MinioManager):
         )
 
     @activity.defn(name='request_predict')
-    async def request_predict(self, input_data: dict[str, Any]) -> MinioDataFramePayload:
+    def request_predict(self, input_data: dict[str, Any]) -> MinioDataFramePayload:
         """
-        Execute predictions using MLFlow models.
+        Load the production wrapper and call ``wrapper.predict`` on the prepared feature frame.
 
-        This activity performs ML model inference using MLFlow models with the
+        The activity normalizes ``NaN`` to ``None`` for JSON-friendly columns, sets the row index
-        transformed data. It handles data format conversion, null value processing,
+        the same way as ``retrain_model`` (UTC ``DatetimeIndex`` from ``DATETIME_FORMAT_WITH_TZ``),
-        and model prediction requests with comprehensive error handling.
+        restores that index on the prediction frame, normalizes the prediction index to
-
+        ``DATETIME_FORMAT_WITH_TZ`` strings like ``request_transform``, and records ``response_time``.
-        The prediction process includes:
+        Non-DataFrame predictions are coerced to a single ``prediction`` column.
-        1. Data format validation and cleanup
-        2. Null value handling for model compatibility
-        3. MLFlow model prediction request
-        4. Response validation and logging
-        5. Performance monitoring and metrics
 
         Args:
-            input_data: Configuration and data for prediction
+         - input_data: Same envelope as ``request_transform`` (``metadata``, ``model_name``,
-                Required keys:
+           ``data``, optional ``model_config`` with ``retention_minutes``).
-                    - metadata (dict): Workflow execution metadata
-                    - data (dict): Transformed data for prediction
-                    - model_name (str): Name of the MLFlow model to use
-                    - model_retention (int): Model retention period in minutes
 
-        Returns:
+        Return:
-            dict: Prediction results from MLFlow model
+            ``MinioDataFramePayload`` with predictions or error status mirroring transform behaviour.
-
-        Raises:
-            Exception: If prediction fails or MLFlow model is unavailable
         """
         metadata = input_data['metadata']
         self.info('Predicting data...', metadata)
 
         payload = MinioDataFramePayload.from_dict(input_data['data'])
-        data = await payload.retrieve(self.minio_repository, metadata)
+        data = payload.retrieve(self.minio_repository, metadata)
 
         model_name = input_data['model_name']
         model_config = input_data.get('model_config', {})
+        model_alias = self._resolve_model_alias(model_config)
 
         self._debug_dataframe('Input data for prediction:', data, metadata)
 
-        # Convert numpy.nan to None for model compatibility
         data.replace(np.nan, None, inplace=True)
 
-        data['timestamp'] = data.index
+        data.index = pd.DatetimeIndex(
-        data['timestamp'] = to_datetime(
+            to_datetime(data.index, format=DATETIME_FORMAT_WITH_TZ, utc=True)
-            data['timestamp'], format=DATETIME_FORMAT_WITH_TZ
-        ).dt.strftime(DATETIME_FORMAT)
-
-        # Request prediction from MLFlow model
-        response_data = await self.model_monitoring_repository.predict(
-            model_name, data, model_config, metadata
         )
+        input_index = data.index
+
+        try:
+            wrapper = self.mlflow_repository.get_cached_model(
+                model_name=model_name,
+                alias=model_alias,
+                retention_minutes=model_config.get('retention_minutes', 0),
+                metadata=metadata,
+            )
+            start_time = datetime.now()
+            predict_data, pred_meta = wrapper.predict({}, data)
+            end_time = datetime.now()
+
+            if pred_meta:
+                self.info(f'Wrapper predict metadata: {pred_meta}', metadata)
+
+            if isinstance(predict_data, DataFrame):
+                self._debug_dataframe(
+                    'Data received from model prediction:', predict_data, metadata
+                )
+                predict_data.columns = pd.Index(['prediction'])
+            else:
+                self.debug(
+                    f'Data received from model prediction (not a DataFrame): {predict_data}',
+                    metadata,
+                )
+                predict_data = pd.DataFrame(predict_data, columns=['prediction'])
+
+            predict_data.index = input_index
+            predict_data['response_time'] = (end_time - start_time).total_seconds()
+            predict_data = self._detect_and_parse_datetime_index(predict_data, metadata)
+
+            response_data: dict[str, Any] = {'success': True, 'content': predict_data}
+        except Exception as e:
+            response_data = {
+                'success': False,
+                'content': {'message': str(e), 'traceback': traceback.format_exc()},
+            }
 
         self.debug(
             f'Prediction response data: \n {create_sample_dict(response_data, max_items=5, max_depth=5)}',
@@ -276,7 +400,7 @@ class MLFlow(MinioManager):
         self.info('Data predicted successfully', metadata)
 
         if not response_data.get('success', False):
-            return await MinioDataFramePayload.from_dataframe(
+            return MinioDataFramePayload.from_dataframe(
                 dataframe=None,
                 minio_repo=self.minio_repository,
                 model_name=model_name,
@@ -287,7 +411,7 @@ class MLFlow(MinioManager):
                 logger=self.logger,
             )
 
-        return await MinioDataFramePayload.from_dataframe(
+        return MinioDataFramePayload.from_dataframe(
             dataframe=response_data['content'],
             minio_repo=self.minio_repository,
             model_name=model_name,
@@ -301,36 +425,23 @@ class MLFlow(MinioManager):
         )
 
     @activity.defn(name='retrain_model')
-    async def retrain_model(self, input_data: dict[str, Any]) -> dict[str, Any]:
+    def retrain_model(self, input_data: dict[str, Any]) -> dict[str, Any]:
         """
-        Retrain MLFlow models with updated training data.
+        Fit an updated wrapper from historical data, then log and register in MLflow.
 
-        This activity orchestrates the complete model retraining process,
+        Flow: load long-format data from MinIO → dedupe/pivot like inference prep → require
-        including data preparation, model retraining execution, and result
+        ``model_config['target']`` → read current ``production`` version for ``source_run_id`` tag →
-        validation. It handles data preprocessing, column cleanup, and
+        run ``wrapper.retrain`` outside run timing → ``start_run`` with retrain tags → log input
-        comprehensive error handling for production model management.
+        CSV artifact → ``store_model`` and ``log_params``. Does not promote; the workflow calls
-
+        ``update_production_model`` after validation.
-        The retraining process includes:
-        1. Data timestamp extraction and validation
-        2. Column cleanup and data preparation
-        3. Data pivoting for model input format
-        4. MLFlow model retraining execution
-        5. Result validation and error handling
 
         Args:
-            input_data (dict): Input data containing:
+         - input_data: Must include ``metadata``, ``model_name``, ``data`` (payload), and
-                - metadata (dict): Workflow execution metadata
+           ``model_config`` with at least ``target``.
-                - data (dict[str, Any]): Training data for model retraining
-                - model_name (str): Name of the MLFlow model to retrain
 
-        Returns:
+        Return:
-            dict: Retraining results containing:
+            On success: ``success``, ``experiment`` (``run_id``, ``experiment_id``, ``experiment_name``),
-                - status (str): Retraining operation status
+            ``message``, ``timestamp``. On failure: ``success: False``, error fields, and optional trace.
-                - timestamp (str): Timestamp of the retraining operation
-                - experiment (str): MLFlow experiment identifier
-
-        Raises:
-            Exception: If retraining fails or encounters critical errors
         """
 
         if self.minio_repository is None:
@@ -339,13 +450,12 @@ class MLFlow(MinioManager):
         metadata = input_data['metadata']
 
         try:
-            # Payload-based retrain input (inline dict or MinIO offloaded).
             payload = MinioDataFramePayload.from_dict(input_data['data'])
-            data = await payload.retrieve(self.minio_repository, metadata)
+            data = payload.retrieve(self.minio_repository, metadata)
 
         except Exception as e:
             trace = traceback.format_exc()
-            await self.send_notification_async(
+            self.send_notification(
                 metadata=metadata,
                 notification_id='ERROR_LOADING_RETRAIN_DATA',
                 message=f'Error loading retrain data: {e}',
@@ -371,7 +481,6 @@ class MLFlow(MinioManager):
         timestamp = data['timestamp'].max()
         self.debug(f'Timestamp: {timestamp}', metadata)
 
-        # Sort by created_at in descending order and keep first occurrence of each variable/timestamp pair
         if 'created_at' in data.columns:
             data = data.sort_values('created_at', ascending=False).drop_duplicates(
                 subset=['variable', 'timestamp'], keep='first'
@@ -382,74 +491,130 @@ class MLFlow(MinioManager):
         data.drop(columns=['model_id'], inplace=True, errors='ignore')
         data.drop(columns=['created_at'], inplace=True, errors='ignore')
 
-        # Pivot data for model input format
         data = data.pivot(index='timestamp', columns='variable', values='value')
         data.fillna(np.nan, inplace=True)
-        # data.reset_index(inplace=True)
         data.columns.name = None
+        data.index.name = None
 
-        data['timestamp'] = data.index
+        data.index = pd.DatetimeIndex(
-        data['timestamp'] = to_datetime(
+            to_datetime(data.index, format=DATETIME_FORMAT_WITH_TZ, utc=True)
-            data['timestamp'], format=DATETIME_FORMAT_WITH_TZ
-        ).dt.strftime(DATETIME_FORMAT)
-        data['timestamp'] = to_datetime(data['timestamp'], format=DATETIME_FORMAT)
-
-        data.columns.name = None
-
-        retrain_output = await self.model_monitoring_repository.retrain_model(
-            data=data, model_name=model_name, model_config=model_config, metadata=metadata
         )
 
-        if not retrain_output['success']:
+        target = model_config.get('target')
-            trace = retrain_output['traceback']
+        if target is None:
-            await self.send_notification_async(
+            msg = 'model_config must include "target" for retraining'
+            self.info(msg, metadata)
+            return {
+                'success': False,
+                'experiment': None,
+                'message': msg,
+                'traceback': '',
+                'timestamp': str(timestamp),
+            }
+
+        try:
+            model_alias = self._resolve_model_alias(model_config)
+            mv_src = self.mlflow_repository._client.get_model_version_by_alias(
+                name=model_name,
+                alias=model_alias,
+            )
+            source_run_id = mv_src.run_id
+
+            wrapper = self.mlflow_repository.get_cached_model(
+                model_name=model_name,
+                alias=model_alias,
+                retention_minutes=0,
                 metadata=metadata,
-                notification_id='RETRAIN_MODEL_ERROR',
-                message=f'Error retraining model {model_name}: {retrain_output["message"]}',
-                block='retrain_model',
-                level=NotificationLevel.ERROR,
-                attachment_content=trace,
             )
-            self.error(trace, metadata=metadata)
 
-        return {**retrain_output, 'timestamp': timestamp}
+            # Keep heavy model fitting outside MLflow run timing.
+            prediction_data = wrapper.retrain(data)
+            prediction_data.rename(columns={target: 'prediction'}, inplace=True)
+
+            # Merge prediction data with retrain data
+            evaluation_data = pd.merge(
+                data, prediction_data, left_index=True, right_index=True, how='left'
+            )
+
+            # Rename target column to "target"
+            evaluation_data.rename(columns={target: 'target'}, inplace=True)
+
+            # Reset index and put as column "timestamp"
+            evaluation_data['timestamp'] = evaluation_data.index
+            evaluation_data.reset_index(drop=True, inplace=True)
+            evaluation_data.sort_values(by='timestamp', inplace=True, ascending=True)
+
+            run_name = f'{model_name}-retrain-{datetime.now().strftime("%Y%m%d%H%M%S")}'
+
+            with self.mlflow_repository.start_run(
+                model_name=model_name,
+                run_name=run_name,
+                experiment_name=model_name,
+                tags={'retrain': 'true', 'source_run_id': source_run_id},
+                metadata=metadata,
+            ) as run_info:
+                tmp_dir = tempfile.mkdtemp(prefix='laborious_retrain_')
+                try:
+                    raw_csv = Path(tmp_dir) / 'retrain_input.csv'
+                    evaluation_csv = Path(tmp_dir) / 'evaluation_data.csv'
+                    data.to_csv(raw_csv, index=False)
+                    evaluation_data.to_csv(evaluation_csv, index=False)
+
+                    mlflow.log_artifact(str(raw_csv))
+                    mlflow.log_artifact(str(evaluation_csv))
+                finally:
+                    rmtree(tmp_dir, ignore_errors=True)
+
+                wrapper.store_model(name=model_name)
+
+                self.mlflow_repository.log_params(
+                    {
+                        'retrain': 'true',
+                        'retrain_date': datetime.now().isoformat(),
+                        'source_run_id': source_run_id,
+                        'retrain_samples': str(data.shape),
+                    }
+                )
+
+                experiment_payload = {
+                    'run_id': run_info.run_id,
+                    'experiment_id': run_info.experiment_id,
+                    'experiment_name': model_name,
+                }
+
+            return {
+                'success': True,
+                'experiment': experiment_payload,
+                'message': 'Model retrained successfully.',
+                'timestamp': str(timestamp),
+            }
+
+        except Exception as e:
+            error_msg = f'Error retraining model {model_name}: {e}'
+            self.info(error_msg, metadata)
+            return {
+                'success': False,
+                'experiment': None,
+                'message': error_msg,
+                'traceback': traceback.format_exc(),
+                'timestamp': str(timestamp),
+            }
 
     @activity.defn(name='update_production_model')
-    async def update_production_model(self, input_data: dict[str, Any]) -> dict[Any, Any]:
+    def update_production_model(self, input_data: dict[str, Any]) -> dict[Any, Any]:
         """
-        Update production model with newly trained model version.
+        Point the ``production`` alias at the model version registered for the retrain run.
 
-        This activity manages the critical process of updating production
+        Resolves the highest numeric registry version whose ``run_id`` matches
-        models with newly trained versions. It handles model deployment,
+        ``experiment['run_id']``, then calls ``promote_to_alias``. On failure, sends a notification
-        status tracking, and comprehensive reporting for operational
+        and re-raises so the workflow can surface the error.
-        visibility and audit trails.
-
-        The update process includes:
-        1. Production model update execution
-        2. Status and metadata tracking
-        3. Comprehensive reporting and logging
-        4. Error handling and notification
-        5. Audit trail maintenance
 
         Args:
-            input_data (dict): Input data containing:
+         - input_data: ``metadata``, ``model_name``, and ``experiment`` with ``run_id`` and
-                - metadata (dict): Workflow execution metadata
+           ``experiment_id`` (as returned from ``retrain_model``).
-                - model_name (str): Name of the MLFlow model to update
-                - experiment (str): MLFlow experiment identifier
-                - model_id (str): Unique identifier for the model version
-                - timestamp (str): Timestamp of the update operation
-                - status (str): Current status of the model update
 
-        Returns:
+        Return:
-            dict[Any, Any]: Comprehensive update report containing:
+            Dict with ``model_name``, promoted ``version``, ``mlflow_run_id``, ``mlflow_experiment_id``.
-                - model_id (str): Model version identifier
-                - model_name (str): Name of the updated model
-                - timestamp (str): Update operation timestamp
-                - status (str): Update operation status
-                - Additional MLFlow response metadata
-
-        Raises:
-            Exception: If production model update fails
         """
         metadata = input_data['metadata']
         model_name = input_data['model_name']
@@ -459,16 +624,30 @@ class MLFlow(MinioManager):
         )
 
         try:
-            response = await self.model_monitoring_repository.update_production_model(
+            run_id = experiment['run_id']
-                experiment=experiment, model_name=model_name, metadata=metadata
+            experiment_id = experiment['experiment_id']
+
+            version = self._resolve_model_version_for_run(run_id)
+
+            promote_alias = self._resolve_model_alias(input_data.get('model_config'))
+            self.mlflow_repository.promote_to_alias(
+                model_name=model_name,
+                version=version,
+                alias=promote_alias,
+                metadata=metadata,
             )
 
             self.info(f'Production model {model_name} updated successfully', metadata)
-            return response
+            return {
+                'model_name': model_name,
+                'version': version,
+                'mlflow_run_id': run_id,
+                'mlflow_experiment_id': experiment_id,
+            }
 
         except Exception as e:
             trace = traceback.format_exc()
-            await self.send_notification_async(
+            self.send_notification(
                 metadata=metadata,
                 notification_id='UPDATE_PRODUCTION_MODEL_ERROR',
                 message=f'Error updating production model {model_name}: {e}',
@@ -479,50 +658,98 @@ class MLFlow(MinioManager):
             self.error(trace, metadata=metadata)
             raise e
 
-    @activity.defn(name='get_reference_data')
+    def _resolve_reference_artifact_name(self, run_id: str) -> str | None:
-    async def get_reference_data(self, input_data: dict[str, Any]) -> list[dict] | None:
         """
-        Get reference data from the MLflow Model Registry.
+        Pick the first available reference CSV artifact path from the MLflow run.
 
-        This method retrieves evaluation reference data stored as artifacts in the
+        Candidates are checked in priority order: ``retrain_input.csv``, then ``train_data.csv``.
-        MLflow Model Registry. The reference data is typically used for model
+        A path matches when it equals the candidate or ends with ``/<candidate>`` for nested layouts.
-        drift detection, performance comparison, and quality validation. The method
-        loads the data from a CSV artifact file and formats timestamps for
-        consistent processing.
-
-        The method handles:
-        1. Loading evaluation data artifact from MLflow Model Registry
-        2. Timestamp parsing and formatting for consistency
-        3. Data conversion to dictionary format for workflow consumption
-        4. Graceful handling of missing reference data
 
         Args:
-            input_data (dict): Input data containing:
+         - run_id: MLflow run UUID linked to the production model version.
-                - metadata (dict): Workflow execution metadata
-                - model_name (str): Name of the MLFlow model to get reference data from
 
-        Returns:
+        Return:
-            list[dict[Hashable, Any]] | None: Reference data from the MLflow Model Registry
+            Artifact path string for ``download_artifacts``, or ``None`` if no candidate exists.
-                as a list of dictionaries. Returns None if reference data is not found
-                or if the artifact does not exist.
-
-        Raises:
-            Exception: If artifact loading fails or encounters errors during processing
         """
+        listed = self.mlflow_repository._client.list_artifacts(run_id)
+        paths = [file_info.path for file_info in listed]
+        for candidate in self._REFERENCE_ARTIFACT_CANDIDATES:
+            for path in paths:
+                if path == candidate or path.endswith(f'/{candidate}'):
+                    return path
+        return None
 
+    def _find_downloaded_csv(self, tmpdir: str, artifact_name: str) -> Path | None:
+        """
+        Locate a downloaded reference CSV in the temp directory.
+
+        Args:
+         - tmpdir: Directory where ``download_artifacts`` wrote files.
+         - artifact_name: Basename of the resolved artifact (e.g. ``retrain_input.csv``).
+
+        Return:
+            ``Path`` to the CSV file if found, else ``None``.
+        """
+        direct = Path(tmpdir) / artifact_name
+        if direct.exists():
+            return direct
+        matches = list(Path(tmpdir).rglob(artifact_name))
+        return matches[0] if matches else None
+
+    @activity.defn(name='get_reference_data')
+    def get_reference_data(self, input_data: dict[str, Any]) -> list[dict] | None:
+        """
+        Download reference training CSV from the MLflow run linked to the production alias.
+
+        Resolves ``retrain_input.csv`` or ``train_data.csv`` via artifact listing before download.
+        ``retrain_input.csv`` is preferred when both exist (most recent retrain snapshot). Used by
+        drift workflows to compare live data against the reference distribution logged with the model.
+        Timestamps are normalized to ``DATETIME_FORMAT`` string columns before returning records.
+
+        Args:
+         - input_data: ``metadata``, ``model_name``, and optional ``model_config`` with ``alias``.
+
+        Return:
+            List of row dicts with normalized timestamps, or ``None`` if resolution or load fails.
+        """
         metadata = input_data['metadata']
         model_name = input_data['model_name']
-        artifact = 'evaluation_data.csv'
 
-        reference_data = await self.model_monitoring_repository.load_artifact_dataframe(
+        try:
-            model_name=model_name, artifact_path=artifact, metadata=metadata
+            model_alias = self._resolve_model_alias(input_data.get('model_config'))
+            mv = self.mlflow_repository._client.get_model_version_by_alias(
+                name=model_name,
+                alias=model_alias,
             )
+            run_id = mv.run_id
 
-        if reference_data is None:
+            artifact_path = self._resolve_reference_artifact_name(run_id)
+            if artifact_path is None:
                 self.warning(f'Reference data not found for model {model_name}', metadata)
                 return None
 
+            artifact_name = Path(artifact_path).name
+            tmpdir = tempfile.mkdtemp(prefix='laborious_eval_')
+            try:
+                self.mlflow_repository.download_artifacts(
+                    run_id=run_id,
+                    artifact_path=artifact_path,
+                    dst_path=tmpdir,
+                    metadata=metadata,
+                )
+                csv_path = self._find_downloaded_csv(tmpdir, artifact_name)
+                if csv_path is None:
+                    self.warning(f'Reference data not found for model {model_name}', metadata)
+                    return None
+                reference_data = pd.read_csv(csv_path)
+            finally:
+                rmtree(tmpdir, ignore_errors=True)
+
             reference_data['timestamp'] = to_datetime(reference_data['timestamp'])
             reference_data['timestamp'] = reference_data['timestamp'].dt.strftime(DATETIME_FORMAT)
 
             return reference_data.to_dict(orient='records')
+
+        except Exception as e:
+            self.warning(f'Reference data not found for model {model_name}: {e}', metadata)
+            return None

laborious/activities/model_metrics.py

@@ -7,14 +7,15 @@ with workflow.unsafe.imports_passed_through():
     from typing import Any
 
     import numpy as np
-    from pandas import DataFrame, Index, to_datetime
+    import pandas as pd
-    from sientia.ModelAnalysis import ModelAnalysis
+    from pandas import DataFrame, Index, Series, to_datetime
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
     from sientia_do.notifications.models import NotificationLevel
     from sientia_do.observability.logger import Logger
     from sientia_do.observability.metrics_controller import MetricsController
     from sientia_do.observability.sientia_monitoring import SientiaMonitoring
-    from sientia_do.temporal.constants import DATETIME_FORMAT, DATETIME_FORMAT_WITH_TZ
+    from sientia_do.temporal.constants import DATETIME_FORMAT_WITH_TZ
+    from sientia_model.analytics.drift_analysis import DriftAnalysis, DriftInsufficientDataError
 
     from laborious import metrics
     from laborious.utils.dataframe_debug import build_dataframe_debug_message
@@ -27,9 +28,12 @@ warnings.filterwarnings(
 
 class ModelMetrics(SientiaMonitoring):
     """
-    Metrics activities for the Laborious system.
+    Metrics and statistical analysis activities for the Laborious pipeline.
 
-    This class provides activities for writing metrics to the Prometheus monitoring system.
+    This class centralizes drift/statistical computations and model-quality
+    aggregates used by scheduled workflows. Besides producing tabular outputs
+    for persistence, it also emits operational metrics (count, lag, error)
+    through ``SientiaMonitoring`` so execution health is observable in runtime.
     """
 
     _MAX_DEBUG_DATAFRAME_ROWS = 100
@@ -44,7 +48,10 @@ class ModelMetrics(SientiaMonitoring):
 
     def close(self) -> None:
         """
-        Close the model metrics activity and clean up resources.
+        Shutdown monitoring resources associated with model metrics activities.
+
+        This is invoked during worker teardown to flush/close metric controller
+        internals and prevent dangling telemetry tasks.
         """
         SientiaMonitoring.shutdown(self)
 
@@ -69,7 +76,26 @@ class ModelMetrics(SientiaMonitoring):
             metadata,
         )
 
-    async def get_drift_metrics(
+    def _drift_analyze_stage_error(
+        self,
+        exc: Exception,
+        context: str,
+        metadata: dict[str, Any],
+        core_labels: dict[str, Any],
+    ) -> None:
+        """
+        Log analyzer failure for a drift stage and increment the analyze error metric.
+
+        Args:
+         - exc (Exception): Failure raised by ``sientia_model``.
+         - context (str): Short label for the log line (e.g. univariate detection).
+         - metadata (dict[str, Any]): Workflow metadata for logging.
+         - core_labels (dict[str, Any]): Tags from ``get_core_labels`` for metrics.
+        """
+        self.error(f'{context}: {exc}', metadata)
+        self.emit_metric_sync(metric_object=metrics.MODEL_ANALYZE_ERROR_COUNT, tags=core_labels)
+
+    def get_drift_metrics(
         self,
         reference_data: DataFrame,
         target_data: DataFrame,
@@ -80,24 +106,38 @@ class ModelMetrics(SientiaMonitoring):
         metadata: dict[str, Any],
     ) -> DataFrame:
         """
-        Calculate univariate drift metrics for a model.
+        Compute univariate and multivariate drift outputs and merge them into one dataframe.
+
+        The method orchestrates three analysis stages (univariate drift,
+        multivariate drift, and dataframe projection), emitting lag/count/error
+        metrics for each stage independently so failures are attributable.
+
         Args:
-            model_analysis (ModelAnalysis): Model analysis object
+         - reference_data (DataFrame): Baseline dataset representing expected behavior.
-            reference_data (DataFrame): Reference data
+         - target_data (DataFrame): Current analysis dataset to compare against reference.
-            target_data (DataFrame): Target data
+         - target_name (str): Target column name used by ``DriftAnalysis`` config.
-            reference_columns (list[str]): Reference columns
+         - reference_columns (Index): Feature columns evaluated for drift.
-            drift_metrics (list[str]): Drift metrics
+         - drift_metrics (list[str]): Enabled univariate methods.
-            metadata (dict[str, Any]): Workflow execution metadata
+         - chunk_period (str): Time bucket granularity used by analysis methods.
+         - metadata (dict[str, Any]): Workflow metadata for logs and notifications.
+
+        Return:
+         DataFrame: Consolidated drift dataframe from ``get_drift_metrics_dataframe`` using
+         ``method`` / ``value`` (and optional ``threshold``, ``drift_type``), ready for
+         activity-level formatting before Postgres export.
         """
+        # ``DriftAnalysis`` uses truthiness checks on ``features`` (e.g. ``if not features``);
+        # a pandas ``Index`` is ambiguous in boolean context — normalize to a list.
+        feature_names: list[str] = list(reference_columns)
 
         config = {
             'target': target_name,
             'prediction': 'prediction',
             'timestamp': 'timestamp',
-            'features': reference_columns,
+            'features': feature_names,
         }
 
-        model_analysis = ModelAnalysis(config=config)
+        drift_analysis = DriftAnalysis(config=config)
 
         self._debug_dataframe(
             f'Reference data: Size {reference_data.shape}', reference_data, metadata
@@ -108,64 +148,88 @@ class ModelMetrics(SientiaMonitoring):
         core_labels = self.get_core_labels(metadata, operation_type='detect_univariate_drift')
         start_time = time.time()
         try:
-            univariate_drift = model_analysis.detect_univariate_drift(
+            univariate_drift = drift_analysis.detect_univariate_drift(
                 reference_df=reference_data,
                 analysis_df=target_data,
-                features=reference_columns,
+                features=feature_names,
                 timestamp_col=config['timestamp'],
                 methods=drift_metrics,
                 chunk_period=chunk_period,
             )
         except Exception as e:
-            self.error(f'Error detecting univariate drift: {e}', metadata)
+            if isinstance(e, DriftInsufficientDataError):
-            await self.emit_metric(
+                raise
-                metric_object=metrics.MODEL_ANALYZE_ERROR_COUNT, tags=core_labels
+            self._drift_analyze_stage_error(
+                e, 'Error detecting univariate drift', metadata, core_labels
             )
-            raise e
+            raise
-        await self.observe_lag(start_time, metrics.MODEL_ANALYZE_LAG, core_labels)
+        self.observe_lag_sync(start_time, metrics.MODEL_ANALYZE_LAG, core_labels)
-        await self.emit_metric(metric_object=metrics.MODEL_ANALYZE_COUNT, tags=core_labels)
+        self.emit_metric_sync(metric_object=metrics.MODEL_ANALYZE_COUNT, tags=core_labels)
 
         core_labels = self.get_core_labels(metadata, operation_type='detect_multivariate_drift')
         start_time = time.time()
         try:
-            multivariate_drift = model_analysis.detect_multivariate_drift(
+            multivariate_drift = drift_analysis.detect_multivariate_drift(
                 reference_df=reference_data,
                 analysis_df=target_data,
-                features=reference_columns,
+                features=feature_names,
                 timestamp_col=config['timestamp'],
                 chunk_period=chunk_period,
             )
         except Exception as e:
-            self.error(f'Error detecting multivariate drift: {e}', metadata)
+            if isinstance(e, DriftInsufficientDataError):
-            await self.emit_metric(
+                raise
-                metric_object=metrics.MODEL_ANALYZE_ERROR_COUNT, tags=core_labels
+            self._drift_analyze_stage_error(
+                e, 'Error detecting multivariate drift', metadata, core_labels
             )
-            raise e
+            raise
-        await self.observe_lag(start_time, metrics.MODEL_ANALYZE_LAG, core_labels)
+        self.observe_lag_sync(start_time, metrics.MODEL_ANALYZE_LAG, core_labels)
-        await self.emit_metric(metric_object=metrics.MODEL_ANALYZE_COUNT, tags=core_labels)
+        self.emit_metric_sync(metric_object=metrics.MODEL_ANALYZE_COUNT, tags=core_labels)
 
         start_time = time.time()
         core_labels = self.get_core_labels(metadata, operation_type='get_drift_metrics_dataframe')
         try:
-            drift_df = model_analysis.get_drift_metrics_dataframe(
+            drift_df = drift_analysis.get_drift_metrics_dataframe(
                 univariate_drift=univariate_drift,
                 multivariate_drift=multivariate_drift,
             )
         except Exception as e:
-            self.error(f'Error getting drift metrics: {e}', metadata)
+            if isinstance(e, DriftInsufficientDataError):
-            await self.emit_metric(
+                raise
-                metric_object=metrics.MODEL_ANALYZE_ERROR_COUNT, tags=core_labels
+            self._drift_analyze_stage_error(
+                e, 'Error building drift metrics dataframe', metadata, core_labels
             )
-            raise e
+            raise
-        await self.observe_lag(start_time, metrics.MODEL_ANALYZE_LAG, core_labels)
+        self.observe_lag_sync(start_time, metrics.MODEL_ANALYZE_LAG, core_labels)
-        await self.emit_metric(metric_object=metrics.MODEL_ANALYZE_COUNT, tags=core_labels)
+        self.emit_metric_sync(metric_object=metrics.MODEL_ANALYZE_COUNT, tags=core_labels)
-
-        self._debug_dataframe(f'Drift dataframe: Size {drift_df.shape}', drift_df, metadata)
 
         return drift_df
 
+    @staticmethod
+    def _to_naive_utc(series: Series) -> Series:
+        """
+        Parse ``series`` as datetime and return a TZ-naive UTC copy.
+
+        ``sientia_model.analytics.drift_analysis.DriftAnalysis`` preserves the
+        timezone of the input dataframe in its outputs, while target rows
+        loaded from PostgreSQL come in with ``+00:00``. Forcing both sides of
+        a comparison to TZ-naive UTC keeps ``isin`` / ``floor`` operations
+        deterministic regardless of how the analyzer constructs its
+        timestamps.
+
+        Args:
+         - series (Series): Input series containing datetime-parseable values.
+
+        Return:
+         Series: Datetime64 series with ``tz=None`` representing UTC instants.
+        """
+        parsed = to_datetime(series)
+        if getattr(parsed.dt, 'tz', None) is not None:
+            parsed = parsed.dt.tz_convert('UTC').dt.tz_localize(None)
+        return parsed
+
     @activity.defn(name='calculate_drift')
-    async def calculate_drift(self, input_data: dict[str, Any]) -> list[dict]:
+    def calculate_drift(self, input_data: dict[str, Any]) -> list[dict]:
         """
         Calculate drift metrics for a model.
 
@@ -195,14 +259,17 @@ class ModelMetrics(SientiaMonitoring):
 
         target_data = target_data.pivot(index='timestamp', columns='variable', values='value')
         target_data['timestamp'] = target_data.index
+        # Keep timestamps as datetime: DriftAnalysis._chunk_dataframe relies on
+        # ``pd.Grouper(freq=...)`` which rejects string timestamp columns.
         target_data['timestamp'] = to_datetime(target_data['timestamp'])
-        target_data['timestamp'] = target_data['timestamp'].dt.strftime(DATETIME_FORMAT)
         target_data = target_data.reset_index(drop=True)
         target_data.dropna(inplace=True)
 
         if reference_raw_data is not None:
             self.info('Using reference data', metadata)
             reference_data = DataFrame(reference_raw_data)
+            if 'timestamp' in reference_data.columns:
+                reference_data['timestamp'] = to_datetime(reference_data['timestamp'])
             accurate = True
         else:
             # Get 30% first rows of target_data
@@ -211,7 +278,7 @@ class ModelMetrics(SientiaMonitoring):
             reference_data = target_data.head(int(len(target_data) * 0.3))
             accurate = False
 
-            await self.send_notification_async(
+            self.send_notification(
                 metadata=metadata,
                 notification_id='MODEL_METRICS_REFERENCE_DATA_WARNING',
                 message='Using 30% first rows of target data as reference data',
@@ -225,7 +292,7 @@ class ModelMetrics(SientiaMonitoring):
         ).columns
 
         try:
-            drift_df = await self.get_drift_metrics(
+            drift_df = self.get_drift_metrics(
                 reference_data=reference_data,
                 target_data=target_data,
                 target_name=target_name,
@@ -235,32 +302,30 @@ class ModelMetrics(SientiaMonitoring):
                 metadata=metadata,
             )
         except Exception as e:
+            if isinstance(e, DriftInsufficientDataError):
+                self.error(str(e), metadata)
+                notification_id = e.notification_id
+                notification_message = str(e)
+            else:
                 self.error(f'Error getting drift metrics: {e}', metadata)
-            await self.send_notification_async(
+                notification_id = 'MODEL_METRICS_GET_DRIFT_METRICS_ERROR'
+                notification_message = f'Error getting drift metrics: {e}'
+            self.send_notification(
                 metadata=metadata,
-                notification_id='MODEL_METRICS_GET_DRIFT_METRICS_ERROR',
+                notification_id=notification_id,
-                message=f'Error getting drift metrics: {e}',
+                message=notification_message,
                 block='model_metrics',
                 level=NotificationLevel.ERROR,
                 attachment_content=traceback.format_exc(),
             )
-            return []
+            raise
 
-        if drift_df.empty:
+        # Drop chunks whose floored timestamp does not appear in the analysis window.
-            self.warning('No drift metrics found', metadata)
+        # ``DriftAnalysis`` chunks over ``analysis_df``; this only excludes rows that
-            return []
+        # do not belong to the current target window (e.g. stray merged reference rows).
-
+        target_floor = self._to_naive_utc(target_data['timestamp']).dt.floor(chunk_period)
-        # Drop unnecessary columns
+        drift_floor = self._to_naive_utc(drift_df['timestamp']).dt.floor(chunk_period)
-        drift_df.drop(columns=['p_value'], inplace=True)
+        drift_df = drift_df[drift_floor.isin(target_floor)]
-
-        # Extract timestamps only until minutes
-        if chunk_period == 'min':
-            target_timestamps = target_data['timestamp'].apply(lambda x: x[:16])
-        else:
-            target_timestamps = target_data['timestamp']
-
-        # Drop rows where timestamp is not in target data, to avoid save drift from reference
-        drift_df = drift_df[drift_df['timestamp'].isin(target_timestamps)]
 
         if drift_df.empty:
             self.warning(
@@ -269,33 +334,36 @@ class ModelMetrics(SientiaMonitoring):
             )
             return []
 
-        # Rename columns to match database columns
+        # Analyzer emits diagnostic columns that are not stored in ``sientia_data.drift_metrics``.
-        drift_df.rename(
+        drift_df = drift_df.drop(columns=['threshold', 'drift_type'], errors='ignore')
-            columns={
-                'metric': 'method',
-                'statistic': 'value',
-            },
-            inplace=True,
-        )
 
-        # Drop duplicates
+        drift_df['model_id'] = str(model_id)
-        drift_df.drop_duplicates(
-            subset=['timestamp', 'method', 'feature'], keep='first', inplace=True
-        )
-
-        drift_df['model_id'] = model_id
         drift_df['accurate'] = accurate
 
-        drift_df['timestamp'] = to_datetime(drift_df['timestamp'])
+        # ``timestamp`` is overridden with the most recent target instant so
-        drift_df['timestamp'] = drift_df['timestamp'].dt.tz_localize('UTC')
+        # every persisted row shares a single business timestamp (the run's
-        drift_df['timestamp'] = drift_df['timestamp'].dt.strftime(DATETIME_FORMAT_WITH_TZ)
+        # logical "now"), matching what downstream consumers expect.
+        latest_target_timestamp = self._to_naive_utc(target_data['timestamp']).max()
+        drift_df['timestamp'] = (
+            pd.Timestamp(latest_target_timestamp)
+            .tz_localize('UTC')
+            .strftime(DATETIME_FORMAT_WITH_TZ)
+        )
+
+        # ``chunk_start_date`` / ``chunk_end_date`` may carry nanosecond
+        # precision (beyond ``timestamptz`` microseconds), so serialize as ISO
+        # text for the ``text`` Postgres columns.
+        for column in ('chunk_start_date', 'chunk_end_date'):
+            drift_df[column] = drift_df[column].apply(
+                lambda value: pd.Timestamp(value).isoformat() if pd.notna(value) else None
+            )
 
         self._debug_dataframe(f'Drift dataframe: Size {drift_df.shape}', drift_df, metadata)
 
         return drift_df.to_dict(orient='records')
 
     @activity.defn(name='calculate_simple_metrics')
-    async def calculate_simple_metrics(self, input_data: dict[str, Any]) -> list[dict]:
+    def calculate_simple_metrics(self, input_data: dict[str, Any]) -> list[dict]:
         """
         Calculate simple metrics for a model. Metrics available are:
             - rmse
@@ -319,7 +387,7 @@ class ModelMetrics(SientiaMonitoring):
         metadata = input_data['metadata']
         model_id = input_data['model_id']
         target_data = DataFrame(input_data['target_data'])
-        metrics = input_data['metrics']
+        metric_names = input_data['metrics']
         interval_minutes = input_data['interval_minutes']
 
         data_size = target_data.shape[0]
@@ -329,9 +397,9 @@ class ModelMetrics(SientiaMonitoring):
         diff = target_data['target'] - target_data['prediction']
         diff_squared = diff**2
 
-        self.info(f'Calculating simple metrics for model {model_id}: {metrics}', metadata)
+        self.info(f'Calculating simple metrics for model {model_id}: {metric_names}', metadata)
 
-        for metric in metrics:
+        for metric in metric_names:
             if metric == 'rmse':
                 output_data.append({'metric': 'rmse', 'value': np.sqrt(np.mean(diff_squared))})
             elif metric == 'mse':

laborious/activities/opc.py

@@ -15,6 +15,45 @@ with workflow.unsafe.imports_passed_through():
     from laborious.utils.repository.opc_repository import OpcRepository
 
 OPC_WRITTING_ERROR_CONFIDENCE = 12
+OPC_SESSION_BAD_CONFIDENCE = 14
+OPC_SESSION_BAD_COMMENT_PREFIX = 'OPC UA session/channel error:'
+OPC_WRITTING_ERROR_MESSAGE = 'Some data could not be written to OPC servers'
+OPC_RECONNECT_IN_PROGRESS_COMMENT = 'OPC UA reconnect in progress'
+OPC_COMMENT_SEPARATOR = ' | '
+
+
+def _opc_session_bad_comment(opc_status: str | None) -> str:
+    status = opc_status or 'Unknown'
+    return f'{OPC_SESSION_BAD_COMMENT_PREFIX} {status}'
+
+
+def _apply_opc_write_error(
+    error_info: dict[str, Any] | None,
+    session_bad_seen: bool,
+    session_bad_status: str | None,
+    reconnect_in_progress_seen: bool,
+) -> tuple[bool, str | None, bool]:
+    """
+    Update session/reconnect flags from an OPC write error payload.
+
+    Args:
+        error_info: Repository error details, or None when the write succeeded.
+        session_bad_seen: Whether a session_bad error was seen so far.
+        session_bad_status: Last known OPC status for session errors.
+        reconnect_in_progress_seen: Whether reconnect_in_progress was seen so far.
+
+    Return:
+        Updated (session_bad_seen, session_bad_status, reconnect_in_progress_seen).
+    """
+    if not error_info:
+        return session_bad_seen, session_bad_status, reconnect_in_progress_seen
+
+    kind = error_info.get('opc_error_kind')
+    if kind == 'session_bad':
+        return True, error_info.get('opc_status', session_bad_status), reconnect_in_progress_seen
+    if kind == 'reconnect_in_progress':
+        return session_bad_seen, session_bad_status, True
+    return session_bad_seen, session_bad_status, reconnect_in_progress_seen
 
 
 class OPC(SientiaMonitoring):
@@ -43,15 +82,13 @@ class OPC(SientiaMonitoring):
         notification_handler: NotificationHandler,
         metrics_controller: MetricsController,
     ):
-        self.logger = logger
-        self.notification_handler = notification_handler
         self.opc_servers = opc_servers
 
         SientiaMonitoring.__init__(self, logger, notification_handler, metrics_controller)
 
         self.opc_repository: dict[str, OpcRepository] = {}
 
-    async def init_opc(self):
+    def init_opc(self):
         """
         Initialize OPC server connections and establish communication channels.
 
@@ -75,10 +112,10 @@ class OPC(SientiaMonitoring):
             the initialization of other OPC servers. Each server is handled
             independently to ensure maximum availability.
         """
-        self.logger.info('Initializing OPC servers...')
+        self.info('Initializing OPC servers...')
         for opc_id, server in self.opc_servers.items():
             self.opc_repository[opc_id] = OpcRepository(
-                opc_id=server['id'],
+                opc_id=opc_id,
                 server_name=server['server_name'],
                 url=server['url'],
                 logger=self.logger,
@@ -87,12 +124,12 @@ class OPC(SientiaMonitoring):
                 private_key_path=server['private_key_path'],
                 server_cert_path=server['server_cert_path'],
                 notification_handler=self.notification_handler,
-                reconnection_interval=server['reconnection_interval'],
+                reconnection_interval=server.get('reconnection_interval', 60),
                 metrics_controller=self.metrics_controller,
             )
-            is_connected, error_data = await self.opc_repository[opc_id].connect()
+            is_connected, error_data = self.opc_repository[opc_id].connect()
             if not is_connected:
-                await self.send_notification_async(
+                self.send_notification(
                     metadata={
                         'model_id': '-',
                         'model_name': '-',
@@ -106,11 +143,9 @@ class OPC(SientiaMonitoring):
                     attachment_content=error_data.get('attachment_content', None),
                 )
             else:
-                self.logger.info(
+                self.info(f'OPC server {opc_id}:{server["server_name"]} connected successfully.')
-                    f'OPC server {opc_id}:{server["server_name"]} connected successfully.'
-                )
 
-    async def write_data(
+    def write_data(
         self,
         server_id: str,
         tag: str,
@@ -118,32 +153,21 @@ class OPC(SientiaMonitoring):
         data_type: str,
         tag_type: str,
         metadata: dict[str, Any],
-    ) -> float | None:
+    ) -> tuple[float | None, dict[str, Any] | None]:
         """
         Write data to a specific OPC server tag with comprehensive error handling.
 
-        This method provides a secure and reliable way to write data to OPC servers
+        Return:
-        with automatic error handling, notification integration, and detailed logging.
+            tuple[float | None, dict[str, Any] | None]: Response time on success, or
-        It validates server availability before attempting write operations and
+                (None, error info_data) on repository failure.
-        provides comprehensive error reporting for operational monitoring.
-
-        Args:
-            - server_id (str): The id of the OPC server.
-            - tag (str): The tag to write to.
-            - data (Any): The data to write.
-            - data_type (str): The data type.
-            - tag_type (str): The tag type.
-
-        Returns:
-            - bool: True if the data was written successfully, False otherwise.
         """
 
         try:
-            is_success, info_data = await self.opc_repository[server_id].write_data(
+            is_success, info_data = self.opc_repository[server_id].write_data(
-                tag, data, data_type, self.logger, metadata
+                tag, data, data_type, metadata
             )
             if not is_success:
-                await self.send_notification_async(
+                self.send_notification(
                     metadata=metadata,
                     notification_id=info_data['notification_id'],
                     message=info_data['message'],
@@ -151,11 +175,11 @@ class OPC(SientiaMonitoring):
                     level=info_data.get('level', NotificationLevel.ERROR),
                     attachment_content=info_data.get('attachment_content', None),
                 )
-                return None
+                return None, info_data
-            return info_data['response_time']
+            return info_data['response_time'], None
         except Exception as e:
             trace = traceback.format_exc()
-            await self.send_notification_async(
+            self.send_notification(
                 metadata=metadata,
                 notification_id=f'WRITE_OPC_{tag_type.upper()}_ERROR',
                 message=f'Error writing data to OPC server: {e}',
@@ -163,9 +187,9 @@ class OPC(SientiaMonitoring):
                 level=NotificationLevel.ERROR,
                 attachment_content=trace,
             )
-            raise e
+            raise
 
-    async def validate_server(self, server_id: str, metadata: dict[str, Any]) -> bool:
+    def validate_server(self, server_id: str, metadata: dict[str, Any]) -> bool:
         """
         Validate that an OPC server is available and configured for write operations.
 
@@ -188,7 +212,7 @@ class OPC(SientiaMonitoring):
         """
         if self.opc_repository.get(server_id) is None:
             message = f'OPC server {server_id} not found to perform write operation.'
-            await self.send_notification_async(
+            self.send_notification(
                 metadata=metadata,
                 notification_id='OPC_SERVER_NOT_FOUND',
                 message=message,
@@ -199,13 +223,69 @@ class OPC(SientiaMonitoring):
             return False
         return True
 
-    async def manage_output_tags(
+    def _write_tags_from_config(
+        self,
+        server_id: str,
+        tags_config: dict[str, dict[str, Any]],
+        data: DataFrame,
+        data_column: str,
+        tag_type: str,
+        log_label: str,
+        metadata: dict[str, Any],
+    ) -> tuple[dict[str, float | None], bool, str | None, bool]:
+        """
+        Write a group of OPC tags and collect response times and error flags.
+
+        Args:
+            server_id: Target OPC server identifier.
+            tags_config: Tag name to configuration mapping.
+            data: DataFrame with prediction/confidence columns.
+            data_column: Column name whose first row value is written.
+            tag_type: Tag category passed to write_data ('prediction' or 'confidence').
+            log_label: Human-readable label for success logs.
+            metadata: Context metadata for logging and notifications.
+
+        Return:
+            (response_times, session_bad_seen, session_bad_status, reconnect_in_progress_seen)
+        """
+        response_times: dict[str, float | None] = {}
+        session_bad_seen = False
+        session_bad_status: str | None = None
+        reconnect_in_progress_seen = False
+
+        for tag, tag_config in tags_config.items():
+            response_time, error_info = self.write_data(
+                server_id=server_id,
+                tag=tag,
+                data=data.head(1)[data_column].values[0],
+                data_type=tag_config['data_type'],
+                tag_type=tag_type,
+                metadata=metadata,
+            )
+            session_bad_seen, session_bad_status, reconnect_in_progress_seen = (
+                _apply_opc_write_error(
+                    error_info,
+                    session_bad_seen,
+                    session_bad_status,
+                    reconnect_in_progress_seen,
+                )
+            )
+            if response_time is not None:
+                self.info(
+                    f'{log_label} written to OPC server {server_id} for tag {tag}.',
+                    metadata,
+                )
+            response_times[tag] = response_time
+
+        return response_times, session_bad_seen, session_bad_status, reconnect_in_progress_seen
+
+    def manage_output_tags(
         self,
         server_id: str,
         config: dict[str, Any],
         data: DataFrame,
         metadata: dict[str, Any],
-    ) -> tuple[bool, dict[str, float | None]]:
+    ) -> tuple[bool, dict[str, float | None], bool, str | None, bool]:
         """
         Manage the writing of prediction and confidence data to OPC server tags.
 
@@ -232,49 +312,50 @@ class OPC(SientiaMonitoring):
                 - overall_success: True if all configured tags were written successfully
                 - total_tags_written: Count of successfully written tags
         """
-
         response_times: dict[str, float | None] = {}
+        session_bad_seen = False
+        session_bad_status: str | None = None
+        reconnect_in_progress_seen = False
 
-        if 'prediction_tags' in config:
+        tag_groups = (
-            for tag, tag_config in config['prediction_tags'].items():
+            ('prediction_tags', 'prediction', 'prediction', 'Prediction data'),
-                response_time = await self.write_data(
+            ('confidence_tags', 'prediction_confidence', 'confidence', 'Confidence data'),
+        )
+        for config_key, data_column, tag_type, log_label in tag_groups:
+            if config_key not in config:
+                continue
+            (
+                group_times,
+                group_session_bad,
+                group_status,
+                group_reconnect,
+            ) = self._write_tags_from_config(
                 server_id=server_id,
-                    tag=tag,
+                tags_config=config[config_key],
-                    data=data.head(1)['prediction'].values[0],
+                data=data,
-                    data_type=tag_config['data_type'],
+                data_column=data_column,
-                    tag_type='prediction',
+                tag_type=tag_type,
+                log_label=log_label,
                 metadata=metadata,
             )
-                if response_time is not None:
+            response_times.update(group_times)
-                    self.info(
+            if group_session_bad:
-                        f'Prediction data written to OPC server {server_id} for tag {tag}.',
+                session_bad_seen = True
-                        metadata,
+                session_bad_status = group_status or session_bad_status
-                    )
+            if group_reconnect:
-                response_times[tag] = response_time
+                reconnect_in_progress_seen = True
-
-        if 'confidence_tags' in config:
-            for tag, tag_config in config['confidence_tags'].items():
-                response_time = await self.write_data(
-                    server_id=server_id,
-                    tag=tag,
-                    data=data.head(1)['prediction_confidence'].values[0],
-                    data_type=tag_config['data_type'],
-                    tag_type='confidence',
-                    metadata=metadata,
-                )
-                if response_time is not None:
-                    self.info(
-                        f'Confidence data written to OPC server {server_id} for tag {tag}.',
-                        metadata,
-                    )
-                response_times[tag] = response_time
 
         success = None not in response_times.values()
-
+        return (
-        return success, response_times
+            success,
+            response_times,
+            session_bad_seen,
+            session_bad_status,
+            reconnect_in_progress_seen,
+        )
 
     @activity.defn(name='write_opc_data')
-    async def write_opc_data(
+    def write_opc_data(
         self, input_data: dict[str, Any]
     ) -> tuple[dict[Hashable, Any], dict[str, dict[str, float | None]]]:
         """
@@ -301,30 +382,61 @@ class OPC(SientiaMonitoring):
         self.info(f'Data to write: {data.size} rows', metadata)
 
         success = True
+        session_bad_seen = False
+        session_bad_status: str | None = None
+        reconnect_in_progress_seen = False
 
-        metrics: dict[str, dict[str, float | None]] = {}
+        opc_metrics: dict[str, dict[str, float | None]] = {}
 
         for server_id, config in opc_output_config.items():
-            if not await self.validate_server(server_id, metadata):
+            if not self.validate_server(server_id, metadata):
                 success = False
                 continue
 
-            local_success, local_response_times = await self.manage_output_tags(
+            (
-                server_id, config, data, metadata
+                local_success,
-            )
+                local_response_times,
-            metrics[server_id] = local_response_times
+                local_session_bad,
+                local_status,
+                local_reconnect_in_progress,
+            ) = self.manage_output_tags(server_id, config, data, metadata)
+            opc_metrics[server_id] = local_response_times
             local_count = len(local_response_times)
             success = success and local_success
+            if local_session_bad:
+                session_bad_seen = True
+                session_bad_status = local_status or session_bad_status
+            if local_reconnect_in_progress:
+                reconnect_in_progress_seen = True
 
+            n_pred = len(config.get('prediction_tags') or {})
+            n_conf = len(config.get('confidence_tags') or {})
             self.info(
-                f'Process completed for OPC server {server_id}: {local_count} of {len(config.get("prediction_tags", []))} prediction tags and {len(config.get("confidence_tags", []))} confidence tags',
+                f'Process completed for OPC server {server_id}: {local_count} of {n_pred} prediction tags and {n_conf} confidence tags',
                 metadata,
             )
 
-        return self.process_confidence(data, success, metadata), metrics
+        return (
+            self.process_confidence(
+                data,
+                success,
+                metadata,
+                session_bad=session_bad_seen,
+                opc_status=session_bad_status,
+                reconnect_in_progress=reconnect_in_progress_seen,
+            ),
+            opc_metrics,
+        )
 
     def process_confidence(
-        self, data: DataFrame, success: bool, metadata: dict[str, Any]
+        self,
+        data: DataFrame,
+        success: bool,
+        metadata: dict[str, Any],
+        *,
+        session_bad: bool = False,
+        opc_status: str | None = None,
+        reconnect_in_progress: bool = False,
     ) -> dict[Hashable, Any]:
         """
         Process prediction confidence based on OPC write operation success.
@@ -352,22 +464,32 @@ class OPC(SientiaMonitoring):
             This allows downstream systems to handle data quality appropriately.
         """
 
-        message = 'Some data could not be written to OPC servers'
-
         if not success:
-            data['prediction_confidence'] = OPC_WRITTING_ERROR_CONFIDENCE
+            comment_parts: list[str] = []
-            data['comments'] = message
+            confidence = OPC_WRITTING_ERROR_CONFIDENCE
+
+            if session_bad:
+                comment_parts.append(_opc_session_bad_comment(opc_status))
+                confidence = OPC_SESSION_BAD_CONFIDENCE
+            if reconnect_in_progress:
+                comment_parts.append(OPC_RECONNECT_IN_PROGRESS_COMMENT)
+                confidence = OPC_SESSION_BAD_CONFIDENCE
+            if not comment_parts:
+                comment_parts.append(OPC_WRITTING_ERROR_MESSAGE)
+
+            comments = OPC_COMMENT_SEPARATOR.join(comment_parts)
+            data['prediction_confidence'] = confidence
+            data['comments'] = comments
             self.debug(
-                f'{message}, setting confidence to  {OPC_WRITTING_ERROR_CONFIDENCE}.',
+                f'OPC write issues, confidence={confidence}, comments={comments}',
                 metadata,
             )
-
         else:
             self.debug('Data written to OPC servers successfully.', metadata)
 
         return data.to_dict()
 
-    async def close(self):
+    def close(self):
         """
         Gracefully shutdown all OPC server connections and cleanup resources.
 
@@ -388,4 +510,5 @@ class OPC(SientiaMonitoring):
             their current state and provides a clean shutdown experience.
         """
         for opc in self.opc_repository.values():
-            await opc.disconnect()
+            opc.disconnect()
+        self.opc_repository.clear()

laborious/activities/storage.py

@@ -1,7 +1,5 @@
 from temporalio import activity, workflow
 
-from laborious.utils.repository.minio_manager import MinioManager
-
 with workflow.unsafe.imports_passed_through():
     # Extend the Temporal Postgres activities for convenient query -> MinIO export
     import traceback
@@ -13,8 +11,9 @@ with workflow.unsafe.imports_passed_through():
     from sientia_do.notifications.models import NotificationLevel
     from sientia_do.observability.logger import Logger
     from sientia_do.observability.metrics_controller import MetricsController
-    from sientia_do.repository.minio_repository import MinioRepository
+    from sientia_do.observability.sientia_monitoring import SientiaMonitoring
-    from sientia_do.temporal.activities.postgres import Postgres
+    from sientia_do.repository.minio_repository_sync import MinioRepository
+    from sientia_do.temporal.activities.postgres_sync import Postgres
     from sientia_do.temporal.constants import now
 
     from laborious.utils.models.minio_dataframe_payload import MinioDataFramePayload
@@ -22,7 +21,7 @@ with workflow.unsafe.imports_passed_through():
 _LOAD_QUERY_OFFLOAD_SKIP_KEYS = frozenset({'model_name', 'key_prefix', 'size_threshold_bytes'})
 
 
-class Storage(Postgres, MinioManager):
+class Storage(Postgres, SientiaMonitoring):
     """
     Extensions for Postgres activities with a helper to export query results
     directly to MinIO as Parquet and return the object name.
@@ -60,14 +59,16 @@ class Storage(Postgres, MinioManager):
             metrics_controller=metrics_controller,
         )
 
-        MinioManager.__init__(
+        self.minio_repository = minio_repository
-            self, minio_repository, logger, notification_handler, metrics_controller
+        SientiaMonitoring.__init__(
+            self,
+            logger=logger,
+            notification_handler=notification_handler,
+            metrics_controller=metrics_controller,
         )
 
     @activity.defn(name='load_query_with_minio_offload')
-    async def load_query_with_minio_offload(
+    def load_query_with_minio_offload(self, input_data: dict[str, Any]) -> MinioDataFramePayload:
-        self, input_data: dict[str, Any]
-    ) -> MinioDataFramePayload:
         """
         Run the custom SQL load, then return a MinIO-aware dataframe wire dict.
 
@@ -88,7 +89,7 @@ class Storage(Postgres, MinioManager):
         metadata: dict = input_data.get('metadata', {})
         model_name = input_data['model_name']
 
-        rows = await self.load_custom_query(
+        rows = self.load_custom_query(
             input_data,
         )
         if not rows:
@@ -99,7 +100,7 @@ class Storage(Postgres, MinioManager):
         else:
             dataframe = pd.DataFrame(rows)
 
-        return await MinioDataFramePayload.from_dataframe(
+        return MinioDataFramePayload.from_dataframe(
             dataframe,
             minio_repo=self.minio_repository,
             workflow_metadata=metadata,
@@ -109,15 +110,29 @@ class Storage(Postgres, MinioManager):
         )
 
     @activity.defn(name='export_payload_to_postgres')
-    async def export_payload_to_postgres(self, input_data: dict[str, Any]) -> dict:
+    def export_payload_to_postgres(self, input_data: dict[str, Any]) -> dict:
         """
-        Export a payload to PostgreSQL.
+        Resolve a MinIO-aware payload into a DataFrame and persist it into PostgreSQL.
+
+        This activity accepts the serialized payload produced by previous steps
+        (inline dict or MinIO object reference), reconstructs the tabular data,
+        and delegates the final write to ``export_data_to_postgres`` using the
+        same input contract expected by the Postgres activity mixin.
+
+        Args:
+         - input_data (dict[str, Any]): Activity input containing ``data`` as a
+           ``MinioDataFramePayload``-compatible dict plus database write options
+           (schema/table/on_conflict/metadata and related fields).
+
+        Return:
+         dict: Result dictionary returned by ``export_data_to_postgres``, including
+         success status and optional write diagnostics.
         """
         metadata = input_data.get('metadata')
         payload = MinioDataFramePayload.from_dict(input_data['data'])
-        data = await payload.retrieve(self.minio_repository, metadata)
+        data = payload.retrieve(self.minio_repository, metadata)
 
-        return await self.export_data_to_postgres(
+        return self.export_data_to_postgres(
             {
                 **input_data,
                 'data': data,
@@ -125,7 +140,7 @@ class Storage(Postgres, MinioManager):
         )
 
     @activity.defn(name='cleanup_minio_objects_expired')
-    async def cleanup_minio_objects_expired(self, input_data: dict[str, Any]) -> dict[str, Any]:
+    def cleanup_minio_objects_expired(self, input_data: dict[str, Any]) -> dict[str, Any]:
         """
         Delete objects under the given prefixes that are older than the retention window.
 
@@ -154,7 +169,7 @@ class Storage(Postgres, MinioManager):
             'deleted_count': 0,
         }
         try:
-            keys = await self.minio_repository.list_objects(
+            keys = self.minio_repository.list_objects(
                 prefix=prefix,
                 recursive=True,
                 metadata=metadata,
@@ -166,7 +181,7 @@ class Storage(Postgres, MinioManager):
                         continue
                     if ts >= cutoff:
                         continue
-                    await self.minio_repository.delete_file(
+                    self.minio_repository.delete_file(
                         object_name=key,
                         metadata=metadata,
                     )
@@ -184,7 +199,7 @@ class Storage(Postgres, MinioManager):
                 report['deleted_count'] += 1
         except Exception as e:
             trace = traceback.format_exc()
-            await self.send_notification_async(
+            self.send_notification(
                 metadata=metadata,
                 notification_id='ERROR_CLEANUP_MINIO_OBJECTS_EXPIRED',
                 message=f'Error cleaning up MinIO objects: {e}',
@@ -201,9 +216,16 @@ class Storage(Postgres, MinioManager):
         return report
 
     def close(self) -> None:
-        """Close Storage resources (MinIO client and Postgres engine)."""
+        """
-        Postgres.close(self)
+        Shutdown Storage resources in deterministic order.
-        MinioManager.close(self)
 
-    def __del__(self):
+        The method first closes Postgres resources via ``Postgres.close`` (engine,
-        self.close()
+        sessions, and monitoring hooks), then closes the optional MinIO repository
+        and clears the local reference to avoid accidental reuse after shutdown.
+        """
+        Postgres.close(self)
+        if self.minio_repository is not None:
+            try:
+                self.minio_repository.close()
+            finally:
+                self.minio_repository = None

laborious/metrics.py

@@ -89,6 +89,40 @@ OPC_CONNECTION_STATUS = Gauge(
     ['pod_id', 'server_name', 'server_url'],
 )
 
+_OPC_SESSION_DEBUG_LABELS = ['pod_id', 'server_name', 'runtime', 'opc_server_id', 'session_id']
+
+OPC_SESSION_CREATED_TOTAL = Counter(
+    'opc_session_created_total',
+    'OPC UA sessions established (after successful connect)',
+    _OPC_SESSION_DEBUG_LABELS,
+)
+
+OPC_SESSION_CLOSED_TOTAL = Counter(
+    'opc_session_closed_total',
+    'OPC UA client disconnects completed (session tear-down initiated)',
+    _OPC_SESSION_DEBUG_LABELS,
+)
+
+OPC_SESSION_REVISED_TIMEOUT_MS = Gauge(
+    'opc_session_revised_timeout_milliseconds',
+    'Server-revised OPC UA session timeout (RevisedSessionTimeout) in ms after connect',
+    _OPC_SESSION_DEBUG_LABELS,
+)
+
+OPC_WRITE_ATTEMPT_LABELS = [*_OPC_SESSION_DEBUG_LABELS, 'model_id', 'model_name', 'result']
+
+OPC_WRITE_ATTEMPTS_TOTAL = Counter(
+    'opc_write_attempts_total',
+    'OPC UA write attempts with session and outcome (result=OK or exception class name)',
+    OPC_WRITE_ATTEMPT_LABELS,
+)
+
+OPC_WRITE_INTER_ARRIVAL_OVER_SESSION_TIMEOUT_TOTAL = Counter(
+    'opc_write_inter_arrival_over_session_timeout_total',
+    'Successful writes where seconds since the previous successful write exceeded RevisedSessionTimeout (ms)',
+    _OPC_SESSION_DEBUG_LABELS,
+)
+
 # ================== Model metrics ==================
 
 MODEL_READ_LAG = Histogram(

laborious/utils/connectors_config.py

@@ -5,29 +5,63 @@ from typing import Any
 
 def build_mlflow_config() -> dict[str, Any]:
     """
-    Build MLFlow server configuration from environment variables.
+    Read MLflow tracking and registry credentials from the environment.
 
-    This function constructs an MLFlow configuration dictionary from
+    Used by ``Activities`` when constructing ``SientiaMLflowRepository``. The ``url`` value is the
-    environment variables with sensible defaults for local development.
+    same string workers and notebooks should use for ``MLFLOW_TRACKING_URI``-style clients.
-    It handles server connection and authentication parameters.
 
     Environment Variables:
-        MLFLOW_HOST: MLFlow server hostname (default: http://localhost)
+        MLFLOW_URL: Host with scheme
-        MLFLOW_PORT: MLFlow server port (default: 5080)
+        MLFLOW_USERNAME: Basic-auth or service user (default: aignosi)
-        MLFLOW_USERNAME: MLFlow username (default: aignosi)
+        MLFLOW_PASSWORD: Password or token (default: aignosi)
-        MLFLOW_PASSWORD: MLFlow password (default: aignosi)
 
-    Returns:
+    Return:
-        dict: MLFlow configuration dictionary with all required parameters
+        dict[str, Any]: ``url``, ``username``, ``password``.
     """
     return {
-        'host': getenv('MLFLOW_HOST', 'http://localhost'),
+        'url': getenv('MLFLOW_URL', 'http://localhost:5080'),
-        'port': int(getenv('MLFLOW_PORT', '5080')),
         'username': getenv('MLFLOW_USERNAME', 'aignosi'),
         'password': getenv('MLFLOW_PASSWORD', 'aignosi'),
     }
 
 
+def build_plugin_store_config() -> dict[str, Any]:
+    """
+    Collect settings for ``PluginStore`` (Git-backed catalog + runtime install via pip).
+
+    Mirrors the model-manager service: the worker passes these kwargs into ``PluginStore`` after
+    ``install_runtime`` resolves wheels from the configured PyPI index. Missing optional env vars
+    become ``None`` so the store can run without auth in local dev.
+
+    Environment Variables:
+        STORE_BASE_URL: Git HTTP(S) server (e.g. Gitea) base URL (default: http://localhost:3000)
+        STORE_OWNER: Namespace or org owning the store repo (default: sientia)
+        STORE_REPO: Repository name (default: model-library-store)
+        STORE_BRANCH: Checkout branch; unset lets the client use default
+        STORE_USERNAME / STORE_PASSWORD: HTTP basic credentials for Git fetch
+        STORE_CACHE_TTL_SECONDS: Optional integer seconds for metadata cache TTL
+        PYPI_SERVER: Index URL for ``pip install`` during runtime install (default: http://localhost:5000)
+        PYPI_USERNAME / PYPI_PASSWORD: Optional index authentication
+
+    Return:
+        dict[str, Any]: Keys aligned with ``PluginStore`` constructor parameter names.
+    """
+
+    cache_ttl_seconds = getenv('STORE_CACHE_TTL_SECONDS')
+    return {
+        'base_url': getenv('STORE_BASE_URL', 'http://localhost:3000'),
+        'owner': getenv('STORE_OWNER', 'sientia'),
+        'repo': getenv('STORE_REPO', 'model-library-store'),
+        'username': getenv('STORE_USERNAME'),
+        'password': getenv('STORE_PASSWORD'),
+        'branch': getenv('STORE_BRANCH'),
+        'cache_ttl_seconds': int(cache_ttl_seconds) if cache_ttl_seconds else None,
+        'pypi_index_url': getenv('PYPI_SERVER', 'http://localhost:5000'),
+        'pypi_username': getenv('PYPI_USERNAME'),
+        'pypi_password': getenv('PYPI_PASSWORD'),
+    }
+
+
 def build_opc_config() -> dict[str, Any]:
     """
     Build OPC server configuration from environment variables.
@@ -73,14 +107,15 @@ def build_minio_config() -> dict[str, Any]:
     Build MinIO (S3-compatible) configuration from environment variables.
 
     Environment Variables:
-        MINIO_ENDPOINT: MinIO endpoint including scheme (default: http://localhost:9000)
+        MINIO_ENDPOINT_URL: Host:port or URL for the S3 API (default: http://localhost:9000)
         MINIO_ACCESS_KEY: Access key (default: minioadmin)
         MINIO_SECRET_KEY: Secret key (default: minioadmin)
-        MINIO_REGION: Region name for S3 client (default: us-east-1)
+        MINIO_DEFAULT_BUCKET: Default bucket for Laborious payloads (default: laborious)
-        MINIO_BUCKET_DEFAULT: Default bucket for uploads (default: laborious)
+        MINIO_RETENTION_HOURS: Offloaded object retention window (default: 24)
-        MINIO_SECURE: Whether to use HTTPS (default: false)
+        MINIO_SECURE: If ``true``, use HTTPS (default: false)
-    Returns:
+
-        dict: MinIO configuration dictionary
+    Return:
+        dict[str, Any]: Keys consumed by ``Activities`` / ``MinioRepository``.
     """
     return {
         'endpoint_url': getenv('MINIO_ENDPOINT_URL', 'http://localhost:9000'),

laborious/utils/models/minio_dataframe_payload.py

@@ -21,7 +21,7 @@ from typing import Any, Literal
 
 from pandas import DataFrame, read_parquet
 from sientia_do.observability.logger import Logger
-from sientia_do.repository.minio_repository import MinioRepository
+from sientia_do.repository.minio_repository_sync import MinioRepository
 from sientia_do.temporal.constants import DATETIME_FORMAT_FILENAME, DATETIME_FORMAT_WITH_TZ, now
 
 # Keys that are part of the serialized wire format (not arbitrary metadata).
@@ -89,7 +89,7 @@ class MinioDataFramePayload:
         metadata: dict[str, Any] | None = None,
     ) -> None:
         """
-        Emit debug logs only when logger is provided
+        Emit a debug message only when a logger instance is available.
 
         Args:
          - logger (Logger | None): Logger instance used for debug messages
@@ -182,7 +182,14 @@ class MinioDataFramePayload:
 
     def cleanup_prefix(self) -> str | None:
         """
-        Return True if cleanup is enabled for this payload.
+        Return the MinIO prefix eligible for retention cleanup.
+
+        Cleanup is only applicable when payload data was offloaded to MinIO
+        (``object_key`` present and inline ``data`` absent). Inline-only payloads
+        return ``None`` because there is no object tree to prune.
+
+        Return:
+         str | None: Prefix used by cleanup listing, or ``None`` when cleanup does not apply.
         """
         if self.object_key is not None and self.data is None:
             return self.object_prefix
@@ -190,12 +197,19 @@ class MinioDataFramePayload:
 
     def has_data(self) -> bool:
         """
-        Return True if the payload has some data internally or in MinIO.
+        Indicate whether the payload contains retrievable tabular content.
+
+        A payload is considered non-empty when either inline ``data`` exists
+        (and is not an empty dict) or an ``object_key`` is available for MinIO
+        download.
+
+        Return:
+         bool: ``True`` when data can be retrieved, ``False`` otherwise.
         """
         return (self.data is not None and self.data != {}) or self.object_key is not None
 
     @classmethod
-    async def from_dataframe(
+    def from_dataframe(
         cls,
         dataframe: DataFrame | None,
         minio_repo: MinioRepository,
@@ -274,7 +288,7 @@ class MinioDataFramePayload:
         dataframe.to_parquet(parquet_buffer, engine='pyarrow', index=True)
         file_bytes = parquet_buffer.getvalue()
 
-        upload_result = await minio_repo.upload_file(
+        upload_result = minio_repo.upload_file(
             file_bytes=file_bytes,
             relative_key=object_key,
             metadata=workflow_metadata,
@@ -299,7 +313,7 @@ class MinioDataFramePayload:
             status=status,
         )
 
-    async def retrieve(
+    def retrieve(
         self,
         minio_repo: MinioRepository,
         workflow_metadata: dict[str, Any] | None = None,
@@ -336,7 +350,7 @@ class MinioDataFramePayload:
             f'MinioDataFramePayload.retrieve downloading object from MinIO: {self.object_key}',
             workflow_metadata,
         )
-        file_bytes = await minio_repo.download_file(
+        file_bytes = minio_repo.download_file(
             object_name=self.object_key, metadata=workflow_metadata
         )
         df = read_parquet(BytesIO(file_bytes))

laborious/utils/repository/minio_manager.py

@@ -1,32 +0,0 @@
-from sientia_do.notifications.handlers import NotificationHandler
-from sientia_do.observability.logger import Logger
-from sientia_do.observability.metrics_controller import MetricsController
-from sientia_do.observability.sientia_monitoring import SientiaMonitoring
-from sientia_do.repository.minio_repository import MinioRepository
-
-
-class MinioManager(SientiaMonitoring):
-    minio_repository: MinioRepository | None = None
-
-    def __init__(
-        self,
-        minio_repository: MinioRepository | None = None,
-        logger: Logger | None = None,
-        notification_handler: NotificationHandler | None = None,
-        metrics_controller: MetricsController | None = None,
-    ):
-        if self.minio_repository is None:
-            self.minio_repository = minio_repository
-        SientiaMonitoring.__init__(self, logger, notification_handler, metrics_controller)
-
-    def close(self) -> None:
-        """
-        Close the MinioManager and clean up resources.
-        """
-        if self.minio_repository is not None:
-            try:
-                self.minio_repository.close()
-            finally:
-                self.minio_repository = None
-
-        SientiaMonitoring.shutdown(self)

laborious/worker/prepare_worker.py

@@ -1,73 +0,0 @@
-import os
-import re
-from collections.abc import Sequence
-from typing import Any
-
-from sientia_do.observability.logger import Logger
-from temporalio.client import Client
-from temporalio.worker import PollerBehaviorAutoscaling, Worker
-
-parameters = [
-    ('MAX_CONCURRENT_WORKFLOW_TASKS', '200'),
-    ('MAX_CONCURRENT_ACTIVITIES', '200'),
-    ('MAX_CONCURRENT_LOCAL_ACTIVITIES', '200'),
-    ('MAX_CACHED_WORKFLOWS', '200'),
-    ('WORKFLOW_POLLER_BEHAVIOUR_MINIMUM', '10'),
-    ('WORKFLOW_POLLER_BEHAVIOUR_INITIAL', '100'),
-    ('WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM', '200'),
-    ('ACTIVITY_POLLER_BEHAVIOUR_MINIMUM', '10'),
-    ('ACTIVITY_POLLER_BEHAVIOUR_INITIAL', '100'),
-    ('ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM', '200'),
-]
-
-
-def camel_to_snake(text: str) -> str:
-    """Convert camelCase or PascalCase to snake_case."""
-    text = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', text)
-    text = re.sub('([a-z0-9])([A-Z])', r'\1_\2', text)
-    return text.lower()
-
-
-def prepare_worker(
-    main_workflow: type,
-    other_workflows: Sequence[type],
-    activities: Sequence[Any],
-    temporal_client: Client,
-    logger: Logger,
-) -> Worker:
-    main_workflow_name = main_workflow.__name__.upper()
-
-    queue_name = f'{camel_to_snake(main_workflow.__name__)}-queue'
-
-    local_workflow_parameters = {}
-
-    for parameter in parameters:
-        local_workflow_parameters[parameter[0]] = int(
-            os.getenv(main_workflow_name + '_' + parameter[0], parameter[1])
-        )
-
-    logger.info(f'Preparing worker for {main_workflow_name} with queue {queue_name}')
-    logger.info(f'Worker runtime config: {local_workflow_parameters}')
-
-    return Worker(
-        temporal_client,
-        task_queue=queue_name,
-        workflows=[main_workflow, *other_workflows],
-        activities=[*activities],
-        max_concurrent_workflow_tasks=local_workflow_parameters['MAX_CONCURRENT_WORKFLOW_TASKS'],
-        max_concurrent_activities=local_workflow_parameters['MAX_CONCURRENT_ACTIVITIES'],
-        max_concurrent_local_activities=local_workflow_parameters[
-            'MAX_CONCURRENT_LOCAL_ACTIVITIES'
-        ],
-        max_cached_workflows=local_workflow_parameters['MAX_CACHED_WORKFLOWS'],
-        workflow_task_poller_behavior=PollerBehaviorAutoscaling(
-            minimum=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIOUR_MINIMUM'],
-            initial=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIOUR_INITIAL'],
-            maximum=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM'],
-        ),
-        activity_task_poller_behavior=PollerBehaviorAutoscaling(
-            minimum=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIOUR_MINIMUM'],
-            initial=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIOUR_INITIAL'],
-            maximum=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM'],
-        ),
-    )

laborious/worker/worker.py

@@ -1,32 +1,32 @@
 """
 Laborious Worker Module
 
-This module provides the main worker implementation for the Sientia DataOps Laborious system.
+Entry process that connects to Temporal, registers Laborious activities, and runs four workers in
-It orchestrates Temporal workers, manages task queues, and handles the lifecycle of
+parallel. Each worker shares the same ``Activities`` instance (single Postgres pool, single MLflow
-prediction and retraining workflows.
+repository, single PluginStore handle) but polls a different task queue.
 
-The worker supports multiple task queues:
+Task queues (see ``sientia_do.temporal.worker.prepare_worker``):
-- predictions_batch-queue: Handles batch prediction workflows (heavy workload)
+- ``predictions_batch-{runtime}-queue`` + sub-workflows on the same queue (ML-heavy path).
-  Includes activities for MLFlow, data quality gates, OPC export, PI Web API export, and PostgreSQL
+- ``minimal_retrain-{runtime}-queue`` (retrain + promote + export).
-- minimal_retrain-queue: Handles model retraining workflows
+- ``drift-queue`` and ``simple_metrics-queue`` without a runtime suffix so existing schedulers
-- drift-queue: Handles drift detection workflows
+  keep stable queue names.
-- simple_metrics-queue: Handles simple metrics calculation workflows
 
-Key Features:
+Bootstrap order:
-- Resource-based scaling with WorkerTuner (CPU and memory aware)
+1. Prometheus app metrics and Mongo-backed notification handler.
-- Automatic polling scaling with PollerBehaviorAutoscaling
+2. ``RUNTIME`` validation and ``PluginStore.install_runtime`` so ``SientiaModel`` code is importable.
-- Prometheus metrics integration
+3. ``Activities`` construction (builds ``SientiaMLflowRepository`` internally from env).
-- Comprehensive error handling and logging
+4. OPC client initialization inside activities.
-- Graceful shutdown with cleanup
+5. Temporal ``Runtime`` with SDK Prometheus bind, client connect, then ``prepare_worker`` per workflow.
-- Multiple worker instances for different workflow types
+
+Shutdown closes workers, notifications, activities (pools + OPC), and clears ``app_up``.
 
 Environment Variables:
-- TEMPORAL_HOST: Temporal server address (default: localhost:7233)
+- RUNTIME: Required non-empty string passed to ``install_runtime``.
-- TEMPORAL_NAMESPACE: Temporal namespace (default: laborious)
+- STORE_* / PYPI_*: Plugin store and private index (see ``build_plugin_store_config``).
-- POD_ID: Kubernetes pod identifier for metrics
+- TEMPORAL_HOST, TEMPORAL_NAMESPACE: Cluster connection.
-- HTTP_METRICS_PORT: Prometheus metrics server port (default: 9090)
+- POD_ID, HTTP_METRICS_PORT, HTTP_SDK_METRICS_PORT: Observability.
-- HTTP_SDK_METRICS_PORT: Temporal SDK metrics port (default: 9091)
+- PROJECT_NAME, MONGODB_*: Notifications (via ``build_mongodb_config`` in handler).
-- PROJECT_NAME: Project name for notifications (default: laborious)
+- POSTGRES_*, MINIO_*, OPC_*, PI_WEB_API_*, MLFLOW_*: Passed through ``Activities`` helpers.
 """
 
 from temporalio import client, workflow
@@ -40,20 +40,22 @@ with workflow.unsafe.imports_passed_through():
     from prometheus_client import start_http_server
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
     from sientia_do.observability.logger import get_logger
+    from sientia_do.observability.metrics_controller import MetricsController
+    from sientia_do.temporal.worker.prepare_worker import prepare_worker
     from sientia_do.utils.connectors_config import (
         build_api_config,
         build_mongodb_config,
         build_postgres_config,
     )
+    from sientia_model.model_repository.plugin_store import PluginStore
 
     from laborious import metrics
     from laborious.activities.activities import Activities
     from laborious.utils.connectors_config import (
         build_minio_config,
-        build_mlflow_config,
         build_opc_config,
+        build_plugin_store_config,
     )
-    from laborious.worker.prepare_worker import prepare_worker
     from laborious.workflows.drift import Drift
     from laborious.workflows.minimal_retrain import MinimalRetrain
     from laborious.workflows.predictions_batch import PredictionsBatch
@@ -63,29 +65,24 @@ with workflow.unsafe.imports_passed_through():
     )
     from laborious.workflows.sub_workflows.prediction_process import PredictionProcess
 
-POD_ID = os.getenv('POD_ID')
+POD_ID = os.getenv('HOSTNAME')
 SDK_METRICS_PORT = int(os.getenv('HTTP_SDK_METRICS_PORT', '9091'))
 
 
 async def main():
     """
-    Main entry point for the Laborious worker application.
+    Run the full worker lifecycle: metrics, notifications, runtime install, workers, gather.
 
-    This function initializes and starts all components of the worker:
+    Exits the process with code 0 on normal completion of all worker tasks, or 1 after logging
-    1. Sets up logging and metadata
+    if any worker raises. ``finally`` always shuts down notifications and activities and sets
-    2. Starts Prometheus metrics server
+    ``app_up`` to 0 before ``sys.exit``.
-    3. Initializes notification handler
-    4. Creates and configures activities
-    5. Initializes OPC connections
-    6. Starts Temporal client and workers
-    7. Manages worker lifecycle and graceful shutdown
-
-    The function runs indefinitely until interrupted or an error occurs.
-    On error, it performs cleanup and exits with a non-zero status code.
 
     Raises:
-        Exception: Any unhandled exception during worker execution
+        Exception: Propagated from ``asyncio.gather`` only before ``finally`` handling; typically
-        SystemExit: On graceful shutdown or error conditions
+            workers run until cancelled.
+
+    Return:
+        None (process terminates via ``sys.exit`` from the ``finally`` block).
     """
     host = os.getenv('TEMPORAL_HOST', 'localhost:7233')
     logger = get_logger(__name__)
@@ -113,20 +110,68 @@ async def main():
         project_name=os.getenv('PROJECT_NAME', 'laborious'),
     )
 
+    metrics_controller = MetricsController(logger=logger)
+
+    runtime = os.getenv('RUNTIME', '').strip()
+    if not runtime:
+        logger.custom_critical(
+            'RUNTIME environment variable is required and must be non-empty',
+            metadata,
+        )
+        metrics.APP_UP.labels(pod_id=POD_ID).set(0)
+        sys.exit(1)
+
+    metadata_runtime = {**metadata, 'runtime': runtime}
+    logger.custom_info(f'Installing PluginStore runtime: {runtime}', metadata_runtime)
+
+    ps_cfg = build_plugin_store_config()
+    plugin_store = PluginStore(
+        base_url=ps_cfg['base_url'],
+        owner=ps_cfg['owner'],
+        repo=ps_cfg['repo'],
+        username=ps_cfg['username'],
+        password=ps_cfg['password'],
+        branch=ps_cfg['branch'],
+        cache_ttl_seconds=ps_cfg['cache_ttl_seconds'],
+        pypi_index_url=ps_cfg['pypi_index_url'],
+        pypi_username=ps_cfg['pypi_username'],
+        pypi_password=ps_cfg['pypi_password'],
+        logger=logger,
+        notification_handler=notification_handler,
+        metrics_controller=metrics_controller,
+    )
+
+    if runtime == 'legacy':
+        to_install_runtime = 'single'
+    else:
+        to_install_runtime = runtime
+
+    try:
+        await plugin_store.install_runtime(
+            runtime_name=to_install_runtime, metadata=metadata_runtime
+        )
+    except Exception as exc:
+        logger.custom_critical(
+            f'Failed to install runtime {to_install_runtime}: {exc}', metadata_runtime
+        )
+        metrics.APP_UP.labels(pod_id=POD_ID).set(0)
+        sys.exit(1)
+
     logger.custom_info('Starting Activities...', metadata)
 
     activities = Activities(
         postgres_config=build_postgres_config(),
-        mlflow_config=build_mlflow_config(),
+        plugin_store=plugin_store,
         minio_config=build_minio_config(),
         opc_config=build_opc_config(),
         pi_web_api_config=build_api_config(),
         logger=logger,
         notification_handler=notification_handler,
+        metrics_controller=metrics_controller,
     )
 
     logger.custom_info('Initializing OPC...', metadata)
-    await activities.init_opc()
+    activities.init_opc()
 
     logger.custom_info(f'Starting SDK Metrics Server on port {SDK_METRICS_PORT}...', metadata)
 
@@ -159,6 +204,7 @@ async def main():
                 activities.export_data_to_postgres,
             ],
             logger=logger,
+            runtime=runtime,
         ),
         prepare_worker(
             temporal_client=temporal_client,
@@ -170,6 +216,7 @@ async def main():
                 activities.export_data_to_postgres,
             ],
             logger=logger,
+            runtime='core',
         ),
         prepare_worker(
             temporal_client=temporal_client,
@@ -182,6 +229,7 @@ async def main():
                 activities.export_data_to_postgres,
             ],
             logger=logger,
+            runtime='core',
         ),
         prepare_worker(
             temporal_client=temporal_client,
@@ -205,11 +253,13 @@ async def main():
                 activities.cleanup_minio_objects_expired,
                 activities.repeat_last_prediction,
                 activities.export_data_to_postgres,
+                activities.export_payload_to_postgres,
                 activities.write_metrics,
                 # Pi Web API
                 activities.write_pi_web_api_data,
             ],
             logger=logger,
+            runtime=runtime,
         ),
     ]
 
@@ -221,17 +271,13 @@ async def main():
 
     exit_code = 0
     try:
-        # This will run the workers and wait for them to complete.
-        # If an exception occurs in any of the worker handlers, it will be propagated here.
         await asyncio.gather(*handlers)
     except BaseException as e:  # NOSONAR
         logger.custom_error(f'An unhandled exception occurred: {e}', metadata)
         exit_code = 1
     finally:
-        if notification_handler:
         notification_handler.shutdown()
-        if activities:
+        activities.shutdown()
-            await activities.shutdown()
         metrics.APP_UP.labels(pod_id=POD_ID).set(0)  # Mark app as DOWN
         sys.exit(exit_code)
 

laborious/workflows/sub_workflows/format_and_export_prediction.py

@@ -161,7 +161,7 @@ class FormatAndExportPrediction:
 
             write_transformed_handler = None
 
-        opc_metrics = {}
+        opc_metrics: dict[str, dict[str, float | None]] = {}
 
         # write to pi web api
         if pi_web_api_output_config:

laborious/workflows/sub_workflows/prediction_process.py

@@ -250,7 +250,7 @@ class PredictionProcess:
     async def path_flag_handler(
         self,
         data: dict[str, Any],
-        path_flag: str,
+        path_flag: str | None,
         input_data: dict,
         confidence: int,
         last_timestamp: str,

pyproject.toml

@@ -121,6 +121,7 @@ markers = [
     "asyncio: marks tests as async",
     "integration: marks tests as integration tests",
     "unit: marks tests as unit tests",
+    "opc: marks E2E tests that use in-process asyncua + real OpcRepository",
 ]
 
 [tool.coverage.run]

requirements-dev.txt

@@ -18,3 +18,4 @@ testcontainers[postgres,minio] # PostgreSQL and MinIO containers for E2E tests
 # Development Tools
 ipython>=8.12.0          # Enhanced Python shell
 ipdb>=0.13.13            # IPython debugger
+ipykernel==6.30.1        # IPython kernel for Jupyter notebooks

requirements-light.txt

@@ -1,12 +0,0 @@
-temporalio
-psycopg2-binary
-sqlalchemy
-asyncua
-redis
-git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.10.4
-prometheus-client
-botocore
-boto3
-s3fs
-pyarrow
-mlflow

requirements-local.txt

@@ -0,0 +1,18 @@
+temporalio
+psycopg2-binary
+sqlalchemy
+asyncua==1.0.6
+redis
+git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.12.1
+git+ssh://git@github.com/Aignosi/sientia-model-library.git@0.10.0
+prometheus-client   
+botocore
+boto3
+s3fs
+pyarrow
+kaleido
+hyperopt
+shap
+pycurl
+scipy<1.14.0
+scikit-learn==1.5.2

requirements.txt

@@ -1,10 +1,10 @@
 temporalio
 psycopg2-binary
 sqlalchemy
-asyncua
+asyncua==1.0.6
 redis
-git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.10.4
+sientia_do>=1.12.1
-git+ssh://git@github.com/Aignosi/sientia-mlops-library.git@0.41.0
+sientia_model>=0.8.2
 prometheus-client
 botocore
 boto3

tests/conftest.py

@@ -1,6 +1,17 @@
 import os
-import sys
+
-from unittest.mock import MagicMock
+from sientia_do.temporal.activities.postgres_sync import Postgres
+
+
+def _noop_postgres_del(_self):
+    """
+    Unit tests use MagicMock metrics controllers; postgres_sync.Postgres.__del__ calls
+    close() during GC and triggers async shutdown. Explicit ``close()`` is covered in tests.
+    """
+    return None
+
+
+Postgres.__del__ = _noop_postgres_del  # type: ignore[method-assign]
 
 # The production code converts SIENTIA_MINIO_OFFLOAD_THRESHOLD_MEGABYTES to int at import-time.
 # Tests must set it to a valid integer string to avoid import errors.
@@ -47,13 +58,9 @@ class DummyMinioDataFramePayload:
 """
 Pytest configuration file with global mocks for external dependencies.
 
-This module mocks the 'sientia' module to avoid requiring its installation
+The historical ``sientia`` package is no longer imported by the codebase;
-during unit tests. The mock is registered in sys.modules before any test
+drift analysis lives in ``sientia_model.analytics.drift_analysis`` and is
-imports are executed.
+imported lazily inside Temporal activities. No global module-level mock is
+required here — unit tests that need to control ``DriftAnalysis`` outputs
+should patch ``laborious.activities.model_metrics.DriftAnalysis`` directly.
 """
-
-# Mock sientia module
-sientia_mock = MagicMock()
-sientia_mock.ModelAnalysis = MagicMock
-sys.modules['sientia'] = sientia_mock
-sys.modules['sientia.ModelAnalysis'] = MagicMock()

tests/laborious/activities/test_activities.py

@@ -1,6 +1,4 @@
-from unittest.mock import ANY, AsyncMock, MagicMock, patch
+from unittest.mock import ANY, MagicMock, patch
-
-from pytest import mark
 
 from laborious.activities.activities import Activities
 from laborious.activities.api import API
@@ -48,7 +46,8 @@ def test___init__(
         'secure': False,
     }
 
-    mlflow_config = {'host': 'localhost', 'port': 5000, 'username': 'mlflow', 'password': 'mlflow'}
+    mlflow_repository = MagicMock()
+    plugin_store = MagicMock()
 
     opc_config = {
         'bootstrap_servers': 'localhost:9092',
@@ -67,12 +66,13 @@ def test___init__(
 
     activities = Activities(
         postgres_config=postgres_config,
-        mlflow_config=mlflow_config,
+        plugin_store=plugin_store,
         minio_config=minio_config,
         opc_config=opc_config,
         pi_web_api_config=pi_web_api_config,
         logger=logger,
         notification_handler=notification_handler,
+        mlflow_repository=mlflow_repository,
     )
 
     assert isinstance(activities, Activities)
@@ -101,10 +101,8 @@ def test___init__(
 
     mock_mlflow_init.assert_called_once_with(
         ANY,
-        mlflow_host=mlflow_config['host'],
+        mlflow_repository=mlflow_repository,
-        mlflow_port=mlflow_config['port'],
+        plugin_store=plugin_store,
-        mlflow_username=mlflow_config['username'],
-        mlflow_password=mlflow_config['password'],
         minio_repository=mock_minio_repository.return_value,
         logger=logger,
         notification_handler=notification_handler,
@@ -156,7 +154,6 @@ def test___init__(
     )
 
 
-@mark.asyncio
 @patch('laborious.activities.activities.Storage')
 @patch('laborious.activities.activities.MLFlow')
 @patch('laborious.activities.activities.OPC')
@@ -164,7 +161,7 @@ def test___init__(
 @patch('laborious.activities.activities.ModelMetrics')
 @patch('laborious.activities.activities.API')
 @patch('laborious.activities.activities.MinioRepository')
-async def test_shutdown(
+def test_shutdown(
     _mock_minio_repository,
     mock_api_init,
     mock_model_metrics_init,
@@ -173,7 +170,7 @@ async def test_shutdown(
     mock_mlflow_init,
     mock_storage_init,
 ):
-    mock_opc_init.close = AsyncMock()
+    mock_opc_init.close = MagicMock()
     postgres_config = {
         'host': 'localhost',
         'port': 5432,
@@ -193,7 +190,8 @@ async def test_shutdown(
         'secure': False,
     }
 
-    mlflow_config = {'host': 'localhost', 'port': 5000, 'username': 'mlflow', 'password': 'mlflow'}
+    mlflow_repository = MagicMock()
+    plugin_store = MagicMock()
 
     opc_config = {
         'bootstrap_servers': 'localhost:9092',
@@ -212,7 +210,77 @@ async def test_shutdown(
 
     activities = Activities(
         postgres_config=postgres_config,
-        mlflow_config=mlflow_config,
+        plugin_store=plugin_store,
+        minio_config=minio_config,
+        opc_config=opc_config,
+        pi_web_api_config=pi_web_api_config,
+        logger=logger,
+        notification_handler=notification_handler,
+        mlflow_repository=mlflow_repository,
+    )
+
+    activities.shutdown()
+    mock_opc_init.close.assert_called_once()
+    mock_storage_init.close.assert_called_once()
+    mock_mlflow_init.close.assert_called_once()
+    mock_gates_init.close.assert_called_once()
+    mock_model_metrics_init.close.assert_called_once()
+    mock_api_init.close.assert_called_once()
+
+
+@patch('laborious.activities.activities.SientiaMLflowRepository')
+@patch('laborious.activities.activities.build_mlflow_config')
+@patch('laborious.activities.activities.Storage.__init__')
+@patch('laborious.activities.activities.MLFlow.__init__')
+@patch('laborious.activities.activities.OPC.__init__')
+@patch('laborious.activities.activities.Gates.__init__')
+@patch('laborious.activities.activities.ModelMetrics.__init__')
+@patch('laborious.activities.activities.API.__init__')
+@patch('laborious.activities.activities.MinioRepository')
+@patch('laborious.activities.activities.MetricsController')
+def test___init___builds_mlflow_repository_when_not_provided(
+    mock_metrics_controller,
+    mock_minio_repository,
+    _mock_api_init,
+    _mock_model_metrics_init,
+    _mock_gates_init,
+    _mock_opc_init,
+    _mock_mlflow_init,
+    _mock_storage_init,
+    mock_build_mlflow_config,
+    mock_mlflow_repository_cls,
+):
+    postgres_config = {
+        'host': 'localhost',
+        'port': 5432,
+        'user': 'postgres',
+        'password': 'postgres',
+        'dbname': 'postgres',
+        'min_connections': 1,
+        'max_connections': 10,
+    }
+    minio_config = {
+        'endpoint_url': 'localhost:9000',
+        'access_key': 'minio',
+        'secret_key': 'minio123',
+        'default_bucket': 'test',
+        'retention_hours': 24,
+        'secure': False,
+    }
+    opc_config = {'bootstrap_servers': 'localhost:9092', 'polling_time': 1000, 'group_id': 'test'}
+    pi_web_api_config = {'base_url': 'https://pi', 'auth_type': 'bearer', 'auth_token': 'token'}
+    logger = MagicMock()
+    notification_handler = MagicMock()
+    plugin_store = MagicMock()
+    mock_build_mlflow_config.return_value = {
+        'url': 'http://mlflow:80',
+        'username': 'u',
+        'password': 'p',
+    }
+
+    Activities(
+        postgres_config=postgres_config,
+        plugin_store=plugin_store,
         minio_config=minio_config,
         opc_config=opc_config,
         pi_web_api_config=pi_web_api_config,
@@ -220,10 +288,12 @@ async def test_shutdown(
         notification_handler=notification_handler,
     )
 
-    await activities.shutdown()
+    mock_build_mlflow_config.assert_called_once()
-    mock_opc_init.close.assert_called_once()
+    mock_mlflow_repository_cls.assert_called_once_with(
-    mock_storage_init.close.assert_called_once()
+        host='http://mlflow:80',
-    mock_mlflow_init.close.assert_called_once()
+        username='u',
-    mock_gates_init.close.assert_called_once()
+        password='p',
-    mock_model_metrics_init.close.assert_called_once()
+        logger=logger,
-    mock_api_init.close.assert_called_once()
+        notification_handler=notification_handler,
+        metrics_controller=mock_metrics_controller.return_value,
+    )

tests/laborious/activities/test_api.py

@@ -1,7 +1,6 @@
-from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
+from unittest.mock import ANY, MagicMock, call, patch
 
-import pytest_asyncio
+from pytest import fixture
-from pytest import fixture, mark
 from sientia_do.notifications.models import NotificationLevel
 
 from laborious.activities.api import API, PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
@@ -71,7 +70,7 @@ def test_get_pi_web_api_core_labels_without_operation_type(mock_pi_web_api_clien
         auth_token='test_token',
         logger=MagicMock(),
         notification_handler=MagicMock(),
-        metrics_controller=AsyncMock(),
+        metrics_controller=MagicMock(),
     )
     with patch.object(
         SientiaMonitoring,
@@ -105,7 +104,7 @@ def test_get_pi_web_api_core_labels_with_operation_type(mock_pi_web_api_client):
         auth_token='test_token',
         logger=MagicMock(),
         notification_handler=MagicMock(),
-        metrics_controller=AsyncMock(),
+        metrics_controller=MagicMock(),
     )
     with patch.object(
         SientiaMonitoring,
@@ -132,17 +131,17 @@ def test__init__():
         auth_token='test_token',
         logger=MagicMock(),
         notification_handler=MagicMock(),
-        metrics_controller=AsyncMock(),
+        metrics_controller=MagicMock(),
     )
 
     assert api.pi_web_api_client is not None
 
 
-@pytest_asyncio.fixture
+@fixture
 @patch('laborious.activities.api.PIWebAPIClient')
 def api(mock_pi_web_api_client):
     mock_client = MagicMock()
-    mock_client.write_value = AsyncMock()
+    mock_client.write_value = MagicMock()
     mock_client.close = MagicMock()
     mock_client.base_url = 'https://test-pi-server.com'
     mock_pi_web_api_client.return_value = mock_client
@@ -153,12 +152,12 @@ def api(mock_pi_web_api_client):
         auth_token='test_token',
         logger=MagicMock(),
         notification_handler=MagicMock(),
-        metrics_controller=AsyncMock(),
+        metrics_controller=MagicMock(),
     )
-    api_instance.send_notification_async = AsyncMock()
+    api_instance.send_notification = MagicMock()
     api_instance.info = MagicMock()
     api_instance.error = MagicMock()
-    api_instance.emit_metric = AsyncMock()
+    api_instance.emit_metric_sync = MagicMock()
     api_instance.get_core_labels = MagicMock(
         return_value={
             'pod_id': 'test_pod',
@@ -170,9 +169,8 @@ def api(mock_pi_web_api_client):
     return api_instance
 
 
-@mark.asyncio
 @patch('laborious.activities.api.DataFrame')
-async def test_write_pi_web_api_data_success(mock_dataframe, api, base_input_data):
+def test_write_pi_web_api_data_success(mock_dataframe, api, base_input_data):
     input_data = {
         **base_input_data,
         'pi_web_api_output_config': {
@@ -190,7 +188,7 @@ async def test_write_pi_web_api_data_success(mock_dataframe, api, base_input_dat
         [{'WebId': 'web_id_3', 'Errors': []}, {'WebId': 'web_id_4', 'Errors': []}],
     ]
 
-    result = await api.write_pi_web_api_data(input_data)
+    result = api.write_pi_web_api_data(input_data)
 
     api.pi_web_api_client.write_value.assert_has_calls(
         [
@@ -220,9 +218,8 @@ async def test_write_pi_web_api_data_success(mock_dataframe, api, base_input_dat
     }
 
 
-@mark.asyncio
 @patch('laborious.activities.api.DataFrame')
-async def test_write_pi_web_api_data_prediction_error(mock_dataframe, api, base_input_data):
+def test_write_pi_web_api_data_prediction_error(mock_dataframe, api, base_input_data):
     mock_dataframe.return_value = _create_mock_dataframe(
         {
             'prediction': [0.75],
@@ -233,9 +230,9 @@ async def test_write_pi_web_api_data_prediction_error(mock_dataframe, api, base_
 
     api.pi_web_api_client.write_value.side_effect = Exception('Prediction write failed')
 
-    result = await api.write_pi_web_api_data(base_input_data)
+    result = api.write_pi_web_api_data(base_input_data)
 
-    api.send_notification_async.assert_called_once_with(
+    api.send_notification.assert_called_once_with(
         metadata=metadata['metadata'],
         notification_id='WRITE_PI_WEB_API_PREDICTION_ERROR',
         message="Error writing prediction data to PI Web API: Prediction write failed\n Tags: {'tag1': 'web_id_1'}",
@@ -248,9 +245,8 @@ async def test_write_pi_web_api_data_prediction_error(mock_dataframe, api, base_
     assert api.pi_web_api_client.write_value.call_count == 1
 
 
-@mark.asyncio
 @patch('laborious.activities.api.DataFrame')
-async def test_write_pi_web_api_data_confidence_error(mock_dataframe, api, base_input_data):
+def test_write_pi_web_api_data_confidence_error(mock_dataframe, api, base_input_data):
     mock_dataframe.return_value = _create_mock_dataframe()
 
     # First call succeeds, second fails
@@ -259,9 +255,9 @@ async def test_write_pi_web_api_data_confidence_error(mock_dataframe, api, base_
         Exception('Confidence write failed'),
     ]
 
-    result = await api.write_pi_web_api_data(base_input_data)
+    result = api.write_pi_web_api_data(base_input_data)
 
-    api.send_notification_async.assert_called_once_with(
+    api.send_notification.assert_called_once_with(
         metadata=metadata['metadata'],
         notification_id='WRITE_PI_WEB_API_CONFIDENCE_ERROR',
         message="Error writing confidence data to PI Web API: Confidence write failed\n Tags: {'tag2': 'web_id_2'}",
@@ -278,9 +274,8 @@ async def test_write_pi_web_api_data_confidence_error(mock_dataframe, api, base_
     assert api.pi_web_api_client.write_value.call_count == 2
 
 
-@mark.asyncio
 @patch('laborious.activities.api.DataFrame')
-async def test_write_pi_web_api_data_empty_tags(mock_dataframe, api, base_input_data):
+def test_write_pi_web_api_data_empty_tags(mock_dataframe, api, base_input_data):
     input_data = {
         **base_input_data,
         'pi_web_api_output_config': {
@@ -298,7 +293,7 @@ async def test_write_pi_web_api_data_empty_tags(mock_dataframe, api, base_input_
         [],
     ]
 
-    result = await api.write_pi_web_api_data(input_data)
+    result = api.write_pi_web_api_data(input_data)
 
     api.pi_web_api_client.write_value.assert_has_calls(
         [
@@ -328,15 +323,35 @@ async def test_write_pi_web_api_data_empty_tags(mock_dataframe, api, base_input_
     }
 
 
-@mark.asyncio
+@patch('laborious.activities.api.DataFrame')
-async def test_close(api):
+def test_write_pi_web_api_data_updates_confidence_and_comments(
+    mock_dataframe, api, base_input_data
+):
+    mock_dataframe.return_value = _create_mock_dataframe()
+    api.pi_web_api_client.write_value.side_effect = [
+        [{'WebId': 'web_id_1', 'Errors': []}],
+        [{'WebId': 'web_id_2', 'Errors': []}],
+    ]
+    with patch.object(
+        api,
+        'process_pi_web_api_response',
+        new=MagicMock(side_effect=[(0.33, 'PI warning'), (0, '')]),
+    ) as process_mock:
+        result = api.write_pi_web_api_data(base_input_data)
+
+    assert process_mock.call_count == 2
+    assert result is not None
+
+
+@patch('laborious.activities.api.SientiaMonitoring.shutdown')
+def test_close(mock_shutdown, api):
     api.close()
 
     api.pi_web_api_client.close.assert_called_once()
+    mock_shutdown.assert_called_once_with(api)
 
 
-@mark.asyncio
+def test_process_pi_web_api_response_success(api):
-async def test_process_pi_web_api_response_success(api):
     """Test successful processing of PI Web API response with all tags written."""
     response_data = [
         {'WebId': 'web_id_1', 'Errors': []},
@@ -350,7 +365,7 @@ async def test_process_pi_web_api_response_success(api):
         'workflow_name': 'test_workflow',
     }
 
-    confidence, message = await api.process_pi_web_api_response(
+    confidence, message = api.process_pi_web_api_response(
         response_data=response_data,
         tags=tags,
         core_labels=core_labels,
@@ -359,9 +374,9 @@ async def test_process_pi_web_api_response_success(api):
 
     assert confidence == 0
     assert message == ''
-    assert api.emit_metric.call_count == 2
+    assert api.emit_metric_sync.call_count == 2
-    # Verify that emit_metric was called with correct tags structure
+    # Verify that emit_metric_sync was called with correct tags structure
-    call_args_list = api.emit_metric.call_args_list
+    call_args_list = api.emit_metric_sync.call_args_list
     assert len(call_args_list) == 2
     # Check that all calls include core_labels and tag_name
     for call_args in call_args_list:
@@ -369,8 +384,7 @@ async def test_process_pi_web_api_response_success(api):
         assert call_args.kwargs['tags']['tag_name'] in ['tag1', 'tag2']
 
 
-@mark.asyncio
+def test_process_pi_web_api_response_with_errors(api):
-async def test_process_pi_web_api_response_with_errors(api):
     """Test processing response with errors in some tags."""
     response_data = [
         {'WebId': 'web_id_1', 'Errors': ['Error writing tag']},
@@ -384,7 +398,7 @@ async def test_process_pi_web_api_response_with_errors(api):
         'workflow_name': 'test_workflow',
     }
 
-    confidence, message = await api.process_pi_web_api_response(
+    confidence, message = api.process_pi_web_api_response(
         response_data=response_data,
         tags=tags,
         core_labels=core_labels,
@@ -396,11 +410,10 @@ async def test_process_pi_web_api_response_with_errors(api):
         message
         == "The number of written tags does not match the number of tag names: Expected ['tag1', 'tag2'] tags, but ['tag2'] tags were written."
     )
-    assert api.emit_metric.call_count == 2
+    assert api.emit_metric_sync.call_count == 2
 
 
-@mark.asyncio
+def test_process_pi_web_api_response_missing_tags(api):
-async def test_process_pi_web_api_response_missing_tags(api):
     """Test processing response when number of written tags doesn't match expected."""
     response_data = [
         {'WebId': 'web_id_1', 'Errors': []},
@@ -413,7 +426,7 @@ async def test_process_pi_web_api_response_missing_tags(api):
         'workflow_name': 'test_workflow',
     }
 
-    confidence, message = await api.process_pi_web_api_response(
+    confidence, message = api.process_pi_web_api_response(
         response_data=response_data,
         tags=tags,
         core_labels=core_labels,
@@ -425,14 +438,13 @@ async def test_process_pi_web_api_response_missing_tags(api):
         message
         == "The number of written tags does not match the number of tag names: Expected ['tag1', 'tag2'] tags, but ['tag1'] tags were written."
     )
-    api.send_notification_async.assert_called_once()
+    api.send_notification.assert_called_once()
-    call_args = api.send_notification_async.call_args
+    call_args = api.send_notification.call_args
     assert call_args.kwargs['notification_id'] == 'WRITE_PI_WEB_API_PREDICTION_ERROR'
     assert call_args.kwargs['level'] == NotificationLevel.ERROR
 
 
-@mark.asyncio
+def test_process_pi_web_api_response_missing_webid(api):
-async def test_process_pi_web_api_response_missing_webid(api):
     """Test processing response when WebId is missing in response item."""
     response_data = [
         {'Errors': []},
@@ -446,7 +458,7 @@ async def test_process_pi_web_api_response_missing_webid(api):
         'workflow_name': 'test_workflow',
     }
 
-    confidence, message = await api.process_pi_web_api_response(
+    confidence, message = api.process_pi_web_api_response(
         response_data=response_data,
         tags=tags,
         core_labels=core_labels,
@@ -461,8 +473,7 @@ async def test_process_pi_web_api_response_missing_webid(api):
     api.error.assert_any_call('The response did not contain some WebIds', metadata['metadata'])
 
 
-@mark.asyncio
+def test_process_pi_web_api_response_missing_tag_name(api):
-async def test_process_pi_web_api_response_missing_tag_name(api):
     """Test processing response when tag name is not found for WebId."""
     response_data = [
         {'WebId': 'unknown_web_id', 'Errors': []},
@@ -475,7 +486,7 @@ async def test_process_pi_web_api_response_missing_tag_name(api):
         'workflow_name': 'test_workflow',
     }
 
-    confidence, message = await api.process_pi_web_api_response(
+    confidence, message = api.process_pi_web_api_response(
         response_data=response_data,
         tags=tags,
         core_labels=core_labels,

tests/laborious/activities/test_gates.py

@@ -1,8 +1,9 @@
-from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
+from unittest.mock import ANY, MagicMock, call, patch
 
 from pandas import DataFrame
-from pytest import fixture, mark
+from pytest import fixture
 from sientia_do.notifications.models import NotificationLevel
+from sientia_do.observability.sientia_monitoring import SientiaMonitoring
 
 from laborious.activities.gates import Gates
 
@@ -17,17 +18,17 @@ def _passthrough_from_dict():
 
 def _minio_payload(retrieve_return, status=None):
     """
-    Build a MinioDataFramePayload-like test double with async retrieve.
+    Build a MinioDataFramePayload-like test double with retrieve.
 
     Args:
-        retrieve_return: Value returned from await retrieve(minio_repo, metadata).
+        retrieve_return: Value returned from retrieve(minio_repo, metadata).
         status: Optional status dict for MLflow response gate (payload.status).
 
     Return:
         MagicMock: Object with async retrieve and optional status.
     """
     p = MagicMock()
-    p.retrieve = AsyncMock(return_value=retrieve_return)
+    p.retrieve = MagicMock(return_value=retrieve_return)
     p.status = status
     return p
 
@@ -37,7 +38,7 @@ def gates_activity():
     gates = Gates(
         logger=MagicMock(),
         notification_handler=MagicMock(),
-        metrics_controller=AsyncMock(),
+        metrics_controller=MagicMock(),
     )
     gates.error = MagicMock()
     gates.debug = MagicMock()
@@ -45,8 +46,7 @@ def gates_activity():
     gates.warning = MagicMock()
     gates.critical = MagicMock()
     gates.send_notification = MagicMock()
-    gates.send_notification_async = AsyncMock()
+    gates.emit_metric_sync = MagicMock()
-    gates.emit_metric = AsyncMock()
     return gates
 
 
@@ -60,8 +60,7 @@ metadata = {
 }
 
 
-@mark.asyncio
+def test_input_gate_invalid_filter(gates_activity):
-async def test_input_gate_invalid_filter(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -71,7 +70,7 @@ async def test_input_gate_invalid_filter(gates_activity):
     }
 
     # Act
-    result = await gates_activity.input_gate(input_data)
+    result = gates_activity.input_gate(input_data)
 
     # Assert
     assert result == (None, 0, '')
@@ -80,9 +79,8 @@ async def test_input_gate_invalid_filter(gates_activity):
     )
 
 
-@mark.asyncio
 @patch('laborious.activities.gates.input_filter_functions')
-async def test_input_gate_filter_exception(mock_input_filter_functions, gates_activity):
+def test_input_gate_filter_exception(mock_input_filter_functions, gates_activity):
     # Arrange
     mock_input_filter_functions.__contains__.return_value = True
     mock_input_filter_functions.__getitem__.return_value = MagicMock(
@@ -96,11 +94,11 @@ async def test_input_gate_filter_exception(mock_input_filter_functions, gates_ac
     }
 
     # Act
-    result = await gates_activity.input_gate(input_data)
+    result = gates_activity.input_gate(input_data)
 
     # Assert
     assert result == (None, 0, '')
-    gates_activity.send_notification_async.assert_called_once_with(
+    gates_activity.send_notification.assert_called_once_with(
         metadata=metadata['metadata'],
         notification_id='INTPUT_GATE_ERROR__EMPTY_DATA',
         message="Error in filter EMPTY_DATA:{'POLICY': 'STOP', 'CONFIG': {}}: \n Test error",
@@ -110,8 +108,7 @@ async def test_input_gate_filter_exception(mock_input_filter_functions, gates_ac
     )
 
 
-@mark.asyncio
+def test_input_gate_no_filters(gates_activity):
-async def test_input_gate_no_filters(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -121,15 +118,14 @@ async def test_input_gate_no_filters(gates_activity):
     }
 
     # Act
-    result = await gates_activity.input_gate(input_data)
+    result = gates_activity.input_gate(input_data)
 
     # Assert
     assert result == (None, 0, '')
     gates_activity.debug.assert_called()
 
 
-@mark.asyncio
+def test_input_gate_with_filter(gates_activity):
-async def test_input_gate_with_filter(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -139,15 +135,14 @@ async def test_input_gate_with_filter(gates_activity):
     }
 
     # Act
-    result = await gates_activity.input_gate(input_data)
+    result = gates_activity.input_gate(input_data)
 
     # Assert
     assert result == ('STOP', -1, 'Input data with bad quality')
     gates_activity.debug.assert_called()
 
 
-@mark.asyncio
+def test_input_gate_with_filter_lowercase_keys(gates_activity):
-async def test_input_gate_with_filter_lowercase_keys(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -157,14 +152,13 @@ async def test_input_gate_with_filter_lowercase_keys(gates_activity):
     }
 
     # Act
-    result = await gates_activity.input_gate(input_data)
+    result = gates_activity.input_gate(input_data)
 
     # Assert
     assert result == ('STOP', -1, 'Input data with bad quality')
 
 
-@mark.asyncio
+def test_input_gate_with_filter_capitalized_keys(gates_activity):
-async def test_input_gate_with_filter_capitalized_keys(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -174,14 +168,13 @@ async def test_input_gate_with_filter_capitalized_keys(gates_activity):
     }
 
     # Act
-    result = await gates_activity.input_gate(input_data)
+    result = gates_activity.input_gate(input_data)
 
     # Assert
     assert result == ('STOP', -1, 'Input data with bad quality')
 
 
-@mark.asyncio
+def test_input_gate_with_filter_not_caught(gates_activity):
-async def test_input_gate_with_filter_not_caught(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -191,15 +184,14 @@ async def test_input_gate_with_filter_not_caught(gates_activity):
     }
 
     # Act
-    result = await gates_activity.input_gate(input_data)
+    result = gates_activity.input_gate(input_data)
 
     # Assert
     assert result == (None, 0, '')
     gates_activity.debug.assert_called()
 
 
-@mark.asyncio
+def test_mlflow_response_gate_invalid_filter(gates_activity):
-async def test_mlflow_response_gate_invalid_filter(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -213,15 +205,14 @@ async def test_mlflow_response_gate_invalid_filter(gates_activity):
     }
 
     # Act
-    result = await gates_activity.mlflow_response_gate(input_data)
+    result = gates_activity.mlflow_response_gate(input_data)
 
     # Assert
     assert result == (None, 0, '')
 
 
-@mark.asyncio
 @patch('laborious.activities.gates.mlflow_response_filter_functions')
-async def test_mlflow_response_gate_filter_exception(
+def test_mlflow_response_gate_filter_exception(
     mock_mlflow_response_filter_functions, gates_activity
 ):
     # Arrange
@@ -241,11 +232,11 @@ async def test_mlflow_response_gate_filter_exception(
     }
 
     # Act
-    result = await gates_activity.mlflow_response_gate(input_data)
+    result = gates_activity.mlflow_response_gate(input_data)
 
     # Assert
     assert result == (None, 0, '')
-    gates_activity.send_notification_async.assert_called_once_with(
+    gates_activity.send_notification.assert_called_once_with(
         metadata=metadata['metadata'],
         notification_id='MLFLOW_GATE_RESPONSE_FILTER__INVALID_FILTER',
         message="Error in filter INVALID_FILTER:{'POLICY': 'STOP'}: \n Test error",
@@ -255,8 +246,7 @@ async def test_mlflow_response_gate_filter_exception(
     )
 
 
-@mark.asyncio
+def test_mlflow_response_gate_no_filters(gates_activity):
-async def test_mlflow_response_gate_no_filters(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -270,15 +260,14 @@ async def test_mlflow_response_gate_no_filters(gates_activity):
     }
 
     # Act
-    result = await gates_activity.mlflow_response_gate(input_data)
+    result = gates_activity.mlflow_response_gate(input_data)
 
     # Assert
     assert result == (None, 0, '')
     gates_activity.debug.assert_called()
 
 
-@mark.asyncio
+def test_mlflow_response_gate_with_filter(gates_activity):
-async def test_mlflow_response_gate_with_filter(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -292,16 +281,15 @@ async def test_mlflow_response_gate_with_filter(gates_activity):
     }
 
     # Act
-    result = await gates_activity.mlflow_response_gate(input_data)
+    result = gates_activity.mlflow_response_gate(input_data)
 
     # Assert
     assert result == ('STOP', -1, 'API error occurred')
     gates_activity.debug.assert_called()
-    gates_activity.send_notification_async.assert_called()
+    gates_activity.send_notification.assert_called()
 
 
-@mark.asyncio
+def test_mlflow_response_gate_with_filter_capitalized_keys(gates_activity):
-async def test_mlflow_response_gate_with_filter_capitalized_keys(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -315,14 +303,13 @@ async def test_mlflow_response_gate_with_filter_capitalized_keys(gates_activity)
     }
 
     # Act
-    result = await gates_activity.mlflow_response_gate(input_data)
+    result = gates_activity.mlflow_response_gate(input_data)
 
     # Assert
     assert result == ('STOP', -1, 'API error occurred')
 
 
-@mark.asyncio
+def test_mlflow_response_gate_with_filter_not_caught(gates_activity):
-async def test_mlflow_response_gate_with_filter_not_caught(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -336,15 +323,14 @@ async def test_mlflow_response_gate_with_filter_not_caught(gates_activity):
     }
 
     # Act
-    result = await gates_activity.mlflow_response_gate(input_data)
+    result = gates_activity.mlflow_response_gate(input_data)
 
     # Assert
     assert result == (None, 0, '')
     gates_activity.debug.assert_called()
 
 
-@mark.asyncio
+def test_mlflow_content_gate_invalid_filter(gates_activity):
-async def test_mlflow_content_gate_invalid_filter(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -355,17 +341,14 @@ async def test_mlflow_content_gate_invalid_filter(gates_activity):
     }
 
     # Act
-    result = await gates_activity.mlflow_content_gate(input_data)
+    result = gates_activity.mlflow_content_gate(input_data)
 
     # Assert
     assert result == (None, 0, '')
 
 
-@mark.asyncio
 @patch('laborious.activities.gates.mlflow_content_filter_functions')
-async def test_mlflow_content_gate_filter_exception(
+def test_mlflow_content_gate_filter_exception(mock_mlflow_content_filter_functions, gates_activity):
-    mock_mlflow_content_filter_functions, gates_activity
-):
     # Arrange
     mock_mlflow_content_filter_functions.__contains__.return_value = True
     mock_mlflow_content_filter_functions.__getitem__.return_value = MagicMock(
@@ -380,12 +363,12 @@ async def test_mlflow_content_gate_filter_exception(
     }
 
     # Act
-    result = await gates_activity.mlflow_content_gate(input_data)
+    result = gates_activity.mlflow_content_gate(input_data)
 
     # Assert
     assert result == (None, 0, '')
     gates_activity.debug.assert_called()
-    gates_activity.send_notification_async.assert_called_once_with(
+    gates_activity.send_notification.assert_called_once_with(
         metadata=metadata['metadata'],
         notification_id='MLFLOW_GATE_CONTENT_FILTER__API_ERROR',
         message="Error in filter API_ERROR:{'POLICY': 'STOP'}: \n Test error",
@@ -395,8 +378,7 @@ async def test_mlflow_content_gate_filter_exception(
     )
 
 
-@mark.asyncio
+def test_mlflow_content_gate_no_filters(gates_activity):
-async def test_mlflow_content_gate_no_filters(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -407,15 +389,14 @@ async def test_mlflow_content_gate_no_filters(gates_activity):
     }
 
     # Act
-    result = await gates_activity.mlflow_content_gate(input_data)
+    result = gates_activity.mlflow_content_gate(input_data)
 
     # Assert
     assert result == (None, 0, '')
     gates_activity.debug.assert_called()
 
 
-@mark.asyncio
+def test_mlflow_content_gate_with_filter(gates_activity):
-async def test_mlflow_content_gate_with_filter(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -426,16 +407,15 @@ async def test_mlflow_content_gate_with_filter(gates_activity):
     }
 
     # Act
-    result = await gates_activity.mlflow_content_gate(input_data)
+    result = gates_activity.mlflow_content_gate(input_data)
 
     # Assert
     assert result == ('STOP', -1, 'Transformed data not passed the content filter')
     gates_activity.debug.assert_called()
-    gates_activity.send_notification_async.assert_called()
+    gates_activity.send_notification.assert_called()
 
 
-@mark.asyncio
+def test_mlflow_content_gate_with_filter_not_caught(gates_activity):
-async def test_mlflow_content_gate_with_filter_not_caught(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -446,15 +426,14 @@ async def test_mlflow_content_gate_with_filter_not_caught(gates_activity):
     }
 
     # Act
-    result = await gates_activity.mlflow_content_gate(input_data)
+    result = gates_activity.mlflow_content_gate(input_data)
 
     # Assert
     assert result == (None, 0, '')
     gates_activity.debug.assert_called()
 
 
-@mark.asyncio
+def test_mlflow_content_gate_filter_returns_false(gates_activity):
-async def test_mlflow_content_gate_filter_returns_false(gates_activity):
     input_data = {
         **metadata,
         'filters': {'NAN_VALUES': {'POLICY': 'STOP', 'CONFIG': {}}},
@@ -463,7 +442,7 @@ async def test_mlflow_content_gate_filter_returns_false(gates_activity):
         'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
     }
 
-    result = await gates_activity.mlflow_content_gate(input_data)
+    result = gates_activity.mlflow_content_gate(input_data)
 
     assert result == (None, 0, '')
     gates_activity.debug.assert_called()
@@ -525,8 +504,7 @@ def test_get_prediction_store_policy_valid_policy(gates_activity):
     assert policy_value == 1
 
 
-@mark.asyncio
+def test_format_prediction_no_timestamp(gates_activity):
-async def test_format_prediction_no_timestamp(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -545,7 +523,7 @@ async def test_format_prediction_no_timestamp(gates_activity):
     }
 
     # Act
-    result = await gates_activity.format_prediction(input_data)
+    result = gates_activity.format_prediction(input_data)
 
     # Assert
     assert result['prediction'] == {0: 1}
@@ -557,8 +535,7 @@ async def test_format_prediction_no_timestamp(gates_activity):
     assert result['comments'] == {0: ''}
 
 
-@mark.asyncio
+def test_format_prediction_with_timestamp_erl(gates_activity):
-async def test_format_prediction_with_timestamp_erl(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -585,7 +562,7 @@ async def test_format_prediction_with_timestamp_erl(gates_activity):
     }
 
     # Act
-    result = await gates_activity.format_prediction(input_data)
+    result = gates_activity.format_prediction(input_data)
 
     # Assert
     assert result['prediction'] == {0: 2, 1: 1}
@@ -597,8 +574,7 @@ async def test_format_prediction_with_timestamp_erl(gates_activity):
     assert result['comments'] == {0: '', 1: ''}
 
 
-@mark.asyncio
+def test_format_prediction_with_timestamp_lts(gates_activity):
-async def test_format_prediction_with_timestamp_lts(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -625,7 +601,7 @@ async def test_format_prediction_with_timestamp_lts(gates_activity):
     }
 
     # Act
-    result = await gates_activity.format_prediction(input_data)
+    result = gates_activity.format_prediction(input_data)
 
     # Assert
     assert result['prediction'] == {0: 3, 1: 2}
@@ -637,8 +613,7 @@ async def test_format_prediction_with_timestamp_lts(gates_activity):
     assert result['comments'] == {0: '', 1: ''}
 
 
-@mark.asyncio
+def test_format_prediction_with_timestamp_invalid_policy(gates_activity):
-async def test_format_prediction_with_timestamp_invalid_policy(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -663,16 +638,15 @@ async def test_format_prediction_with_timestamp_invalid_policy(gates_activity):
     gates_activity.get_prediction_store_policy = MagicMock(return_value=('invalid', 1))
 
     try:
-        await gates_activity.format_prediction(input_data)
+        gates_activity.format_prediction(input_data)
     except ValueError as e:
         assert str(e) == 'Invalid policy type: invalid'
     else:
         raise AssertionError('Expected ValueError')
 
 
-@mark.asyncio
+@patch('laborious.activities.gates.MinioDataFramePayload.from_dataframe', new_callable=MagicMock)
-@patch('laborious.activities.gates.MinioDataFramePayload.from_dataframe', new_callable=AsyncMock)
+def test_format_transformed_data_single_row(mock_from_dataframe, gates_activity):
-async def test_format_transformed_data_single_row(mock_from_dataframe, gates_activity):
     # Arrange
     payload_result = MagicMock()
     mock_from_dataframe.return_value = payload_result
@@ -691,7 +665,7 @@ async def test_format_transformed_data_single_row(mock_from_dataframe, gates_act
     }
 
     # Act
-    result = await gates_activity.format_transformed_data(input_data)
+    result = gates_activity.format_transformed_data(input_data)
 
     # Assert
     assert result is payload_result
@@ -705,9 +679,8 @@ async def test_format_transformed_data_single_row(mock_from_dataframe, gates_act
     gates_activity.info.assert_called()
 
 
-@mark.asyncio
+@patch('laborious.activities.gates.MinioDataFramePayload.from_dataframe', new_callable=MagicMock)
-@patch('laborious.activities.gates.MinioDataFramePayload.from_dataframe', new_callable=AsyncMock)
+def test_format_transformed_data_multiple_rows(mock_from_dataframe, gates_activity):
-async def test_format_transformed_data_multiple_rows(mock_from_dataframe, gates_activity):
     # Arrange
     payload_result = MagicMock()
     mock_from_dataframe.return_value = payload_result
@@ -732,7 +705,7 @@ async def test_format_transformed_data_multiple_rows(mock_from_dataframe, gates_
     }
 
     # Act
-    result = await gates_activity.format_transformed_data(input_data)
+    result = gates_activity.format_transformed_data(input_data)
 
     # Assert
     assert result is payload_result
@@ -746,9 +719,8 @@ async def test_format_transformed_data_multiple_rows(mock_from_dataframe, gates_
     gates_activity.info.assert_called()
 
 
-@mark.asyncio
+@patch('laborious.activities.gates.MinioDataFramePayload.from_dataframe', new_callable=MagicMock)
-@patch('laborious.activities.gates.MinioDataFramePayload.from_dataframe', new_callable=AsyncMock)
+def test_format_transformed_data_empty_data(mock_from_dataframe, gates_activity):
-async def test_format_transformed_data_empty_data(mock_from_dataframe, gates_activity):
     # Arrange
     payload_result = MagicMock()
     mock_from_dataframe.return_value = payload_result
@@ -760,7 +732,7 @@ async def test_format_transformed_data_empty_data(mock_from_dataframe, gates_act
     }
 
     # Act
-    result = await gates_activity.format_transformed_data(input_data)
+    result = gates_activity.format_transformed_data(input_data)
 
     # Assert
     assert result is payload_result
@@ -774,8 +746,7 @@ async def test_format_transformed_data_empty_data(mock_from_dataframe, gates_act
     gates_activity.info.assert_called()
 
 
-@mark.asyncio
+def test_format_default_prediction(gates_activity):
-async def test_format_default_prediction(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -786,7 +757,7 @@ async def test_format_default_prediction(gates_activity):
     }
 
     # Act
-    result = await gates_activity.format_default_prediction(input_data)
+    result = gates_activity.format_default_prediction(input_data)
 
     # Assert
     assert result['prediction'] == {0: 0}
@@ -799,8 +770,7 @@ async def test_format_default_prediction(gates_activity):
     gates_activity.debug.assert_called()
 
 
-@mark.asyncio
+def test_format_retrain_report(gates_activity):
-async def test_format_retrain_report(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -819,7 +789,7 @@ async def test_format_retrain_report(gates_activity):
     }
 
     # Act
-    result = await gates_activity.format_retrain_report(input_data)
+    result = gates_activity.format_retrain_report(input_data)
 
     # Assert
     assert result['model_id'] == {0: 'test_model'}
@@ -831,8 +801,7 @@ async def test_format_retrain_report(gates_activity):
     assert result['mlflow_experiment_id'] == {0: 'test_mlflow_experiment_id'}
 
 
-@mark.asyncio
+def test_format_retrain_report_failure(gates_activity):
-async def test_format_retrain_report_failure(gates_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -851,7 +820,7 @@ async def test_format_retrain_report_failure(gates_activity):
     }
 
     # Act
-    result = await gates_activity.format_retrain_report(input_data)
+    result = gates_activity.format_retrain_report(input_data)
 
     # Assert
     assert result['model_id'] == {0: 'test_model'}
@@ -865,9 +834,8 @@ async def test_format_retrain_report_failure(gates_activity):
     gates_activity.debug.assert_called()
 
 
-@mark.asyncio
 @patch('laborious.activities.gates.metrics')
-async def test_write_metrics(mock_metrics, gates_activity):
+def test_write_metrics(mock_metrics, gates_activity):
     """Test write_metrics method."""
     input_data = {
         **metadata,
@@ -878,7 +846,7 @@ async def test_write_metrics(mock_metrics, gates_activity):
         },
         'opc_metrics': {'server1': {'tag1': 0.1, 'tag2': 0.2}},
     }
-    await gates_activity.write_metrics(input_data)
+    gates_activity.write_metrics(input_data)
     core_tags = {
         'pod_id': gates_activity.pod_id,
         'runtime': gates_activity.runtime,
@@ -886,7 +854,7 @@ async def test_write_metrics(mock_metrics, gates_activity):
         'model_name': metadata['metadata']['model_name'],
         'workflow_name': metadata['metadata']['workflow_name'],
     }
-    gates_activity.emit_metric.assert_has_calls(
+    gates_activity.emit_metric_sync.assert_has_calls(
         [
             call(
                 metric_object=mock_metrics.PREDICTIONS_WRITTEN_COUNT,
@@ -894,7 +862,7 @@ async def test_write_metrics(mock_metrics, gates_activity):
             ),
         ]
     )
-    gates_activity.emit_metric.assert_has_calls(
+    gates_activity.emit_metric_sync.assert_has_calls(
         [
             call(
                 metric_object=mock_metrics.PREDICTION_CONFIDENCE_MONITOR,
@@ -904,7 +872,7 @@ async def test_write_metrics(mock_metrics, gates_activity):
             ),
         ]
     )
-    gates_activity.emit_metric.assert_has_calls(
+    gates_activity.emit_metric_sync.assert_has_calls(
         [
             call(
                 metric_object=mock_metrics.PREDICTION_RESPONSE_TIME_MONITOR,
@@ -914,7 +882,7 @@ async def test_write_metrics(mock_metrics, gates_activity):
             ),
         ]
     )
-    gates_activity.emit_metric.assert_has_calls(
+    gates_activity.emit_metric_sync.assert_has_calls(
         [
             call(
                 metric_object=mock_metrics.PREDICTION_OPC_WRITING_COUNT,
@@ -926,7 +894,7 @@ async def test_write_metrics(mock_metrics, gates_activity):
             ),
         ]
     )
-    gates_activity.emit_metric.assert_has_calls(
+    gates_activity.emit_metric_sync.assert_has_calls(
         [
             call(
                 metric_object=mock_metrics.PREDICTION_OPC_WRITING_RESPONSE_TIME_MONITOR,
@@ -940,7 +908,7 @@ async def test_write_metrics(mock_metrics, gates_activity):
             ),
         ]
     )
-    gates_activity.emit_metric.assert_has_calls(
+    gates_activity.emit_metric_sync.assert_has_calls(
         [
             call(
                 metric_object=mock_metrics.PREDICTION_OPC_WRITING_COUNT,
@@ -952,7 +920,7 @@ async def test_write_metrics(mock_metrics, gates_activity):
             ),
         ]
     )
-    gates_activity.emit_metric.assert_has_calls(
+    gates_activity.emit_metric_sync.assert_has_calls(
         [
             call(
                 metric_object=mock_metrics.PREDICTION_OPC_WRITING_RESPONSE_TIME_MONITOR,
@@ -968,9 +936,8 @@ async def test_write_metrics(mock_metrics, gates_activity):
     )
 
 
-@mark.asyncio
 @patch('laborious.activities.gates.metrics')
-async def test_write_metrics_with_none_opc_response_time(mock_metrics, gates_activity):
+def test_write_metrics_with_none_opc_response_time(mock_metrics, gates_activity):
     """Test write_metrics method with None response_time in opc_metrics."""
     input_data = {
         **metadata,
@@ -981,10 +948,10 @@ async def test_write_metrics_with_none_opc_response_time(mock_metrics, gates_act
         },
         'opc_metrics': {'server1': {'tag1': 0.1, 'tag2': None}},
     }
-    await gates_activity.write_metrics(input_data)
+    gates_activity.write_metrics(input_data)
 
     # Verify that metrics for tag1 are emitted
-    gates_activity.emit_metric.assert_any_call(
+    gates_activity.emit_metric_sync.assert_any_call(
         metric_object=mock_metrics.PREDICTION_OPC_WRITING_RESPONSE_TIME_MONITOR,
         method='observe',
         tags={
@@ -1002,7 +969,38 @@ async def test_write_metrics_with_none_opc_response_time(mock_metrics, gates_act
     # Verify that metrics for tag2 (with None response_time) are NOT emitted
     calls = [
         c
-        for c in gates_activity.emit_metric.call_args_list
+        for c in gates_activity.emit_metric_sync.call_args_list
         if len(c[1].get('tags', {})) > 0 and c[1]['tags'].get('tag') == 'tag2'
     ]
     assert len(calls) == 0, 'Metrics should not be emitted for None response_time'
+
+
+@patch.object(SientiaMonitoring, 'shutdown')
+def test_close_disposes_minio_repository(mock_shutdown):
+    """
+    ``Gates.close`` should close the optional MinIO client and clear the repository reference.
+    """
+    minio = MagicMock()
+    gates = Gates(
+        minio_repository=minio,
+        logger=MagicMock(),
+        notification_handler=MagicMock(),
+        metrics_controller=MagicMock(),
+    )
+    gates.close()
+    minio.close.assert_called_once()
+    assert gates.minio_repository is None
+    mock_shutdown.assert_called_once_with(gates)
+
+
+@patch.object(SientiaMonitoring, 'shutdown')
+def test_close_without_minio_repository(mock_shutdown):
+    """When no MinIO repository is configured, ``close`` only shuts down monitoring."""
+    gates = Gates(
+        minio_repository=None,
+        logger=MagicMock(),
+        notification_handler=MagicMock(),
+        metrics_controller=MagicMock(),
+    )
+    gates.close()
+    mock_shutdown.assert_called_once_with(gates)

tests/laborious/activities/test_model_metrics.py

@@ -1,8 +1,10 @@
-from unittest.mock import ANY, AsyncMock, MagicMock, patch
+from unittest.mock import ANY, MagicMock, patch
 
-from pandas import DataFrame
+from pandas import DataFrame, Timestamp
-from pytest import fixture, mark
+from pytest import fixture, raises
 from sientia_do.notifications.models import NotificationLevel
+from sientia_do.temporal.constants import DATETIME_FORMAT_WITH_TZ
+from sientia_model.analytics.drift_analysis import DriftInsufficientDataError
 
 from laborious.activities.model_metrics import ModelMetrics
 
@@ -12,7 +14,7 @@ def model_metrics_activity():
     model_metrics = ModelMetrics(
         logger=MagicMock(),
         notification_handler=MagicMock(),
-        metrics_controller=AsyncMock(),
+        metrics_controller=MagicMock(),
     )
     model_metrics.error = MagicMock()
     model_metrics.debug = MagicMock()
@@ -20,8 +22,7 @@ def model_metrics_activity():
     model_metrics.warning = MagicMock()
     model_metrics.critical = MagicMock()
     model_metrics.send_notification = MagicMock()
-    model_metrics.send_notification_async = AsyncMock()
+    model_metrics.emit_metric_sync = MagicMock()
-    model_metrics.emit_metric = AsyncMock()
     model_metrics.get_core_labels = MagicMock(
         return_value={
             'pod_id': 'test_pod',
@@ -29,7 +30,7 @@ def model_metrics_activity():
             'workflow_name': 'test_workflow',
         }
     )
-    model_metrics.observe_lag = AsyncMock()
+    model_metrics.observe_lag_sync = MagicMock()
     model_metrics.pod_id = 'test_pod'
     return model_metrics
 
@@ -44,8 +45,7 @@ metadata = {
 }
 
 
-@mark.asyncio
+def test_calculate_drift_invalid_chunk_period(model_metrics_activity):
-async def test_calculate_drift_invalid_chunk_period(model_metrics_activity):
     # Arrange
     input_data = {
         **metadata,
@@ -64,7 +64,7 @@ async def test_calculate_drift_invalid_chunk_period(model_metrics_activity):
 
     # Act & Assert
     try:
-        await model_metrics_activity.calculate_drift(input_data)
+        model_metrics_activity.calculate_drift(input_data)
     except ValueError as e:
         assert str(e) == 'Invalid chunk period: invalid, must be "min" or "s"'
         model_metrics_activity.error.assert_called_once_with(
@@ -74,40 +74,28 @@ async def test_calculate_drift_invalid_chunk_period(model_metrics_activity):
         raise AssertionError('Expected ValueError')
 
 
-@mark.asyncio
+def _sample_drift_metrics_df(ts: Timestamp) -> DataFrame:
-@patch('laborious.activities.model_metrics.DataFrame')
+    """Minimal analyzer-shaped dataframe (univariate row + columns the activity expects)."""
-@patch('laborious.activities.model_metrics.to_datetime')
+    return DataFrame(
-async def test_calculate_drift_with_reference_data(
-    mock_to_datetime, mock_dataframe, model_metrics_activity
-):
-    # Arrange
-    mock_to_datetime.return_value.dt.strftime.return_value = '2023-05-26 11:12:27'
-    mock_to_datetime.return_value.dt.tz_localize.return_value.dt.strftime.return_value = (
-        '2023-05-26 11:12:27+00:00'
-    )
-
-    mock_drift_df = MagicMock()
-    mock_drift_df.empty = False
-    mock_drift_df.drop.return_value = mock_drift_df
-    mock_drift_df.__getitem__.return_value.isin.return_value = [True]
-    mock_drift_df.__getitem__.return_value = mock_drift_df
-    mock_drift_df.__getitem__.return_value.dt.tz_localize.return_value.dt.strftime.return_value = (
-        '2023-05-26 11:12:27+00:00'
-    )
-    mock_drift_df.rename.return_value = mock_drift_df
-    mock_drift_df.drop_duplicates.return_value = mock_drift_df
-    mock_drift_df.to_dict.return_value = [
         {
-            'method': 'ks_test',
+            'timestamp': [ts],
-            'value': 0.5,
+            'feature': ['feature1'],
-            'feature': 'feature1',
+            'method': ['ks_test'],
-            'timestamp': '2023-05-26 11:12:27+00:00',
+            'value': [0.5],
-            'model_id': 'test_model_id',
+            'alert': [False],
-            'accurate': True,
+            'chunk_index': [0],
+            'chunk_start_date': [ts],
+            'chunk_end_date': [ts],
+            'threshold': [0.1],
+            'drift_type': ['univariate'],
         }
-    ]
+    )
 
-    model_metrics_activity.get_drift_metrics = AsyncMock(return_value=mock_drift_df)
+
+def test_calculate_drift_with_reference_data(model_metrics_activity):
+    ts = Timestamp('2023-05-26 11:12:27')
+    drift_df = _sample_drift_metrics_df(ts)
+    model_metrics_activity.get_drift_metrics = MagicMock(return_value=drift_df)
 
     reference_data = DataFrame(
         {
@@ -117,20 +105,11 @@ async def test_calculate_drift_with_reference_data(
         }
     )
 
-    mock_target_df = MagicMock()
-    mock_target_df.pivot.return_value = mock_target_df
-    mock_target_df.index = ['2023-05-26 11:12:27']
-    mock_target_df.reset_index.return_value = mock_target_df
-    mock_target_df.dropna.return_value = mock_target_df
-    mock_target_df.__getitem__.return_value.apply.return_value = ['2023-05-26 11:12:27']
-    mock_target_df.drop.return_value.columns = ['feature1']
-    mock_dataframe.return_value = mock_target_df
-
     input_data = {
         **metadata,
         'model_name': 'test_model',
         'model_id': 'test_model_id',
-        'reference_data': reference_data.to_dict(),
+        'reference_data': reference_data.to_dict('list'),
         'target_data': {
             'timestamp': ['2023-05-26 11:12:27'],
             'variable': ['feature1'],
@@ -141,85 +120,40 @@ async def test_calculate_drift_with_reference_data(
         'chunk_period': 'min',
     }
 
-    # Act
+    result = model_metrics_activity.calculate_drift(input_data)
-    result = await model_metrics_activity.calculate_drift(input_data)
 
-    # Assert
+    expected_timestamp = ts.tz_localize('UTC').strftime(DATETIME_FORMAT_WITH_TZ)
-    assert isinstance(result, list)
+    assert result == [
-    assert result == mock_drift_df.to_dict.return_value  # type: ignore[comparison-overlap]
-    model_metrics_activity.info.assert_called()
-    model_metrics_activity.get_drift_metrics.assert_called_once()
-    # Verify transformations were called
-    mock_drift_df.drop.assert_called_once_with(columns=['p_value'], inplace=True)
-    mock_drift_df.__getitem__.assert_called()
-    mock_drift_df.rename.assert_called_once_with(
-        columns={'metric': 'method', 'statistic': 'value'}, inplace=True
-    )
-    mock_drift_df.drop_duplicates.assert_called_once_with(
-        subset=['timestamp', 'method', 'feature'], keep='first', inplace=True
-    )
-    mock_drift_df.to_dict.assert_called_once_with(orient='records')
-
-
-@mark.asyncio
-@patch('laborious.activities.model_metrics.DataFrame')
-@patch('laborious.activities.model_metrics.to_datetime')
-async def test_calculate_drift_without_reference_data(
-    mock_to_datetime, mock_dataframe, model_metrics_activity
-):
-    # Arrange
-    mock_to_datetime.return_value.dt.strftime.return_value = '2023-05-26 11:12:27'
-    mock_to_datetime.return_value.dt.tz_localize.return_value.dt.strftime.return_value = (
-        '2023-05-26 11:12:27+00:00'
-    )
-
-    mock_drift_df = MagicMock()
-    mock_drift_df.empty = False
-    mock_drift_df.drop.return_value = mock_drift_df
-    mock_drift_df.__getitem__.return_value.isin.return_value = [True]
-    mock_drift_df.__getitem__.return_value = mock_drift_df
-    mock_drift_df.__getitem__.return_value.dt.tz_localize.return_value.dt.strftime.return_value = (
-        '2023-05-26 11:12:27+00:00'
-    )
-    mock_drift_df.rename.return_value = mock_drift_df
-    mock_drift_df.drop_duplicates.return_value = mock_drift_df
-    mock_drift_df.to_dict.return_value = [
         {
+            'timestamp': expected_timestamp,
+            'feature': 'feature1',
             'method': 'ks_test',
             'value': 0.5,
-            'feature': 'feature1',
+            'alert': False,
-            'timestamp': '2023-05-26 11:12:27+00:00',
+            'chunk_index': 0,
+            'chunk_start_date': ts.isoformat(),
+            'chunk_end_date': ts.isoformat(),
             'model_id': 'test_model_id',
-            'accurate': False,
+            'accurate': True,
         }
     ]
+    model_metrics_activity.info.assert_called()
+    model_metrics_activity.get_drift_metrics.assert_called_once()
 
-    model_metrics_activity.get_drift_metrics = AsyncMock(return_value=mock_drift_df)
 
+def test_calculate_drift_without_reference_data(model_metrics_activity):
+    # Ten rows so int(len * 0.3) >= 1 for the built-in reference slice.
+    ts_last = Timestamp('2023-05-26 11:12:36')
+    drift_df = _sample_drift_metrics_df(ts_last)
+    model_metrics_activity.get_drift_metrics = MagicMock(return_value=drift_df)
+
+    timestamps = [f'2023-05-26 11:12:{27 + i:02d}' for i in range(10)]
     target_data_dict = {
-        'timestamp': ['2023-05-26 11:12:27', '2023-05-26 11:12:28', '2023-05-26 11:12:29'],
+        'timestamp': timestamps,
-        'variable': ['feature1', 'feature1', 'feature1'],
+        'variable': ['feature1'] * 10,
-        'value': [1.0, 2.0, 3.0],
+        'value': [float(i) for i in range(10)],
     }
 
-    mock_target_df = MagicMock()
-    mock_target_df.pivot.return_value = mock_target_df
-    mock_target_df.index = ['2023-05-26 11:12:27', '2023-05-26 11:12:28', '2023-05-26 11:12:29']
-    mock_target_df.reset_index.return_value = mock_target_df
-    mock_target_df.dropna.return_value = mock_target_df
-    mock_target_df.sort_values.return_value = mock_target_df
-    mock_target_df.head.return_value = DataFrame(
-        {'timestamp': ['2023-05-26 11:12:27'], 'feature1': [1.0]}
-    )
-    mock_target_df.__getitem__.return_value.apply.return_value = [
-        '2023-05-26 11:12:27',
-        '2023-05-26 11:12:28',
-        '2023-05-26 11:12:29',
-    ]
-    mock_target_df.drop.return_value.columns = ['feature1']
-    mock_dataframe.return_value = mock_target_df
-    mock_dataframe.side_effect = lambda x=None: mock_target_df if x is not None else mock_target_df
-
     input_data = {
         **metadata,
         'model_name': 'test_model',
@@ -231,14 +165,25 @@ async def test_calculate_drift_without_reference_data(
         'chunk_period': 's',
     }
 
-    # Act
+    result = model_metrics_activity.calculate_drift(input_data)
-    result = await model_metrics_activity.calculate_drift(input_data)
 
-    # Assert
+    expected_timestamp = ts_last.tz_localize('UTC').strftime(DATETIME_FORMAT_WITH_TZ)
-    assert isinstance(result, list)
+    assert result == [
-    assert result == mock_drift_df.to_dict.return_value
+        {
+            'timestamp': expected_timestamp,
+            'feature': 'feature1',
+            'method': 'ks_test',
+            'value': 0.5,
+            'alert': False,
+            'chunk_index': 0,
+            'chunk_start_date': ts_last.isoformat(),
+            'chunk_end_date': ts_last.isoformat(),
+            'model_id': 'test_model_id',
+            'accurate': False,
+        }
+    ]
     model_metrics_activity.warning.assert_called()
-    model_metrics_activity.send_notification_async.assert_called_once_with(
+    model_metrics_activity.send_notification.assert_called_once_with(
         metadata=metadata['metadata'],
         notification_id='MODEL_METRICS_REFERENCE_DATA_WARNING',
         message='Using 30% first rows of target data as reference data',
@@ -246,28 +191,90 @@ async def test_calculate_drift_without_reference_data(
         level=NotificationLevel.WARNING,
         attachment_content=ANY,
     )
-    # Verify transformations were called
-    mock_drift_df.drop.assert_called_once_with(columns=['p_value'], inplace=True)
-    mock_drift_df.__getitem__.assert_called()
-    mock_drift_df.rename.assert_called_once_with(
-        columns={'metric': 'method', 'statistic': 'value'}, inplace=True
-    )
-    mock_drift_df.drop_duplicates.assert_called_once_with(
-        subset=['timestamp', 'method', 'feature'], keep='first', inplace=True
-    )
-    mock_drift_df.to_dict.assert_called_once_with(orient='records')
 
 
-@mark.asyncio
+def test_calculate_drift_empty_drift_df(model_metrics_activity):
+    """Empty analyzer merge yields no rows and no insufficient-data alert (lib owns that failure mode)."""
+    ts = Timestamp('2023-05-26 11:12:27')
+    empty_df = _sample_drift_metrics_df(ts).iloc[0:0]
+    model_metrics_activity.get_drift_metrics = MagicMock(return_value=empty_df)
+
+    reference_data = DataFrame(
+        {
+            'timestamp': ['2023-05-26 11:12:27'],
+            'target': [1.0],
+            'feature1': [1.0],
+        }
+    )
+
+    input_data = {
+        **metadata,
+        'model_name': 'test_model',
+        'model_id': 'test_model_id',
+        'reference_data': reference_data.to_dict(),
+        'target_data': {
+            'timestamp': ['2023-05-26 11:12:27'],
+            'variable': ['feature1'],
+            'value': [1.0],
+        },
+        'target_name': 'target',
+        'drift_metrics': ['ks_test'],
+        'chunk_period': 'min',
+    }
+
+    result = model_metrics_activity.calculate_drift(input_data)
+    assert result == []
+    model_metrics_activity.send_notification.assert_not_called()
+
+
+def test_calculate_drift_empty_after_timestamp_filter(model_metrics_activity):
+    """Rows dropped by target-window alignment yield an empty export list, not an insufficient-data error."""
+    drift_df = _sample_drift_metrics_df(Timestamp('2020-01-01'))
+    model_metrics_activity.get_drift_metrics = MagicMock(return_value=drift_df)
+
+    reference_data = DataFrame(
+        {
+            'timestamp': ['2023-05-26 11:12:27'],
+            'target': [1.0],
+            'feature1': [1.0],
+        }
+    )
+
+    input_data = {
+        **metadata,
+        'model_name': 'test_model',
+        'model_id': 'test_model_id',
+        'reference_data': reference_data.to_dict(),
+        'target_data': {
+            'timestamp': ['2023-05-26 11:12:27'],
+            'variable': ['feature1'],
+            'value': [1.0],
+        },
+        'target_name': 'target',
+        'drift_metrics': ['ks_test'],
+        'chunk_period': 'min',
+    }
+
+    result = model_metrics_activity.calculate_drift(input_data)
+    assert result == []
+    model_metrics_activity.send_notification.assert_not_called()
+
+
 @patch('laborious.activities.model_metrics.DataFrame')
 @patch('laborious.activities.model_metrics.to_datetime')
-async def test_calculate_drift_empty_drift_df(
+def test_calculate_drift_drift_insufficient_data_error_from_lib(
     mock_to_datetime, mock_dataframe, model_metrics_activity
 ):
-    # Arrange
+    """``DriftInsufficientDataError`` maps to MODEL_METRICS_DRIFT_INSUFFICIENT_DATA, not GET error."""
     mock_to_datetime.return_value.dt.strftime.return_value = '2023-05-26 11:12:27'
 
-    model_metrics_activity.get_drift_metrics = AsyncMock(return_value=DataFrame())
+    lib_msg = (
+        '[MODEL_METRICS_DRIFT_INSUFFICIENT_DATA] Drift analysis produced no time chunks '
+        "(chunk_period='min', analysis_rows=1)."
+    )
+    model_metrics_activity.get_drift_metrics = MagicMock(
+        side_effect=DriftInsufficientDataError(lib_msg, analysis_rows=1)
+    )
 
     reference_data = DataFrame(
         {
@@ -301,46 +308,24 @@ async def test_calculate_drift_empty_drift_df(
         'chunk_period': 'min',
     }
 
-    # Act
+    with raises(DriftInsufficientDataError, match='MODEL_METRICS_DRIFT_INSUFFICIENT_DATA'):
-    result = await model_metrics_activity.calculate_drift(input_data)
+        model_metrics_activity.calculate_drift(input_data)
 
-    # Assert
+    model_metrics_activity.error.assert_called_once_with(lib_msg, metadata['metadata'])
-    assert result == []
+    model_metrics_activity.send_notification.assert_called_once_with(
-    model_metrics_activity.warning.assert_called_with(
+        metadata=metadata['metadata'],
-        'No drift metrics found', metadata['metadata']
+        notification_id='MODEL_METRICS_DRIFT_INSUFFICIENT_DATA',
+        message=lib_msg,
+        block='model_metrics',
+        level=NotificationLevel.ERROR,
+        attachment_content=ANY,
     )
 
 
-@mark.asyncio
+def test_calculate_drift_success_min(model_metrics_activity):
-@patch('laborious.activities.model_metrics.DataFrame')
+    ts = Timestamp('2023-05-26 11:12:27')
-@patch('laborious.activities.model_metrics.to_datetime')
+    drift_df = _sample_drift_metrics_df(ts)
-async def test_calculate_drift_empty_after_timestamp_filter(
+    model_metrics_activity.get_drift_metrics = MagicMock(return_value=drift_df)
-    mock_to_datetime, mock_dataframe, model_metrics_activity
-):
-    # Arrange
-    mock_to_datetime.return_value.dt.strftime.return_value = '2023-05-26 11:12:27'
-
-    mock_drift_df = MagicMock()
-    mock_drift_df.empty = False
-    mock_drift_df.drop.return_value = mock_drift_df
-
-    # Set up __getitem__ to handle filtering - timestamp access returns series with isin=False
-    # and filtering returns empty DataFrame
-    mock_timestamp_series = MagicMock()
-    mock_timestamp_series.isin.return_value = [False]
-    mock_empty_df = MagicMock()
-    mock_empty_df.empty = True
-
-    def getitem_side_effect(key):
-        if key == 'timestamp':
-            return mock_timestamp_series
-        else:
-            # This is the filtering operation - return empty DataFrame
-            return mock_empty_df
-
-    mock_drift_df.__getitem__.side_effect = getitem_side_effect
-
-    model_metrics_activity.get_drift_metrics = AsyncMock(return_value=mock_drift_df)
 
     reference_data = DataFrame(
         {
@@ -350,20 +335,11 @@ async def test_calculate_drift_empty_after_timestamp_filter(
         }
     )
 
-    mock_target_df = MagicMock()
-    mock_target_df.pivot.return_value = mock_target_df
-    mock_target_df.index = ['2023-05-26 11:12:27']
-    mock_target_df.reset_index.return_value = mock_target_df
-    mock_target_df.dropna.return_value = mock_target_df
-    mock_target_df.__getitem__.return_value.apply.return_value = ['2023-05-26 11:12:27']
-    mock_target_df.drop.return_value.columns = ['feature1']
-    mock_dataframe.return_value = mock_target_df
-
     input_data = {
         **metadata,
         'model_name': 'test_model',
         'model_id': 'test_model_id',
-        'reference_data': reference_data.to_dict(),
+        'reference_data': reference_data.to_dict('list'),
         'target_data': {
             'timestamp': ['2023-05-26 11:12:27'],
             'variable': ['feature1'],
@@ -374,139 +350,31 @@ async def test_calculate_drift_empty_after_timestamp_filter(
         'chunk_period': 'min',
     }
 
-    # Act
+    result = model_metrics_activity.calculate_drift(input_data)
-    result = await model_metrics_activity.calculate_drift(input_data)
 
-    # Assert
+    expected_timestamp = ts.tz_localize('UTC').strftime(DATETIME_FORMAT_WITH_TZ)
-    assert result == []
+    assert result == [
-    model_metrics_activity.warning.assert_called_with(
-        'No drift metrics found after dropping rows where timestamp is not in target data',
-        metadata['metadata'],
-    )
-    # Verify transformations were called
-    mock_drift_df.drop.assert_called_once_with(columns=['p_value'], inplace=True)
-    mock_drift_df.__getitem__.assert_called()
-
-
-@mark.asyncio
-@patch('laborious.activities.model_metrics.DataFrame')
-@patch('laborious.activities.model_metrics.to_datetime')
-async def test_calculate_drift_success_min(
-    mock_to_datetime, mock_dataframe, model_metrics_activity
-):
-    # Arrange
-    mock_to_datetime.return_value.dt.strftime.return_value = '2023-05-26 11:12:27'
-    mock_to_datetime.return_value.dt.tz_localize.return_value.dt.strftime.return_value = (
-        '2023-05-26 11:12:27+00:00'
-    )
-
-    mock_drift_df = MagicMock()
-    mock_drift_df.empty = False
-    mock_drift_df.drop.return_value = mock_drift_df
-    mock_drift_df.__getitem__.return_value.isin.return_value = [True]
-    mock_drift_df.__getitem__.return_value = mock_drift_df
-    mock_drift_df.__getitem__.return_value.dt.tz_localize.return_value.dt.strftime.return_value = (
-        '2023-05-26 11:12:27+00:00'
-    )
-    mock_drift_df.rename.return_value = mock_drift_df
-    mock_drift_df.drop_duplicates.return_value = mock_drift_df
-    mock_drift_df.to_dict.return_value = [
         {
+            'timestamp': expected_timestamp,
+            'feature': 'feature1',
             'method': 'ks_test',
             'value': 0.5,
-            'feature': 'feature1',
+            'alert': False,
-            'timestamp': '2023-05-26 11:12:27+00:00',
+            'chunk_index': 0,
+            'chunk_start_date': ts.isoformat(),
+            'chunk_end_date': ts.isoformat(),
             'model_id': 'test_model_id',
             'accurate': True,
         }
     ]
-
-    model_metrics_activity.get_drift_metrics = AsyncMock(return_value=mock_drift_df)
-
-    reference_data = DataFrame(
-        {
-            'timestamp': ['2023-05-26 11:12:27'],
-            'target': [1.0],
-            'feature1': [1.0],
-        }
-    )
-
-    mock_target_df = MagicMock()
-    mock_target_df.pivot.return_value = mock_target_df
-    mock_target_df.index = ['2023-05-26 11:12:27']
-    mock_target_df.reset_index.return_value = mock_target_df
-    mock_target_df.dropna.return_value = mock_target_df
-    mock_target_df.__getitem__.return_value.isin.return_value = [True]
-    mock_target_df.drop.return_value.columns = ['feature1']
-    mock_dataframe.return_value = mock_target_df
-
-    input_data = {
-        **metadata,
-        'model_name': 'test_model',
-        'model_id': 'test_model_id',
-        'reference_data': reference_data.to_dict(),
-        'target_data': {
-            'timestamp': ['2023-05-26 11:12:27'],
-            'variable': ['feature1'],
-            'value': [1.0],
-        },
-        'target_name': 'target',
-        'drift_metrics': ['ks_test'],
-        'chunk_period': 'min',
-    }
-
-    # Act
-    result = await model_metrics_activity.calculate_drift(input_data)
-
-    # Assert
-    assert isinstance(result, list)
-    assert result == mock_drift_df.to_dict.return_value  # type: ignore[comparison-overlap]
     model_metrics_activity.info.assert_called()
     model_metrics_activity.get_drift_metrics.assert_called_once()
-    # Verify transformations were called
-    mock_drift_df.drop.assert_called_once_with(columns=['p_value'], inplace=True)
-    mock_drift_df.__getitem__.assert_called()
-    mock_drift_df.rename.assert_called_once_with(
-        columns={'metric': 'method', 'statistic': 'value'}, inplace=True
-    )
-    mock_drift_df.drop_duplicates.assert_called_once_with(
-        subset=['timestamp', 'method', 'feature'], keep='first', inplace=True
-    )
-    mock_drift_df.to_dict.assert_called_once_with(orient='records')
 
 
-@mark.asyncio
+def test_calculate_drift_success_s(model_metrics_activity):
-@patch('laborious.activities.model_metrics.DataFrame')
+    ts = Timestamp('2023-05-26 11:12:27')
-@patch('laborious.activities.model_metrics.to_datetime')
+    drift_df = _sample_drift_metrics_df(ts)
-async def test_calculate_drift_success_s(mock_to_datetime, mock_dataframe, model_metrics_activity):
+    model_metrics_activity.get_drift_metrics = MagicMock(return_value=drift_df)
-    # Arrange
-    mock_to_datetime.return_value.dt.strftime.return_value = '2023-05-26 11:12:27'
-    mock_to_datetime.return_value.dt.tz_localize.return_value.dt.strftime.return_value = (
-        '2023-05-26 11:12:27+00:00'
-    )
-
-    mock_drift_df = MagicMock()
-    mock_drift_df.empty = False
-    mock_drift_df.drop.return_value = mock_drift_df
-    mock_drift_df.__getitem__.return_value.isin.return_value = [True]
-    mock_drift_df.__getitem__.return_value = mock_drift_df
-    mock_drift_df.__getitem__.return_value.dt.tz_localize.return_value.dt.strftime.return_value = (
-        '2023-05-26 11:12:27+00:00'
-    )
-    mock_drift_df.rename.return_value = mock_drift_df
-    mock_drift_df.drop_duplicates.return_value = mock_drift_df
-    mock_drift_df.to_dict.return_value = [
-        {
-            'method': 'ks_test',
-            'value': 0.5,
-            'feature': 'feature1',
-            'timestamp': '2023-05-26 11:12:27+00:00',
-            'model_id': 'test_model_id',
-            'accurate': True,
-        }
-    ]
-
-    model_metrics_activity.get_drift_metrics = AsyncMock(return_value=mock_drift_df)
 
     reference_data = DataFrame(
         {
@@ -516,20 +384,11 @@ async def test_calculate_drift_success_s(mock_to_datetime, mock_dataframe, model
         }
     )
 
-    mock_target_df = MagicMock()
-    mock_target_df.pivot.return_value = mock_target_df
-    mock_target_df.index = ['2023-05-26 11:12:27']
-    mock_target_df.reset_index.return_value = mock_target_df
-    mock_target_df.dropna.return_value = mock_target_df
-    mock_target_df.__getitem__.return_value.isin.return_value = [True]
-    mock_target_df.drop.return_value.columns = ['feature1']
-    mock_dataframe.return_value = mock_target_df
-
     input_data = {
         **metadata,
         'model_name': 'test_model',
         'model_id': 'test_model_id',
-        'reference_data': reference_data.to_dict(),
+        'reference_data': reference_data.to_dict('list'),
         'target_data': {
             'timestamp': ['2023-05-26 11:12:27'],
             'variable': ['feature1'],
@@ -540,36 +399,36 @@ async def test_calculate_drift_success_s(mock_to_datetime, mock_dataframe, model
         'chunk_period': 's',
     }
 
-    # Act
+    result = model_metrics_activity.calculate_drift(input_data)
-    result = await model_metrics_activity.calculate_drift(input_data)
 
-    # Assert
+    expected_timestamp = ts.tz_localize('UTC').strftime(DATETIME_FORMAT_WITH_TZ)
-    assert isinstance(result, list)
+    assert result == [
-    assert result == mock_drift_df.to_dict.return_value  # type: ignore[comparison-overlap]
+        {
+            'timestamp': expected_timestamp,
+            'feature': 'feature1',
+            'method': 'ks_test',
+            'value': 0.5,
+            'alert': False,
+            'chunk_index': 0,
+            'chunk_start_date': ts.isoformat(),
+            'chunk_end_date': ts.isoformat(),
+            'model_id': 'test_model_id',
+            'accurate': True,
+        }
+    ]
     model_metrics_activity.info.assert_called()
     model_metrics_activity.get_drift_metrics.assert_called_once()
-    # Verify transformations were called
-    mock_drift_df.drop.assert_called_once_with(columns=['p_value'], inplace=True)
-    mock_drift_df.__getitem__.assert_called()
-    mock_drift_df.rename.assert_called_once_with(
-        columns={'metric': 'method', 'statistic': 'value'}, inplace=True
-    )
-    mock_drift_df.drop_duplicates.assert_called_once_with(
-        subset=['timestamp', 'method', 'feature'], keep='first', inplace=True
-    )
-    mock_drift_df.to_dict.assert_called_once_with(orient='records')
 
 
-@mark.asyncio
 @patch('laborious.activities.model_metrics.DataFrame')
 @patch('laborious.activities.model_metrics.to_datetime')
-async def test_calculate_drift_get_drift_metrics_error(
+def test_calculate_drift_get_drift_metrics_error(
     mock_to_datetime, mock_dataframe, model_metrics_activity
 ):
     # Arrange
     mock_to_datetime.return_value.dt.strftime.return_value = '2023-05-26 11:12:27'
 
-    model_metrics_activity.get_drift_metrics = AsyncMock(
+    model_metrics_activity.get_drift_metrics = MagicMock(
         side_effect=Exception('Get drift metrics error')
     )
 
@@ -605,15 +464,14 @@ async def test_calculate_drift_get_drift_metrics_error(
         'chunk_period': 'min',
     }
 
-    # Act
+    # Act / Assert
-    result = await model_metrics_activity.calculate_drift(input_data)
+    with raises(Exception, match='Get drift metrics error'):
+        model_metrics_activity.calculate_drift(input_data)
 
-    # Assert
-    assert result == []
     model_metrics_activity.error.assert_called_once_with(
         'Error getting drift metrics: Get drift metrics error', metadata['metadata']
     )
-    model_metrics_activity.send_notification_async.assert_called_once_with(
+    model_metrics_activity.send_notification.assert_called_once_with(
         metadata=metadata['metadata'],
         notification_id='MODEL_METRICS_GET_DRIFT_METRICS_ERROR',
         message='Error getting drift metrics: Get drift metrics error',
@@ -623,12 +481,11 @@ async def test_calculate_drift_get_drift_metrics_error(
     )
 
 
-@mark.asyncio
 @patch('laborious.activities.model_metrics.to_datetime')
 @patch('laborious.activities.model_metrics.time.time')
-@patch('laborious.activities.model_metrics.ModelAnalysis')
+@patch('laborious.activities.model_metrics.DriftAnalysis')
 @patch('laborious.activities.model_metrics.metrics')
-async def test_get_drift_metrics_success(
+def test_get_drift_metrics_success(
     mock_metrics, mock_model_analysis, mock_time, mock_to_datetime, model_metrics_activity
 ):
     # Arrange
@@ -637,8 +494,8 @@ async def test_get_drift_metrics_success(
     mock_drift_df = DataFrame(
         {
             'timestamp': ['2023-05-26 11:12:27'],
-            'metric': ['ks_test'],
+            'method': ['ks_test'],
-            'statistic': [0.5],
+            'value': [0.5],
             'feature': ['feature1'],
         }
     )
@@ -668,7 +525,7 @@ async def test_get_drift_metrics_success(
     ).columns
 
     # Act
-    result = await model_metrics_activity.get_drift_metrics(
+    result = model_metrics_activity.get_drift_metrics(
         reference_data=reference_data,
         target_data=target_data,
         target_name='target',
@@ -681,16 +538,15 @@ async def test_get_drift_metrics_success(
     # Assert
     assert isinstance(result, DataFrame)
     model_metrics_activity.debug.assert_called()
-    model_metrics_activity.observe_lag.assert_called()
+    model_metrics_activity.observe_lag_sync.assert_called()
-    model_metrics_activity.emit_metric.assert_called()
+    model_metrics_activity.emit_metric_sync.assert_called()
 
 
-@mark.asyncio
 @patch('laborious.activities.model_metrics.to_datetime')
 @patch('laborious.activities.model_metrics.time.time')
-@patch('laborious.activities.model_metrics.ModelAnalysis')
+@patch('laborious.activities.model_metrics.DriftAnalysis')
 @patch('laborious.activities.model_metrics.metrics')
-async def test_get_drift_metrics_univariate_error(
+def test_get_drift_metrics_univariate_error(
     mock_metrics, mock_model_analysis, mock_time, mock_to_datetime, model_metrics_activity
 ):
     # Arrange
@@ -722,7 +578,7 @@ async def test_get_drift_metrics_univariate_error(
 
     # Act & Assert
     try:
-        await model_metrics_activity.get_drift_metrics(
+        model_metrics_activity.get_drift_metrics(
             reference_data=reference_data,
             target_data=target_data,
             target_name='target',
@@ -736,19 +592,18 @@ async def test_get_drift_metrics_univariate_error(
         model_metrics_activity.error.assert_called_once_with(
             'Error detecting univariate drift: Univariate drift error', metadata['metadata']
         )
-        model_metrics_activity.emit_metric.assert_called_with(
+        model_metrics_activity.emit_metric_sync.assert_called_with(
             metric_object=mock_metrics.MODEL_ANALYZE_ERROR_COUNT, tags=ANY
         )
     else:
         raise AssertionError('Expected Exception')
 
 
-@mark.asyncio
 @patch('laborious.activities.model_metrics.to_datetime')
 @patch('laborious.activities.model_metrics.time.time')
-@patch('laborious.activities.model_metrics.ModelAnalysis')
+@patch('laborious.activities.model_metrics.DriftAnalysis')
 @patch('laborious.activities.model_metrics.metrics')
-async def test_get_drift_metrics_multivariate_error(
+def test_get_drift_metrics_multivariate_error(
     mock_metrics, mock_model_analysis, mock_time, mock_to_datetime, model_metrics_activity
 ):
     mock_time.return_value = 1000.0
@@ -769,7 +624,7 @@ async def test_get_drift_metrics_multivariate_error(
     ).columns
 
     try:
-        await model_metrics_activity.get_drift_metrics(
+        model_metrics_activity.get_drift_metrics(
             reference_data=reference_data,
             target_data=target_data,
             target_name='target',
@@ -783,19 +638,18 @@ async def test_get_drift_metrics_multivariate_error(
         model_metrics_activity.error.assert_called_once_with(
             'Error detecting multivariate drift: Multivariate drift error', metadata['metadata']
         )
-        model_metrics_activity.emit_metric.assert_called_with(
+        model_metrics_activity.emit_metric_sync.assert_called_with(
             metric_object=mock_metrics.MODEL_ANALYZE_ERROR_COUNT, tags=ANY
         )
     else:
         raise AssertionError('Expected Exception')
 
 
-@mark.asyncio
 @patch('laborious.activities.model_metrics.to_datetime')
 @patch('laborious.activities.model_metrics.time.time')
-@patch('laborious.activities.model_metrics.ModelAnalysis')
+@patch('laborious.activities.model_metrics.DriftAnalysis')
 @patch('laborious.activities.model_metrics.metrics')
-async def test_get_drift_metrics_dataframe_error(
+def test_get_drift_metrics_dataframe_error(
     mock_metrics, mock_model_analysis, mock_time, mock_to_datetime, model_metrics_activity
 ):
     mock_time.return_value = 1000.0
@@ -817,7 +671,7 @@ async def test_get_drift_metrics_dataframe_error(
     ).columns
 
     try:
-        await model_metrics_activity.get_drift_metrics(
+        model_metrics_activity.get_drift_metrics(
             reference_data=reference_data,
             target_data=target_data,
             target_name='target',
@@ -829,17 +683,16 @@ async def test_get_drift_metrics_dataframe_error(
     except Exception as e:
         assert str(e) == 'Dataframe error'
         model_metrics_activity.error.assert_called_once_with(
-            'Error getting drift metrics: Dataframe error', metadata['metadata']
+            'Error building drift metrics dataframe: Dataframe error', metadata['metadata']
         )
-        model_metrics_activity.emit_metric.assert_called_with(
+        model_metrics_activity.emit_metric_sync.assert_called_with(
             metric_object=mock_metrics.MODEL_ANALYZE_ERROR_COUNT, tags=ANY
         )
     else:
         raise AssertionError('Expected Exception')
 
 
-@mark.asyncio
+def test_calculate_simple_metrics_success_all_metrics(model_metrics_activity):
-async def test_calculate_simple_metrics_success_all_metrics(model_metrics_activity):
     # Arrange
     target_data = DataFrame(
         {
@@ -858,7 +711,7 @@ async def test_calculate_simple_metrics_success_all_metrics(model_metrics_activi
     }
 
     # Act
-    result = DataFrame(await model_metrics_activity.calculate_simple_metrics(input_data))
+    result = DataFrame(model_metrics_activity.calculate_simple_metrics(input_data))
 
     # Assert
     assert len(result['metric']) == 4
@@ -877,8 +730,7 @@ async def test_calculate_simple_metrics_success_all_metrics(model_metrics_activi
     model_metrics_activity.debug.assert_called_once()
 
 
-@mark.asyncio
+def test_calculate_simple_metrics_success_rmse_only(model_metrics_activity):
-async def test_calculate_simple_metrics_success_rmse_only(model_metrics_activity):
     # Arrange
     target_data = DataFrame(
         {
@@ -897,7 +749,7 @@ async def test_calculate_simple_metrics_success_rmse_only(model_metrics_activity
     }
 
     # Act
-    result = DataFrame(await model_metrics_activity.calculate_simple_metrics(input_data))
+    result = DataFrame(model_metrics_activity.calculate_simple_metrics(input_data))
 
     # Assert
     assert len(result['metric']) == 1
@@ -911,8 +763,7 @@ async def test_calculate_simple_metrics_success_rmse_only(model_metrics_activity
     )
 
 
-@mark.asyncio
+def test_calculate_simple_metrics_success_mse_only(model_metrics_activity):
-async def test_calculate_simple_metrics_success_mse_only(model_metrics_activity):
     # Arrange
     target_data = DataFrame(
         {
@@ -931,7 +782,7 @@ async def test_calculate_simple_metrics_success_mse_only(model_metrics_activity)
     }
 
     # Act
-    result = DataFrame(await model_metrics_activity.calculate_simple_metrics(input_data))
+    result = DataFrame(model_metrics_activity.calculate_simple_metrics(input_data))
 
     # Assert
     assert len(result['metric']) == 1
@@ -945,8 +796,7 @@ async def test_calculate_simple_metrics_success_mse_only(model_metrics_activity)
     )
 
 
-@mark.asyncio
+def test_calculate_simple_metrics_success_mae_only(model_metrics_activity):
-async def test_calculate_simple_metrics_success_mae_only(model_metrics_activity):
     # Arrange
     target_data = DataFrame(
         {
@@ -965,7 +815,7 @@ async def test_calculate_simple_metrics_success_mae_only(model_metrics_activity)
     }
 
     # Act
-    result = DataFrame(await model_metrics_activity.calculate_simple_metrics(input_data))
+    result = DataFrame(model_metrics_activity.calculate_simple_metrics(input_data))
 
     # Assert
     assert len(result['metric']) == 1
@@ -979,8 +829,7 @@ async def test_calculate_simple_metrics_success_mae_only(model_metrics_activity)
     )
 
 
-@mark.asyncio
+def test_calculate_simple_metrics_success_r2_only(model_metrics_activity):
-async def test_calculate_simple_metrics_success_r2_only(model_metrics_activity):
     # Arrange
     target_data = DataFrame(
         {
@@ -999,7 +848,7 @@ async def test_calculate_simple_metrics_success_r2_only(model_metrics_activity):
     }
 
     # Act
-    result = DataFrame(await model_metrics_activity.calculate_simple_metrics(input_data))
+    result = DataFrame(model_metrics_activity.calculate_simple_metrics(input_data))
 
     # Assert
     assert len(result['metric']) == 1
@@ -1013,8 +862,7 @@ async def test_calculate_simple_metrics_success_r2_only(model_metrics_activity):
     )
 
 
-@mark.asyncio
+def test_calculate_simple_metrics_r2_zero_ss_tot(model_metrics_activity):
-async def test_calculate_simple_metrics_r2_zero_ss_tot(model_metrics_activity):
     # Arrange
     # All target values are the same, so ss_tot will be 0
     target_data = DataFrame(
@@ -1034,7 +882,7 @@ async def test_calculate_simple_metrics_r2_zero_ss_tot(model_metrics_activity):
     }
 
     # Act
-    result = DataFrame(await model_metrics_activity.calculate_simple_metrics(input_data))
+    result = DataFrame(model_metrics_activity.calculate_simple_metrics(input_data))
 
     # Assert
     assert len(result['metric']) == 1
@@ -1049,8 +897,7 @@ async def test_calculate_simple_metrics_r2_zero_ss_tot(model_metrics_activity):
     )
 
 
-@mark.asyncio
+def test_calculate_simple_metrics_success_multiple_metrics_subset(model_metrics_activity):
-async def test_calculate_simple_metrics_success_multiple_metrics_subset(model_metrics_activity):
     # Arrange
     target_data = DataFrame(
         {
@@ -1069,7 +916,7 @@ async def test_calculate_simple_metrics_success_multiple_metrics_subset(model_me
     }
 
     # Act
-    result = DataFrame(await model_metrics_activity.calculate_simple_metrics(input_data))
+    result = DataFrame(model_metrics_activity.calculate_simple_metrics(input_data))
 
     # Assert
     assert len(result['metric']) == 2
@@ -1084,8 +931,7 @@ async def test_calculate_simple_metrics_success_multiple_metrics_subset(model_me
     )
 
 
-@mark.asyncio
+def test_calculate_simple_metrics_unknown_metric_ignored(model_metrics_activity):
-async def test_calculate_simple_metrics_unknown_metric_ignored(model_metrics_activity):
     target_data = DataFrame(
         {
             'timestamp': ['2023-05-26 11:12:27', '2023-05-26 11:12:28'],
@@ -1102,7 +948,7 @@ async def test_calculate_simple_metrics_unknown_metric_ignored(model_metrics_act
         'interval_minutes': 5,
     }
 
-    result = DataFrame(await model_metrics_activity.calculate_simple_metrics(input_data))
+    result = DataFrame(model_metrics_activity.calculate_simple_metrics(input_data))
 
     assert len(result['metric']) == 1
     assert result['metric'].values[0] == 'rmse'

tests/laborious/activities/test_opc.py

@@ -1,11 +1,20 @@
-from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
+from unittest.mock import ANY, MagicMock, call, patch
 
-import pytest_asyncio
+import pytest
 from pandas import DataFrame
 from pytest import mark
 from sientia_do.notifications.models import NotificationLevel
 
-from laborious.activities.opc import OPC
+from laborious.activities.opc import (
+    OPC,
+    OPC_COMMENT_SEPARATOR,
+    OPC_RECONNECT_IN_PROGRESS_COMMENT,
+    OPC_SESSION_BAD_COMMENT_PREFIX,
+    OPC_SESSION_BAD_CONFIDENCE,
+    OPC_WRITTING_ERROR_CONFIDENCE,
+    OPC_WRITTING_ERROR_MESSAGE,
+    _apply_opc_write_error,
+)
 
 metadata = {
     'metadata': {
@@ -23,27 +32,26 @@ def test__init__():
         opc_servers=servers,
         logger=MagicMock(),
         notification_handler=MagicMock(),
-        metrics_controller=AsyncMock(),
+        metrics_controller=MagicMock(),
     )
 
     assert opc.opc_servers == servers
     assert opc.opc_repository == {}
 
 
-@mark.asyncio
 @patch('laborious.activities.opc.OpcRepository')
-@patch('laborious.activities.opc.OPC.send_notification_async')
+@patch('laborious.activities.opc.OPC.send_notification')
-async def test_init_opc(mock_send_notification, mock_opc_repository):
+def test_init_opc(mock_send_notification, mock_opc_repository):
     mock_logger = MagicMock()
-    mock_metrics_controller = AsyncMock()
+    mock_metrics_controller = MagicMock()
     server1 = MagicMock(
-        connect=AsyncMock(return_value=(True, {})), write_data=AsyncMock(return_value=(True, {}))
+        connect=MagicMock(return_value=(True, {})), write_data=MagicMock(return_value=(True, {}))
     )
     server2 = MagicMock(
-        connect=AsyncMock(return_value=(True, {})), write_data=AsyncMock(return_value=(True, {}))
+        connect=MagicMock(return_value=(True, {})), write_data=MagicMock(return_value=(True, {}))
     )
     server3 = MagicMock(
-        connect=AsyncMock(
+        connect=MagicMock(
             return_value=(
                 False,
                 {
@@ -55,7 +63,7 @@ async def test_init_opc(mock_send_notification, mock_opc_repository):
                 },
             )
         ),
-        write_data=AsyncMock(return_value=(True, {})),
+        write_data=MagicMock(return_value=(True, {})),
     )
     mock_opc_repository.side_effect = [server1, server2, server3]
     mock_notification_handler = MagicMock()
@@ -97,7 +105,7 @@ async def test_init_opc(mock_send_notification, mock_opc_repository):
         notification_handler=mock_notification_handler,
         metrics_controller=mock_metrics_controller,
     )
-    await opc.init_opc()
+    opc.init_opc()
 
     assert opc.opc_servers == servers
     assert opc.logger == mock_logger
@@ -162,9 +170,9 @@ async def test_init_opc(mock_send_notification, mock_opc_repository):
     )
 
 
-@pytest_asyncio.fixture
+@pytest.fixture
 @patch('laborious.activities.opc.OpcRepository')
-async def opc(mock_opc_repository):
+def opc(mock_opc_repository):
     servers = {
         'server1': {
             'id': 'server1',
@@ -178,18 +186,17 @@ async def opc(mock_opc_repository):
         }
     }
 
-    mock_opc_repository.return_value.write_data = AsyncMock(return_value=(True, {}))
+    mock_opc_repository.return_value.write_data = MagicMock(return_value=(True, {}))
-    mock_opc_repository.return_value.connect = AsyncMock(return_value=(True, {}))
+    mock_opc_repository.return_value.connect = MagicMock(return_value=(True, {}))
     opc = OPC(
         opc_servers=servers,
         logger=MagicMock(),
         notification_handler=MagicMock(),
-        metrics_controller=AsyncMock(),
+        metrics_controller=MagicMock(),
     )
-    await opc.init_opc()
+    opc.init_opc()
     opc.send_notification = MagicMock()
-    opc.send_notification_async = AsyncMock()
+    opc.emit_metric_sync = MagicMock()
-    opc.emit_metric = AsyncMock()
     return opc
 
 
@@ -202,11 +209,10 @@ WRITE_DATA_CASES = [
 
 
 @mark.parametrize('tag,data_type,data', WRITE_DATA_CASES)
-@mark.asyncio
+def test_write_data_success(opc, tag, data_type, data):
-async def test_write_data_success(opc, tag, data_type, data):
     opc.opc_repository['server1'].write_data.return_value = (True, {'response_time': 0.1})
 
-    result = await opc.write_data(
+    response_time, error_info = opc.write_data(
         server_id='server1',
         tag=tag,
         data=data,
@@ -214,14 +220,12 @@ async def test_write_data_success(opc, tag, data_type, data):
         tag_type='prediction',
         metadata=metadata,
     )
-    assert result == 0.1
+    assert response_time == 0.1
-    opc.opc_repository['server1'].write_data.assert_called_once_with(
+    assert error_info is None
-        tag, data, data_type, opc.logger, metadata
+    opc.opc_repository['server1'].write_data.assert_called_once_with(tag, data, data_type, metadata)
-    )
 
 
-@mark.asyncio
+def test_write_data_failed(opc):
-async def test_write_data_failed(opc):
     opc.opc_repository['server1'].write_data.return_value = (
         False,
         {
@@ -233,7 +237,7 @@ async def test_write_data_failed(opc):
         },
     )
 
-    result = await opc.write_data(
+    response_time, error_info = opc.write_data(
         server_id='server1',
         tag='tag1',
         data=50,
@@ -241,9 +245,10 @@ async def test_write_data_failed(opc):
         tag_type='prediction',
         metadata=metadata,
     )
-    assert result is None
+    assert response_time is None
+    assert error_info is not None
 
-    opc.send_notification_async.assert_called_once_with(
+    opc.send_notification.assert_called_once_with(
         metadata=metadata,
         notification_id='OPC_WRITE_DATA_ERROR_server1',
         message='Failed to write data to OPC server: Test error',
@@ -253,12 +258,11 @@ async def test_write_data_failed(opc):
     )
 
 
-@mark.asyncio
+def test_write_data_exception(opc):
-async def test_write_data_exception(opc):
     opc.opc_repository['server1'].write_data.side_effect = Exception('Test error')
 
     try:
-        await opc.write_data(
+        opc.write_data(
             server_id='server1',
             tag='tag1',
             data=50,
@@ -268,7 +272,7 @@ async def test_write_data_exception(opc):
         )
 
     except Exception:
-        opc.send_notification_async.assert_called_once_with(
+        opc.send_notification.assert_called_once_with(
             metadata=metadata,
             notification_id='WRITE_OPC_PREDICTION_ERROR',
             message='Error writing data to OPC server: Test error',
@@ -281,17 +285,205 @@ async def test_write_data_exception(opc):
         raise AssertionError('Expected an exception to be raised')
 
 
-@mark.asyncio
+@mark.parametrize(
-async def test_manage_output_tags_success(opc):
+    'error_info,initial_seen,initial_status,initial_reconnect,expected',
-    opc.write_data = AsyncMock(return_value=0.1)
+    [
+        (None, False, None, False, (False, None, False)),
+        ({}, False, None, False, (False, None, False)),
+        (
+            {'opc_error_kind': 'session_bad', 'opc_status': 'BadSessionIdInvalid'},
+            False,
+            None,
+            False,
+            (True, 'BadSessionIdInvalid', False),
+        ),
+        (
+            {'opc_error_kind': 'session_bad', 'opc_status': 'NewStatus'},
+            True,
+            'OldStatus',
+            False,
+            (True, 'NewStatus', False),
+        ),
+        (
+            {'opc_error_kind': 'session_bad'},
+            True,
+            'KeptStatus',
+            False,
+            (True, 'KeptStatus', False),
+        ),
+        (
+            {'opc_error_kind': 'reconnect_in_progress'},
+            False,
+            None,
+            False,
+            (False, None, True),
+        ),
+        (
+            {'opc_error_kind': 'other'},
+            True,
+            'Status',
+            True,
+            (True, 'Status', True),
+        ),
+    ],
+)
+def test_apply_opc_write_error(
+    error_info, initial_seen, initial_status, initial_reconnect, expected
+):
+    result = _apply_opc_write_error(
+        error_info,
+        initial_seen,
+        initial_status,
+        initial_reconnect,
+    )
+    assert result == expected
 
+
+def test_write_tags_from_config_prediction_success(opc):
+    opc.write_data = MagicMock(return_value=(0.1, None))
+    data = DataFrame({'prediction': [0.75], 'prediction_confidence': [0.95]})
+    tags_config = {'tag1': {'data_type': 'float'}}
+
+    response_times, session_bad, opc_status, reconnect = opc._write_tags_from_config(
+        server_id='server1',
+        tags_config=tags_config,
+        data=data,
+        data_column='prediction',
+        tag_type='prediction',
+        log_label='Prediction data',
+        metadata=metadata['metadata'],
+    )
+
+    assert response_times == {'tag1': 0.1}
+    assert session_bad is False
+    assert opc_status is None
+    assert reconnect is False
+    opc.write_data.assert_called_once_with(
+        server_id='server1',
+        tag='tag1',
+        data=0.75,
+        data_type='float',
+        tag_type='prediction',
+        metadata=metadata['metadata'],
+    )
+
+
+def test_write_tags_from_config_confidence_success(opc):
+    opc.write_data = MagicMock(return_value=(0.2, None))
+    data = DataFrame({'prediction': [0.75], 'prediction_confidence': [0.95]})
+    tags_config = {'tag2': {'data_type': 'float'}}
+
+    response_times, session_bad, opc_status, reconnect = opc._write_tags_from_config(
+        server_id='server1',
+        tags_config=tags_config,
+        data=data,
+        data_column='prediction_confidence',
+        tag_type='confidence',
+        log_label='Confidence data',
+        metadata=metadata['metadata'],
+    )
+
+    assert response_times == {'tag2': 0.2}
+    assert session_bad is False
+    assert opc_status is None
+    assert reconnect is False
+    opc.write_data.assert_called_once_with(
+        server_id='server1',
+        tag='tag2',
+        data=0.95,
+        data_type='float',
+        tag_type='confidence',
+        metadata=metadata['metadata'],
+    )
+
+
+def test_write_tags_from_config_write_failure(opc):
+    opc.write_data = MagicMock(return_value=(None, {}))
+    data = DataFrame({'prediction': [0.75], 'prediction_confidence': [0.95]})
+
+    response_times, session_bad, opc_status, reconnect = opc._write_tags_from_config(
+        server_id='server1',
+        tags_config={'tag1': {'data_type': 'float'}},
+        data=data,
+        data_column='prediction',
+        tag_type='prediction',
+        log_label='Prediction data',
+        metadata=metadata['metadata'],
+    )
+
+    assert response_times == {'tag1': None}
+    assert session_bad is False
+    assert opc_status is None
+    assert reconnect is False
+
+
+def test_write_tags_from_config_session_bad(opc):
+    opc.write_data = MagicMock(
+        return_value=(
+            None,
+            {
+                'opc_error_kind': 'session_bad',
+                'opc_status': 'BadSessionIdInvalid',
+            },
+        )
+    )
+    data = DataFrame({'prediction': [0.75], 'prediction_confidence': [0.95]})
+
+    response_times, session_bad, opc_status, reconnect = opc._write_tags_from_config(
+        server_id='server1',
+        tags_config={'tag1': {'data_type': 'float'}},
+        data=data,
+        data_column='prediction',
+        tag_type='prediction',
+        log_label='Prediction data',
+        metadata=metadata['metadata'],
+    )
+
+    assert response_times == {'tag1': None}
+    assert session_bad is True
+    assert opc_status == 'BadSessionIdInvalid'
+    assert reconnect is False
+
+
+def test_write_tags_from_config_reconnect_in_progress(opc):
+    opc.write_data = MagicMock(
+        return_value=(
+            None,
+            {'opc_error_kind': 'reconnect_in_progress'},
+        )
+    )
+    data = DataFrame({'prediction': [0.75], 'prediction_confidence': [0.95]})
+
+    response_times, session_bad, opc_status, reconnect = opc._write_tags_from_config(
+        server_id='server1',
+        tags_config={'tag1': {'data_type': 'float'}},
+        data=data,
+        data_column='prediction',
+        tag_type='prediction',
+        log_label='Prediction data',
+        metadata=metadata['metadata'],
+    )
+
+    assert response_times == {'tag1': None}
+    assert session_bad is False
+    assert opc_status is None
+    assert reconnect is True
+
+
+def test_manage_output_tags_success(opc):
+    opc._write_tags_from_config = MagicMock(
+        side_effect=[
+            ({'tag1': 0.1}, False, None, False),
+            ({'tag2': 0.1}, False, None, False),
+        ]
+    )
     data = DataFrame({'prediction': [0.75], 'prediction_confidence': [0.95]})
     config = {
         'prediction_tags': {'tag1': {'data_type': 'float'}},
         'confidence_tags': {'tag2': {'data_type': 'float'}},
     }
 
-    output_data, opc_metrics = await opc.manage_output_tags(
+    output_data, opc_metrics, session_bad, opc_status, reconnect = opc.manage_output_tags(
         server_id='server1',
         config=config,
         data=data,
@@ -300,88 +492,55 @@ async def test_manage_output_tags_success(opc):
 
     assert output_data is True
     assert opc_metrics == {'tag1': 0.1, 'tag2': 0.1}
-    opc.write_data.assert_has_calls(
+    assert session_bad is False
-        [
+    assert opc_status is None
-            call(
+    assert reconnect is False
-                server_id='server1',
+    assert opc._write_tags_from_config.call_count == 2
-                tag='tag1',
+
-                data=0.75,
+
-                data_type='float',
+def test_manage_output_tags_failed(opc):
-                tag_type='prediction',
+    opc._write_tags_from_config = MagicMock(
-                metadata=metadata['metadata'],
+        side_effect=[
-            ),
+            ({'tag1': 0.1}, False, None, False),
-            call(
+            ({'tag2': None}, False, None, False),
-                server_id='server1',
-                tag='tag2',
-                data=0.95,
-                data_type='float',
-                tag_type='confidence',
-                metadata=metadata['metadata'],
-            ),
         ]
     )
-
-
-@mark.asyncio
-@mark.parametrize('side_effect', [[0.1, None], [None, 0.2]])
-async def test_manage_output_tags_failed(opc, side_effect):
-    opc.write_data = AsyncMock(side_effect=side_effect)
     data = DataFrame({'prediction': [0.75], 'prediction_confidence': [0.95]})
     config = {
         'prediction_tags': {'tag1': {'data_type': 'float'}},
         'confidence_tags': {'tag2': {'data_type': 'float'}},
     }
-    output_data, opc_metrics = await opc.manage_output_tags(
+
+    output_data, opc_metrics, _, _, _ = opc.manage_output_tags(
         server_id='server1',
         config=config,
         data=data,
         metadata=metadata['metadata'],
     )
+
     assert output_data is False
-    assert opc_metrics == {'tag1': side_effect[0], 'tag2': side_effect[1]}
+    assert opc_metrics == {'tag1': 0.1, 'tag2': None}
-    opc.write_data.assert_has_calls(
-        [
-            call(
-                server_id='server1',
-                tag='tag1',
-                data=0.75,
-                data_type='float',
-                tag_type='prediction',
-                metadata=metadata['metadata'],
-            ),
-            call(
-                server_id='server1',
-                tag='tag2',
-                data=0.95,
-                data_type='float',
-                tag_type='confidence',
-                metadata=metadata['metadata'],
-            ),
-        ]
-    )
 
 
-@mark.asyncio
+def test_manage_output_tags_do_nothing(opc):
-async def test_manage_output_tags_do_nothing(opc):
+    opc._write_tags_from_config = MagicMock()
-    opc.write_data = AsyncMock(return_value=0.1)
     data = DataFrame({'prediction': [0.75], 'prediction_confidence': [0.95]})
-    config = {
+    config = {'_invalid_key': {'tag1': {'data_type': 'float'}}}
-        '_invalid_key': {'tag1': {'data_type': 'float'}},
+
-    }
+    output_data, opc_metrics, _, _, _ = opc.manage_output_tags(
-    output_data, opc_metrics = await opc.manage_output_tags(
         server_id='server1',
         config=config,
         data=data,
         metadata=metadata['metadata'],
     )
+
     assert output_data is True
     assert opc_metrics == {}
-    opc.write_data.assert_not_called()
+    opc._write_tags_from_config.assert_not_called()
 
 
-@mark.asyncio
 @patch('laborious.activities.opc.DataFrame')
-async def test_write_opc_data_success(mock_dataframe, opc):
+def test_write_opc_data_success(mock_dataframe, opc):
     # Arrange
     input_data = {
         **metadata,
@@ -395,10 +554,12 @@ async def test_write_opc_data_success(mock_dataframe, opc):
     }
 
     # Act
-    opc.manage_output_tags = AsyncMock(return_value=(True, {'tag1': 0.1, 'tag2': 0.2}))
+    opc.manage_output_tags = MagicMock(
+        return_value=(True, {'tag1': 0.1, 'tag2': 0.2}, False, None, False)
+    )
 
     opc.process_confidence = MagicMock(return_value={'data': 'data'})
-    output_data, opc_metrics = await opc.write_opc_data(input_data)
+    output_data, opc_metrics = opc.write_opc_data(input_data)
 
     # Assert
     assert output_data == {'data': 'data'}
@@ -413,11 +574,13 @@ async def test_write_opc_data_success(mock_dataframe, opc):
         mock_dataframe.return_value,
         True,
         metadata['metadata'],
+        session_bad=False,
+        opc_status=None,
+        reconnect_in_progress=False,
     )
 
 
-@mark.asyncio
+def test_write_opc_data_empty_config(opc):
-async def test_write_opc_data_empty_config(opc):
     # Arrange
     input_data = {
         **metadata,
@@ -427,15 +590,14 @@ async def test_write_opc_data_empty_config(opc):
     }
 
     # Act
-    await opc.write_opc_data(input_data)
+    opc.write_opc_data(input_data)
 
     # Assert
     opc.opc_repository['server1'].write_data.assert_not_called()
 
 
-@mark.asyncio
+def test_write_opc_data_no_validate_server(opc):
-async def test_write_opc_data_no_validate_server(opc):
+    opc.validate_server = MagicMock(return_value=False)
-    opc.validate_server = AsyncMock(return_value=False)
     input_data = {
         **metadata,
         'data': {'prediction': [0.75], 'prediction_confidence': [0.95]},
@@ -448,7 +610,7 @@ async def test_write_opc_data_no_validate_server(opc):
     }
 
     # Act
-    await opc.write_opc_data(input_data)
+    opc.write_opc_data(input_data)
 
     # Assert
     opc.opc_repository['server1'].write_data.assert_not_called()
@@ -462,21 +624,97 @@ async def test_write_opc_data_no_validate_server(opc):
     ],
 )
 def test_process_confidence(opc, data, success, expected):
-    # Act
+    result = opc.process_confidence(data, success, metadata['metadata'])
-    result = opc.process_confidence(data, success, metadata)
-
-    # Assert
     assert result['prediction_confidence'][0] == expected
 
 
-@mark.asyncio
+def test_process_confidence_session_bad(opc):
-async def test_validate_server(opc):
+    data = DataFrame({'prediction_confidence': [0.9]})
-    assert await opc.validate_server('server1', metadata) is True
+    result = opc.process_confidence(
-    assert await opc.validate_server('server2', metadata) is False
+        data,
+        False,
+        metadata['metadata'],
+        session_bad=True,
+        opc_status='BadSessionIdInvalid',
+    )
+    assert result['prediction_confidence'][0] == OPC_SESSION_BAD_CONFIDENCE
+    assert result['comments'][0].startswith(OPC_SESSION_BAD_COMMENT_PREFIX)
+    assert 'BadSessionIdInvalid' in result['comments'][0]
 
 
-@mark.asyncio
+def test_process_confidence_generic_failure(opc):
-async def test_close(opc):
+    data = DataFrame({'prediction_confidence': [0.9]})
-    opc.opc_repository['server1'].disconnect = AsyncMock(return_value=True)
+    result = opc.process_confidence(data, False, metadata['metadata'])
-    await opc.close()
+    assert result['prediction_confidence'][0] == OPC_WRITTING_ERROR_CONFIDENCE
-    opc.opc_repository['server1'].disconnect.assert_called_once()
+    assert result['comments'][0] == OPC_WRITTING_ERROR_MESSAGE
+
+
+def test_manage_output_tags_merges_error_flags(opc):
+    opc._write_tags_from_config = MagicMock(
+        side_effect=[
+            ({'tag1': None}, True, 'BadSessionIdInvalid', False),
+            ({'tag2': 0.2}, False, None, True),
+        ]
+    )
+    data = DataFrame({'prediction': [0.75], 'prediction_confidence': [0.95]})
+    config = {
+        'prediction_tags': {'tag1': {'data_type': 'float'}},
+        'confidence_tags': {'tag2': {'data_type': 'float'}},
+    }
+
+    (
+        success,
+        metrics,
+        session_bad_seen,
+        opc_status,
+        reconnect_in_progress,
+    ) = opc.manage_output_tags('server1', config, data, metadata['metadata'])
+
+    assert success is False
+    assert session_bad_seen is True
+    assert reconnect_in_progress is True
+    assert opc_status == 'BadSessionIdInvalid'
+    assert metrics == {'tag1': None, 'tag2': 0.2}
+
+
+def test_process_confidence_reconnect_in_progress(opc):
+    data = DataFrame({'prediction_confidence': [0.9]})
+    result = opc.process_confidence(
+        data,
+        False,
+        metadata['metadata'],
+        reconnect_in_progress=True,
+    )
+    assert result['prediction_confidence'][0] == OPC_SESSION_BAD_CONFIDENCE
+    assert result['comments'][0] == OPC_RECONNECT_IN_PROGRESS_COMMENT
+
+
+def test_process_confidence_concatenates_multiple_comments(opc):
+    data = DataFrame({'prediction_confidence': [0.9]})
+    session_comment = f'{OPC_SESSION_BAD_COMMENT_PREFIX} BadSessionIdInvalid'
+
+    result = opc.process_confidence(
+        data,
+        False,
+        metadata['metadata'],
+        session_bad=True,
+        opc_status='BadSessionIdInvalid',
+        reconnect_in_progress=True,
+    )
+
+    assert result['prediction_confidence'][0] == OPC_SESSION_BAD_CONFIDENCE
+    assert result['comments'][0] == OPC_COMMENT_SEPARATOR.join(
+        [session_comment, OPC_RECONNECT_IN_PROGRESS_COMMENT]
+    )
+
+
+def test_validate_server(opc):
+    assert opc.validate_server('server1', metadata) is True
+    assert opc.validate_server('server2', metadata) is False
+
+
+def test_close(opc):
+    repo = opc.opc_repository['server1']
+    repo.disconnect = MagicMock(return_value=True)
+    opc.close()
+    repo.disconnect.assert_called_once()

tests/laborious/activities/test_storage.py

@@ -1,10 +1,11 @@
 import datetime
 import os
-from unittest.mock import ANY, AsyncMock, MagicMock, patch
+from unittest.mock import ANY, MagicMock, patch
 
-from pytest import fixture, mark, raises
+from pytest import fixture, raises
 from sientia_do.notifications.models import NotificationLevel
-from sientia_do.temporal.activities.postgres import Postgres
+from sientia_do.observability.sientia_monitoring import SientiaMonitoring
+from sientia_do.temporal.activities.postgres_sync import Postgres
 
 from laborious.activities.storage import Storage
 
@@ -17,6 +18,15 @@ def _passthrough_from_dict():
         yield
 
 
+@fixture(autouse=True)
+def _patch_monitoring_shutdown():
+    """
+    Avoid running real async SientiaMonitoring.shutdown when Storage.close runs inside tests.
+    """
+    with patch.object(SientiaMonitoring, 'shutdown') as mock_shutdown:
+        yield mock_shutdown
+
+
 metadata = {
     'metadata': {
         'model_id': 'test_model_id',
@@ -42,7 +52,7 @@ def storage(mock_minio_repository):
         minio_repository=mock_minio_repository.return_value,
         logger=MagicMock(),
         notification_handler=MagicMock(),
-        metrics_controller=AsyncMock(),
+        metrics_controller=MagicMock(),
     )
 
 
@@ -50,7 +60,7 @@ def storage(mock_minio_repository):
 def test___init___not_hasattr(mock_minio_repository):
     logger = MagicMock()
     notification_handler = MagicMock()
-    metrics_controller = AsyncMock()
+    metrics_controller = MagicMock()
     minio_repo = mock_minio_repository.return_value
     storage = Storage(
         host='localhost',
@@ -77,7 +87,7 @@ def test___init___none_minio_repository(mock_minio_repository, storage):
     storage.minio_repository = None
     logger = MagicMock()
     notification_handler = MagicMock()
-    metrics_controller = AsyncMock()
+    metrics_controller = MagicMock()
     storage.__init__(
         host='localhost',
         port=5432,
@@ -111,54 +121,55 @@ def test___init___done_repository(mock_minio_repository, storage):
         minio_repository=mock_minio_repository.return_value,
         logger=MagicMock(),
         notification_handler=MagicMock(),
-        metrics_controller=AsyncMock(),
+        metrics_controller=MagicMock(),
     )
     mock_minio_repository.assert_not_called()
     assert storage.minio_repository is not None
 
 
-def test_close(storage):
+def test_close(storage, _patch_monitoring_shutdown):
     storage.minio_repository = MagicMock()
 
     storage.close()
 
     assert storage.minio_repository is None
+    _patch_monitoring_shutdown.assert_called_once_with(storage)
 
 
-def test___del__(storage):
+def test_close_when_minio_repository_already_none(storage, _patch_monitoring_shutdown):
-    storage.close = MagicMock()
+    """Closing without an initialized MinIO repository skips MinIO teardown."""
+    storage.minio_repository = None
 
-    storage.__del__()
+    storage.close()
 
-    storage.close.assert_called_once()
+    assert storage.minio_repository is None
+    _patch_monitoring_shutdown.assert_called_once_with(storage)
 
 
-@mark.asyncio
+def test_load_query_with_minio_offload_no_rows(storage):
-async def test_load_query_with_minio_offload_no_rows(storage):
+    storage.load_custom_query = MagicMock(return_value=None)
-    storage.load_custom_query = AsyncMock(return_value=None)
     storage_result = {'success': False}
     with patch(
         'laborious.activities.storage.MinioDataFramePayload.from_dataframe',
-        new_callable=AsyncMock,
+        new_callable=MagicMock,
         return_value=storage_result,
     ) as mock_from_dataframe:
-        result = await storage.load_query_with_minio_offload(
+        result = storage.load_query_with_minio_offload(
             {**metadata, 'query': 'SELECT 1', 'model_name': 'm', 'key_prefix': 'predictions/s'}
         )
     assert result == storage_result
-    mock_from_dataframe.assert_awaited_once()
+    mock_from_dataframe.assert_called_once()
 
 
-@mark.asyncio
+def test_load_query_with_minio_offload_inline(storage):
-async def test_load_query_with_minio_offload_inline(storage):
+    storage.load_custom_query = MagicMock(return_value=[{'a': 1}])
-    storage.load_custom_query = AsyncMock(return_value=[{'a': 1}])
     storage_result = {'success': True, 'data': {'a': [1]}, 'object_key': None}
     with patch(
         'laborious.activities.storage.MinioDataFramePayload.from_dataframe',
-        new_callable=AsyncMock,
+        new_callable=MagicMock,
         return_value=storage_result,
     ) as mock_from_dataframe:
-        result = await storage.load_query_with_minio_offload(
+        result = storage.load_query_with_minio_offload(
             {
                 **metadata,
                 'query': 'SELECT 1',
@@ -167,44 +178,42 @@ async def test_load_query_with_minio_offload_inline(storage):
             }
         )
     assert result == storage_result
-    mock_from_dataframe.assert_awaited_once()
+    mock_from_dataframe.assert_called_once()
 
 
-@mark.asyncio
+def test_load_query_with_minio_offload_minio(storage):
-async def test_load_query_with_minio_offload_minio(storage):
+    storage.load_custom_query = MagicMock(return_value=[{'a': 1}])
-    storage.load_custom_query = AsyncMock(return_value=[{'a': 1}])
     storage_result = {'success': True, 'data': None, 'object_key': 'object-key'}
     with patch(
         'laborious.activities.storage.MinioDataFramePayload.from_dataframe',
-        new_callable=AsyncMock,
+        new_callable=MagicMock,
         return_value=storage_result,
     ) as mock_from_dataframe:
-        result = await storage.load_query_with_minio_offload(
+        result = storage.load_query_with_minio_offload(
             {**metadata, 'query': 'SELECT 1', 'model_name': 'm', 'key_prefix': 'predictions/s'}
         )
 
     assert result == storage_result
-    mock_from_dataframe.assert_awaited_once()
+    mock_from_dataframe.assert_called_once()
 
 
-@mark.asyncio
 @patch.dict(os.environ, {'SIENTIA_MINIO_RETENTION_HOURS': '1'})
 @patch('laborious.activities.storage.now')
-async def test_cleanup_minio_objects_expired(mock_now, storage):
+def test_cleanup_minio_objects_expired(mock_now, storage):
     mock_now.return_value = datetime.datetime(2025, 1, 10, 12, 0, 0)
-    storage.minio_repository.list_objects = AsyncMock(
+    storage.minio_repository.list_objects = MagicMock(
         return_value=[
             'sientia/streamlit-connectors/training_datasets/m/m-initial-2024-12-01_00-00-00.parquet',
             'sientia/streamlit-connectors/training_datasets/m/m-initial-2025-01-10_12-00-00.parquet',
         ]
     )
-    storage.minio_repository.delete_file = AsyncMock()
+    storage.minio_repository.delete_file = MagicMock()
-    storage.send_notification_async = AsyncMock()
+    storage.send_notification = MagicMock()
 
     data_mock = MagicMock()
     data_mock.cleanup_prefix.return_value = 'training_datasets/m'
 
-    result = await storage.cleanup_minio_objects_expired({**metadata, 'data': data_mock})
+    result = storage.cleanup_minio_objects_expired({**metadata, 'data': data_mock})
 
     assert result['deleted_count'] == 1
     assert result['failed_count'] == 0
@@ -224,72 +233,65 @@ async def test_cleanup_minio_objects_expired(mock_now, storage):
     )
 
 
-@mark.asyncio
+def test_load_query_with_minio_offload_minio_not_initialized(storage):
-async def test_load_query_with_minio_offload_minio_not_initialized(storage):
     storage.minio_repository = None
 
     with raises(ValueError, match='Minio repository not initialized'):
-        await storage.load_query_with_minio_offload(
+        storage.load_query_with_minio_offload({**metadata, 'query': 'SELECT 1', 'model_name': 'm'})
-            {**metadata, 'query': 'SELECT 1', 'model_name': 'm'}
-        )
 
 
-@mark.asyncio
+def test_export_payload_to_postgres(storage):
-async def test_export_payload_to_postgres(storage):
+    payload = MagicMock()
-    payload = AsyncMock()
+    payload.retrieve = MagicMock(return_value=MagicMock())
-    payload.retrieve = AsyncMock(return_value=MagicMock())
+    storage.export_data_to_postgres = MagicMock(return_value={'success': True})
-    storage.export_data_to_postgres = AsyncMock(return_value={'success': True})
 
-    result = await storage.export_payload_to_postgres(
+    result = storage.export_payload_to_postgres(
         {**metadata, 'data': payload, 'schema': 'public', 'table': 't'}
     )
 
-    payload.retrieve.assert_awaited_once_with(storage.minio_repository, metadata['metadata'])
+    payload.retrieve.assert_called_once_with(storage.minio_repository, metadata['metadata'])
-    storage.export_data_to_postgres.assert_awaited_once()
+    storage.export_data_to_postgres.assert_called_once()
     assert result == {'success': True}
 
 
-@mark.asyncio
+def test_cleanup_minio_objects_expired_minio_not_initialized(storage):
-async def test_cleanup_minio_objects_expired_minio_not_initialized(storage):
     storage.minio_repository = None
 
     data_mock = MagicMock()
     data_mock.cleanup_prefix.return_value = 'test'
     with raises(ValueError, match='Minio repository not initialized'):
-        await storage.cleanup_minio_objects_expired({**metadata, 'data': data_mock})
+        storage.cleanup_minio_objects_expired({**metadata, 'data': data_mock})
 
 
-@mark.asyncio
 @patch('laborious.activities.storage.now')
-async def test_cleanup_minio_objects_expired_unparseable_key(mock_now, storage):
+def test_cleanup_minio_objects_expired_unparseable_key(mock_now, storage):
     mock_now.return_value = datetime.datetime(2025, 1, 10, 12, 0, 0)
-    storage.minio_repository.list_objects = AsyncMock(
+    storage.minio_repository.list_objects = MagicMock(
         return_value=['some/random/key-without-timestamp.parquet']
     )
-    storage.minio_repository.delete_file = AsyncMock()
+    storage.minio_repository.delete_file = MagicMock()
-    storage.send_notification_async = AsyncMock()
+    storage.send_notification = MagicMock()
 
     data_mock = MagicMock()
     data_mock.cleanup_prefix.return_value = 'test'
-    result = await storage.cleanup_minio_objects_expired({**metadata, 'data': data_mock})
+    result = storage.cleanup_minio_objects_expired({**metadata, 'data': data_mock})
 
     assert result['deleted_count'] == 0
     assert result['failed_count'] == 0
     storage.minio_repository.delete_file.assert_not_called()
 
 
-@mark.asyncio
 @patch('laborious.activities.storage.now')
-async def test_cleanup_minio_objects_expired_delete_fails(mock_now, storage):
+def test_cleanup_minio_objects_expired_delete_fails(mock_now, storage):
     mock_now.return_value = datetime.datetime(2025, 1, 10, 12, 0, 0)
     old_key = 'training_datasets/m/m-initial-2024-12-01_00-00-00.parquet'
-    storage.minio_repository.list_objects = AsyncMock(return_value=[old_key])
+    storage.minio_repository.list_objects = MagicMock(return_value=[old_key])
-    storage.minio_repository.delete_file = AsyncMock(side_effect=Exception('delete error'))
+    storage.minio_repository.delete_file = MagicMock(side_effect=Exception('delete error'))
-    storage.send_notification_async = AsyncMock()
+    storage.send_notification = MagicMock()
 
     data_mock = MagicMock()
     data_mock.cleanup_prefix.return_value = 'training_datasets/m'
-    result = await storage.cleanup_minio_objects_expired({**metadata, 'data': data_mock})
+    result = storage.cleanup_minio_objects_expired({**metadata, 'data': data_mock})
 
     assert result['deleted_count'] == 0
     assert result['failed_count'] == 1
@@ -298,21 +300,20 @@ async def test_cleanup_minio_objects_expired_delete_fails(mock_now, storage):
     assert result['failed'][old_key]['message'] == 'delete error'
 
 
-@mark.asyncio
 @patch('laborious.activities.storage.now')
-async def test_cleanup_minio_objects_expired_list_objects_error(mock_now, storage):
+def test_cleanup_minio_objects_expired_list_objects_error(mock_now, storage):
     mock_now.return_value = datetime.datetime(2025, 1, 10, 12, 0, 0)
-    storage.minio_repository.list_objects = AsyncMock(side_effect=Exception('list error'))
+    storage.minio_repository.list_objects = MagicMock(side_effect=Exception('list error'))
-    storage.send_notification_async = AsyncMock()
+    storage.send_notification = MagicMock()
     storage.error = MagicMock()
 
     data_mock = MagicMock()
     data_mock.cleanup_prefix.return_value = 'training_datasets/m'
-    result = await storage.cleanup_minio_objects_expired({**metadata, 'data': data_mock})
+    result = storage.cleanup_minio_objects_expired({**metadata, 'data': data_mock})
 
     assert result['deleted_count'] == 0
     assert result['failed_count'] == 0
-    storage.send_notification_async.assert_called_once_with(
+    storage.send_notification.assert_called_once_with(
         metadata=metadata['metadata'],
         notification_id='ERROR_CLEANUP_MINIO_OBJECTS_EXPIRED',
         message='Error cleaning up MinIO objects: list error',

tests/laborious/utils/models/test_minio_dataframe_payload.py

@@ -1,8 +1,7 @@
 from datetime import datetime
 from io import BytesIO
-from unittest.mock import AsyncMock, MagicMock, patch
+from unittest.mock import MagicMock, patch
 
-import pytest
 from pandas import DataFrame
 
 from laborious.utils.models.minio_dataframe_payload import (
@@ -54,17 +53,15 @@ def test_has_data_true_when_object_key_set():
     assert payload.has_data() is True
 
 
-@pytest.mark.asyncio
+def test_retrieve_inline_dict_as_dataframe():
-async def test_retrieve_inline_dict_as_dataframe():
     payload = MinioDataFramePayload(last_timestamp='t', data={'a': [1, 2]})
-    minio = AsyncMock()
+    minio = MagicMock()
-    out = await payload.retrieve(minio, {'metadata': {}})
+    out = payload.retrieve(minio, {'metadata': {}})
     assert list(out.columns) == ['a']
     minio.download_file.assert_not_called()
 
 
-@pytest.mark.asyncio
+def test_retrieve_downloads_parquet_when_offloaded():
-async def test_retrieve_downloads_parquet_when_offloaded():
     source = DataFrame({'a': [1, 2]})
     buf = BytesIO()
     source.to_parquet(buf, engine='pyarrow', index=True)
@@ -76,12 +73,12 @@ async def test_retrieve_downloads_parquet_when_offloaded():
         object_key='training_datasets/m/f.parquet',
         object_prefix='training_datasets/m',
     )
-    minio = AsyncMock()
+    minio = MagicMock()
-    minio.download_file = AsyncMock(return_value=file_bytes)
+    minio.download_file = MagicMock(return_value=file_bytes)
 
-    out = await payload.retrieve(minio, {'metadata': {}})
+    out = payload.retrieve(minio, {'metadata': {}})
 
-    minio.download_file.assert_awaited_once_with(
+    minio.download_file.assert_called_once_with(
         object_name='training_datasets/m/f.parquet',
         metadata={'metadata': {}},
     )
@@ -113,21 +110,19 @@ def test_parse_object_timestamp_bad_datetime():
     assert MinioDataFramePayload.parse_object_timestamp(key) is None
 
 
-@pytest.mark.asyncio
+def test_retrieve_empty_when_no_data():
-async def test_retrieve_empty_when_no_data():
     payload = MinioDataFramePayload(last_timestamp='t', data=None, object_key=None)
-    minio = AsyncMock()
+    minio = MagicMock()
-    out = await payload.retrieve(minio, {})
+    out = payload.retrieve(minio, {})
     assert out.empty
     minio.download_file.assert_not_called()
 
 
-@pytest.mark.asyncio
 @patch('laborious.utils.models.minio_dataframe_payload.now')
-async def test_from_dataframe_none(mock_now):
+def test_from_dataframe_none(mock_now):
     mock_now.return_value = datetime(2024, 1, 1, 0, 0, 0)
-    minio = AsyncMock()
+    minio = MagicMock()
-    result = await MinioDataFramePayload.from_dataframe(
+    result = MinioDataFramePayload.from_dataframe(
         dataframe=None,
         minio_repo=minio,
         model_name='m',
@@ -139,15 +134,14 @@ async def test_from_dataframe_none(mock_now):
     assert result.object_key is None
 
 
-@pytest.mark.asyncio
 @patch('laborious.utils.models.minio_dataframe_payload.now')
-async def test_from_dataframe_empty(mock_now):
+def test_from_dataframe_empty(mock_now):
     mock_now.return_value = datetime(2024, 1, 1, 0, 0, 0)
-    minio = AsyncMock()
+    minio = MagicMock()
     mock_df = MagicMock()
     mock_df.__bool__ = MagicMock(return_value=True)
     mock_df.empty = True
-    result = await MinioDataFramePayload.from_dataframe(
+    result = MinioDataFramePayload.from_dataframe(
         dataframe=mock_df,
         minio_repo=minio,
         model_name='m',
@@ -174,12 +168,11 @@ def _mock_dataframe(data_dict, timestamp_values=None):
     return mock_df
 
 
-@pytest.mark.asyncio
 @patch('laborious.utils.models.minio_dataframe_payload.OFFLOAD_THRESHOLD_BYTES', 10**9)
-async def test_from_dataframe_inline():
+def test_from_dataframe_inline():
-    minio = AsyncMock()
+    minio = MagicMock()
     df = _mock_dataframe({'timestamp': ['2024-01-01'], 'value': [42]})
-    result = await MinioDataFramePayload.from_dataframe(
+    result = MinioDataFramePayload.from_dataframe(
         dataframe=df,
         minio_repo=minio,
         model_name='m',
@@ -190,17 +183,30 @@ async def test_from_dataframe_inline():
     assert result.last_timestamp == '2024-01-01'
 
 
-@pytest.mark.asyncio
+@patch('laborious.utils.models.minio_dataframe_payload.OFFLOAD_THRESHOLD_BYTES', 10**9)
+def test_from_dataframe_inline_uses_provided_last_timestamp():
+    minio = MagicMock()
+    df = _mock_dataframe({'timestamp': ['2024-01-01'], 'value': [42]})
+    result = MinioDataFramePayload.from_dataframe(
+        dataframe=df,
+        minio_repo=minio,
+        model_name='m',
+        operation='initial',
+        last_timestamp='2024-01-02',
+    )
+    assert result.last_timestamp == '2024-01-02'
+
+
 @patch('laborious.utils.models.minio_dataframe_payload.now')
 @patch('laborious.utils.models.minio_dataframe_payload.OFFLOAD_THRESHOLD_BYTES', 0)
-async def test_from_dataframe_offloaded(mock_now):
+def test_from_dataframe_offloaded(mock_now):
     mock_now.return_value = datetime(2024, 1, 1, 0, 0, 0)
-    minio = AsyncMock()
+    minio = MagicMock()
-    minio.upload_file = AsyncMock(return_value={'minio_object_name': 'full/key.parquet'})
+    minio.upload_file = MagicMock(return_value={'minio_object_name': 'full/key.parquet'})
     minio.bucket = 'test-bucket'
 
     df = _mock_dataframe({'timestamp': ['2024-01-01'], 'value': [42]})
-    result = await MinioDataFramePayload.from_dataframe(
+    result = MinioDataFramePayload.from_dataframe(
         dataframe=df,
         minio_repo=minio,
         model_name='m',
@@ -211,7 +217,7 @@ async def test_from_dataframe_offloaded(mock_now):
     assert result.object_key == 'full/key.parquet'
     assert result.bucket == 'test-bucket'
     assert result.uri == 's3://test-bucket/full/key.parquet'
-    minio.upload_file.assert_awaited_once()
+    minio.upload_file.assert_called_once()
 
 
 def test_from_dict_inline():
@@ -264,3 +270,9 @@ def test_from_dict_passthrough_existing_instance():
     original = MinioDataFramePayload(last_timestamp='2024-01-01', data={'a': 1}, bucket='b')
     result = MinioDataFramePayload.from_dict(original)
     assert result is original
+
+
+def test_debug_with_logger_calls_custom_debug():
+    logger = MagicMock()
+    MinioDataFramePayload._debug(logger, 'msg', {'a': 1})
+    logger.custom_debug.assert_called_once_with('msg', {'a': 1})

tests/laborious/utils/repository/test_opc_repository.py

@@ -1,12 +1,20 @@
+import concurrent.futures
 import json
 from datetime import datetime
-from unittest.mock import ANY, AsyncMock, MagicMock, Mock, patch
+from unittest.mock import ANY, MagicMock, Mock, patch
 
 import pytest
-from asyncua.crypto.security_policies import SecurityPolicyBasic256
+from asyncua.crypto import security_policies
+from asyncua.ua.uaerrors import BadNodeIdUnknown, BadSessionIdInvalid
 from sientia_do.notifications.models import NotificationLevel
 
-from laborious.utils.repository.opc_repository import OpcRepository
+from laborious.utils.repository.opc_repository import (
+    OpcClientAlreadyExistsError,
+    OpcClientNotInitializedError,
+    OpcRepository,
+    OpcSessionAlreadyConnectedError,
+    is_reconnectable_opcua_bad,
+)
 
 
 @pytest.fixture
@@ -27,19 +35,30 @@ def opc_repository(mock_logger):
         cert_path='/path/to/cert.pem',
         private_key_path='/path/to/key.pem',
         server_cert_path='/path/to/server_cert.pem',
-        metrics_controller=AsyncMock(),
+        metrics_controller=MagicMock(),
     )
     repository.disconnection_interval = 0.1
     repository.send_notification = MagicMock()
-    repository.send_notification_async = AsyncMock()
+    repository.send_notification = MagicMock()
-    repository.emit_metric = AsyncMock()
+    repository.emit_metric_sync = MagicMock()
+    repository.info = MagicMock()
+    repository.error = MagicMock()
+    repository.warning = MagicMock()
+    repository.debug = MagicMock()
+    repository._session_ready.set()
     return repository
 
 
 @pytest.fixture
 def mock_client():
     with patch('laborious.utils.repository.opc_repository.Client') as mock:
-        client_instance = AsyncMock()
+        client_instance = MagicMock()
+        aio = MagicMock()
+        client_instance.aio_obj = aio
+        aio.uaclient = MagicMock()
+        aio.uaclient.protocol = MagicMock(state='closed')
+        aio.session_timeout = 600_000
+        aio.secure_channel_timeout = 600_000
         mock.return_value = client_instance
         yield client_instance
 
@@ -65,89 +84,129 @@ def test_init(opc_repository):
     assert opc_repository.reconnection_interval == 60
     assert opc_repository.client is None
     assert opc_repository.last_reconnection_time is None
-    assert opc_repository.error_count == 0
 
 
-@pytest.mark.asyncio
+def test_set_security(opc_repository, mock_client):
-async def test_set_security(opc_repository, mock_client):
     opc_repository.client = mock_client
-    await opc_repository.set_security()
+    opc_repository.set_security()
 
     mock_client.application_uri = 'urn:test:server'
     mock_client.set_security.assert_called_once_with(
-        SecurityPolicyBasic256,
+        security_policies.SecurityPolicyBasic256,
-        certificate='/path/to/cert.pem',
+        '/path/to/cert.pem',
-        private_key='/path/to/key.pem',
+        '/path/to/key.pem',
-        server_certificate='/path/to/server_cert.pem',
+        None,
+        '/path/to/server_cert.pem',
     )
-    assert mock_client.secure_channel_timeout == 10000000
+    assert mock_client.aio_obj.secure_channel_timeout == 600_000
-    assert mock_client.session_timeout == 10000000
+    assert mock_client.aio_obj.session_timeout == 600_000
 
 
-@pytest.mark.asyncio
+def test_set_security_missing_certificates(opc_repository):
-async def test_set_security_missing_certificates(opc_repository):
     opc_repository.cert_path = None
     opc_repository.private_key_path = None
 
     try:
-        await opc_repository.set_security()
+        opc_repository.set_security()
     except ValueError as e:
         assert str(e) == 'Certificate and private key paths must be provided for secure connection.'
 
 
-@pytest.mark.asyncio
+def test_set_security_missing_client(opc_repository):
-async def test_set_security_missing_client(opc_repository):
     opc_repository.client = None
     try:
-        await opc_repository.set_security()
+        opc_repository.set_security()
     except ValueError as e:
         assert str(e) == 'Client must be initialized before setting security'
 
 
-@pytest.mark.asyncio
+def test_connect_with_security(opc_repository, mock_client):
-async def test_connect_with_security(opc_repository, mock_client):
+    opc_repository._create_client = MagicMock()
-    opc_repository.try_connect = AsyncMock(return_value=(True, {}))
+    opc_repository._open_session = MagicMock(return_value=(True, {}))
-    result = await opc_repository.connect()
+    result = opc_repository.connect()
 
-    opc_repository.try_connect.assert_called_once()
+    opc_repository._create_client.assert_called_once()
-    assert opc_repository.client == mock_client
+    opc_repository._open_session.assert_called_once()
     assert result == (True, {})
 
 
-@pytest.mark.asyncio
+def test_connect_without_security(opc_repository, mock_client):
-async def test_connect_without_security(opc_repository, mock_client):
     opc_repository.cert_path = None
-    opc_repository.try_connect = AsyncMock(return_value=(True, {}))
+    opc_repository._create_client = MagicMock()
-    opc_repository.set_security = AsyncMock()
+    opc_repository._open_session = MagicMock(return_value=(True, {}))
-    result = await opc_repository.connect()
+    opc_repository.set_security = MagicMock()
+    result = opc_repository.connect()
 
-    opc_repository.try_connect.assert_called_once()
+    opc_repository._create_client.assert_called_once()
+    opc_repository._open_session.assert_called_once()
     opc_repository.set_security.assert_not_called()
-    assert opc_repository.client == mock_client
     assert result == (True, {})
 
 
-@pytest.mark.asyncio
+def test_connect_raises_when_session_already_open(opc_repository, mock_client):
-async def test_try_connect_success(opc_repository):
+    opc_repository.client = mock_client
-    opc_repository.last_reconnection_time = None
+    proto = MagicMock()
-    opc_repository.client = AsyncMock()
+    proto.state = 'open'
-    result = await opc_repository.try_connect()
+    mock_client.aio_obj.uaclient.protocol = proto
+
+    with pytest.raises(OpcSessionAlreadyConnectedError, match='disconnect'):
+        opc_repository.connect()
+
+
+def test_create_client_raises_when_client_exists(opc_repository, mock_client):
+    opc_repository.client = mock_client
+
+    with pytest.raises(OpcClientAlreadyExistsError, match='already exists'):
+        opc_repository._create_client()
+
+
+def test_open_session_success(opc_repository):
+    closed_proto = MagicMock()
+    closed_proto.state = 'closed'
+    opc_repository.client = MagicMock()
+    aio = MagicMock()
+    opc_repository.client.aio_obj = aio
+    aio.uaclient = MagicMock(protocol=closed_proto)
+    aio.session_timeout = 600_000
+    aio.secure_channel_timeout = 600_000
+
+    open_proto = MagicMock()
+    open_proto.state = 'open'
+    open_proto.authentication_token = 'tok'
+
+    def connect_side_effect():
+        aio.uaclient.protocol = open_proto
+
+    opc_repository.client.connect = MagicMock(side_effect=connect_side_effect)
+
+    result = opc_repository._open_session()
 
     opc_repository.client.connect.assert_called_once()
-    assert opc_repository.last_reconnection_time is not None
     assert result == (True, {})
+    assert opc_repository._session_ready.is_set()
 
 
-@pytest.mark.asyncio
+def test_open_session_raises_when_already_connected(opc_repository, mock_client):
-async def test_try_connect_fail(opc_repository):
+    opc_repository.client = mock_client
-    opc_repository.last_reconnection_time = None
+    proto = MagicMock()
-    opc_repository.disconnect = AsyncMock()
+    proto.state = 'open'
+    mock_client.aio_obj.uaclient.protocol = proto
+
+    with pytest.raises(OpcSessionAlreadyConnectedError, match='disconnect'):
+        opc_repository._open_session()
+
+
+def test_open_session_fail(opc_repository):
+    opc_repository._disconnect_locked = MagicMock()
     opc_repository.client = MagicMock()
-    opc_repository.client.connect.side_effect = Exception('Test error')
+    aio = MagicMock()
+    opc_repository.client.aio_obj = aio
+    aio.uaclient = MagicMock(protocol=MagicMock(state='closed'))
+    opc_repository.client.connect = MagicMock(side_effect=Exception('Test error'))
 
-    is_connected, error_data = await opc_repository.try_connect()
+    is_connected, error_data = opc_repository._open_session()
 
-    opc_repository.disconnect.assert_called_once()
+    opc_repository._disconnect_locked.assert_called_once()
     opc_repository.client.connect.assert_called_once()
     assert is_connected is False
     assert error_data['notification_id'] == f'OPC_CONNECTION_ERROR_{opc_repository.id}'
@@ -157,36 +216,26 @@ async def test_try_connect_fail(opc_repository):
     assert error_data['attachment_content'] is not None
 
 
-@pytest.mark.asyncio
+def test_open_session_raises_when_no_client(opc_repository):
-async def test_try_connect_no_client(opc_repository):
     opc_repository.client = None
-    result = await opc_repository.try_connect()
+
-    assert result == (
+    with pytest.raises(OpcClientNotInitializedError, match='not initialized'):
-        False,
+        opc_repository._open_session()
-        {
-            'notification_id': f'OPC_CONNECTION_ERROR_{opc_repository.id}',
-            'message': 'Client is not initialized',
-            'block': 'opc_repository',
-            'level': NotificationLevel.ERROR,
-        },
-    )
 
 
-@pytest.mark.asyncio
+def test_disconnection_fallback_success(opc_repository, mock_client):
-async def test_disconnection_fallback_success(opc_repository, mock_client):
     opc_repository.client = mock_client
     mock_client.disconnect.return_value = True
-    result = await opc_repository.disconnection_fallback()
+    result = opc_repository._disconnection_fallback()
 
     mock_client.disconnect.assert_called_once()
     assert result == []
 
 
-@pytest.mark.asyncio
+def test_disconnection_fallback_fail(opc_repository, mock_client):
-async def test_disconnection_fallback_fail(opc_repository, mock_client):
     opc_repository.client = mock_client
     mock_client.disconnect.side_effect = Exception('Test error')
-    result = await opc_repository.disconnection_fallback()
+    result = opc_repository._disconnection_fallback()
     assert result == [
         {'attempt': 1, 'error': 'Test error', 'traceback': ANY},
         {'attempt': 2, 'error': 'Test error', 'traceback': ANY},
@@ -197,32 +246,30 @@ async def test_disconnection_fallback_fail(opc_repository, mock_client):
     assert mock_client.disconnect.call_count == 5
 
 
-@pytest.mark.asyncio
+def test_disconnect(opc_repository, mock_client):
-async def test_disconnect(opc_repository, mock_client):
     opc_repository.client = mock_client
-    opc_repository.disconnection_fallback = AsyncMock(return_value=[])
+    opc_repository._disconnection_fallback = MagicMock(return_value=[])
-    await opc_repository.disconnect()
+    opc_repository.disconnect()
 
-    opc_repository.disconnection_fallback.assert_called_once()
+    opc_repository._disconnection_fallback.assert_called_once()
     assert opc_repository.client is None
+    assert opc_repository._allow_reconnect is False
 
 
-@pytest.mark.asyncio
+def test_disconnect_no_client(opc_repository):
-async def test_disconnect_no_client(opc_repository):
     opc_repository.client = None
-    assert await opc_repository.disconnect() is None
+    assert opc_repository.disconnect() is None
 
 
-@pytest.mark.asyncio
+def test_disconnect_error(opc_repository, mock_client):
-async def test_disconnect_error(opc_repository, mock_client):
     opc_repository.client = mock_client
-    opc_repository.disconnection_fallback = AsyncMock(
+    opc_repository._disconnection_fallback = MagicMock(
         return_value=[{'attempt': 1, 'error': 'Test error', 'traceback': 'text'}]
     )
-    await opc_repository.disconnect()
+    opc_repository.disconnect()
 
-    opc_repository.disconnection_fallback.assert_called_once()
+    opc_repository._disconnection_fallback.assert_called_once()
-    opc_repository.send_notification_async.assert_called_once_with(
+    opc_repository.send_notification.assert_called_once_with(
         metadata=opc_repository.metadata,
         notification_id=f'OPC_DISCONNECTION_ERROR_{opc_repository.id}',
         message='Failed to disconnect from OPC server in 5 attempts.',
@@ -235,141 +282,70 @@ async def test_disconnect_error(opc_repository, mock_client):
     assert opc_repository.client is None
 
 
-@pytest.mark.asyncio
+def test_validate_connection_none_client(opc_repository):
-async def test_validate_connection_none_client(opc_repository):
     opc_repository.client = None
-    opc_repository.connect = AsyncMock(return_value=(True, {}))
+    response = opc_repository.validate_connection()
-    response = await opc_repository.validate_connection()
+    assert response == (False, opc_repository._not_connected_error())
-    assert response == (True, {})
-    opc_repository.connect.assert_called_once()
 
 
-# @pytest.mark.asyncio
+def test_validate_connection_session_not_open(opc_repository):
-# async def test_validate_connection_error_count_disconnect_error(opc_repository):
-#     opc_repository.error_count = 6
-#     opc_repository.client = AsyncMock()
-#     opc_repository.disconnect = AsyncMock(side_effect=Exception('Test error'))
-#     opc_repository.connect = AsyncMock(return_value=(True, {}))
-
-#     response = await opc_repository.validate_connection()
-#     assert response == opc_repository.connect.return_value
-#     opc_repository.disconnect.assert_called_once()
-#     opc_repository.connect.assert_called_once()
-#     opc_repository.logger.custom_error.assert_has_calls(
-#         [
-#             call('Failed to disconnect from OPC server: Test error', ANY),
-#         ]
-#     )
-
-
-@pytest.mark.asyncio
-async def test_validate_connection_error_validate_connection_error(opc_repository):
-    opc_repository.client = MagicMock(uaclient=Exception('Test error'))
-    opc_repository.error_count = 0
-
-    response = await opc_repository.validate_connection()
-
-    assert response == (
-        False,
-        {
-            'notification_id': f'OPC_CONNECTION_CHECK_ERROR_{opc_repository.id}',
-            'message': "Failed to validate connection to OPC server: 'Exception' object has no attribute 'protocol'",
-            'block': 'opc_repository',
-            'level': NotificationLevel.ERROR,
-            'attachment_content': ANY,
-        },
-    )
-
-
-@pytest.mark.asyncio
-@patch('laborious.utils.repository.opc_repository.datetime')
-async def test_validate_connection_lost_not_time_to_reconnect(_mock_datetime, opc_repository):
-    _mock_datetime.now = MagicMock(return_value=datetime(2025, 1, 1, 0, 0, 0))
-    opc_repository.error_count = 0
     opc_repository.client = MagicMock()
-    opc_repository.client.uaclient.protocol = None
+    opc_repository.client.aio_obj.uaclient.protocol = None
-    opc_repository.last_reconnection_time = datetime(2025, 1, 1, 0, 0, 0)
-    opc_repository.connect = MagicMock(return_value=(True, {}))
 
-    response = await opc_repository.validate_connection()
+    response = opc_repository.validate_connection()
-    opc_repository.connect.assert_not_called()
+
-    assert response == (
+    assert response == (False, opc_repository._not_connected_error())
-        False,
+    opc_repository.error.assert_called_once()
-        {
-            'notification_id': f'OPC_CONNECTION_AWAITING_RECONNECTION_WINDOW_{opc_repository.id}',
-            'message': f'OPC server {opc_repository.id} is not connected, waiting for next reconnection window...',
-            'block': 'opc_repository',
-            'level': NotificationLevel.WARNING,
-        },
-    )
 
 
-@pytest.mark.asyncio
+def test_validate_connection_success(opc_repository):
-@patch('laborious.utils.repository.opc_repository.datetime')
-async def test_validate_connection_lost_time_to_reconnect(mock_datetime, opc_repository):
-    mock_datetime.now = MagicMock(return_value=datetime(2025, 1, 1, 1, 0, 0))
-    opc_repository.error_count = 0
-    opc_repository.client = AsyncMock()
-    opc_repository.client.uaclient.protocol = None
-    opc_repository.last_reconnection_time = datetime(2025, 1, 1, 0, 0, 0)
-    opc_repository.connect = AsyncMock(return_value=(True, {}))
-
-    response = await opc_repository.validate_connection()
-    opc_repository.connect.assert_called_once()
-    assert response == opc_repository.connect.return_value
-
-
-@pytest.mark.asyncio
-async def test_validate_connection_success(opc_repository):
     opc_repository.client = MagicMock()
-    opc_repository.error_count = 0
+    proto = MagicMock()
-    opc_repository.client.uaclient.protocol = MagicMock()
+    proto.state = 'open'
-    opc_repository.client.uaclient.protocol.state = 'open'
+    opc_repository.client.aio_obj.uaclient.protocol = proto
 
-    output = await opc_repository.validate_connection()
+    output = opc_repository.validate_connection()
     assert output == (True, {})
 
 
-@pytest.mark.asyncio
+def test_write_data_validate_connection_do_nothing(opc_repository):
-async def test_write_data_validate_connection_do_nothing(opc_repository):
+    opc_repository.validate_connection = MagicMock(return_value=(True, {}))
-    opc_repository.validate_connection = AsyncMock(return_value=(True, {}))
+    opc_repository.client = MagicMock(get_node=MagicMock())
-    opc_repository.client = AsyncMock(get_node=MagicMock())
+    mock_node = MagicMock()
-    mock_node = AsyncMock()
     opc_repository.client.get_node.return_value = mock_node
 
-    result = await opc_repository.write_data(
+    result = opc_repository.write_data('ns=2;s=TestNode', 42.0, 'float', metadata['metadata'])
-        'ns=2;s=TestNode', 42.0, 'float', opc_repository.logger, metadata['metadata']
-    )
 
     opc_repository.validate_connection.assert_called_once()
     opc_repository.client.get_node.assert_called_once_with('ns=2;s=TestNode')
     assert result == (True, {'response_time': ANY})
 
 
-@pytest.mark.asyncio
+def test_write_data_validate_connection_failed(opc_repository):
-async def test_write_data_validate_connection_failed(opc_repository):
+    opc_repository.validate_connection = MagicMock(return_value=(False, {}))
-    opc_repository.validate_connection = AsyncMock(return_value=(False, {}))
+    opc_repository.client = MagicMock()
-    opc_repository.client = AsyncMock()
+    opc_repository._start_reconnect = MagicMock()
-    opc_repository.error_count = 0
 
-    result = await opc_repository.write_data(
+    is_success, error_data = opc_repository.write_data(
-        'ns=2;s=TestNode', 42.0, 'float', opc_repository.logger, metadata['metadata']
+        'ns=2;s=TestNode', 42.0, 'float', metadata['metadata']
     )
 
     opc_repository.validate_connection.assert_called_once()
     opc_repository.client.get_node.assert_not_called()
-    assert result == (False, {})
+    opc_repository._start_reconnect.assert_called_once()
+    assert opc_repository._start_reconnect.call_args.args[0] == 'ProtocolClosed'
+    assert is_success is False
+    assert error_data['opc_error_kind'] == 'connection_lost'
+    assert error_data['opc_status'] == 'ProtocolClosed'
 
 
-@pytest.mark.asyncio
+def test_write_data_get_node_failed(opc_repository):
-async def test_write_data_get_node_failed(opc_repository):
+    opc_repository.validate_connection = MagicMock(return_value=(True, {}))
-    opc_repository.validate_connection = AsyncMock(return_value=(True, {}))
+    opc_repository.client = MagicMock()
-    opc_repository.client = AsyncMock()
-    opc_repository.error_count = 0
     opc_repository.client.get_node = MagicMock(side_effect=Exception('Test error'))
 
-    is_success, error_data = await opc_repository.write_data(
+    is_success, error_data = opc_repository.write_data(
-        'ns=2;s=TestNode', 42.0, 'float', opc_repository.logger, metadata['metadata']
+        'ns=2;s=TestNode', 42.0, 'float', metadata['metadata']
     )
 
     opc_repository.validate_connection.assert_called_once()
@@ -385,15 +361,14 @@ async def test_write_data_get_node_failed(opc_repository):
     assert error_data['attachment_content'] is not None
 
 
-@pytest.mark.asyncio
+def test_write_data_invalid_data_type(opc_repository, mock_client):
-async def test_write_data_invalid_data_type(opc_repository, mock_client):
+    opc_repository.validate_connection = MagicMock(return_value=(True, {}))
-    opc_repository.validate_connection = AsyncMock(return_value=(True, {}))
     opc_repository.client = mock_client
-    mock_node = AsyncMock()
+    mock_node = MagicMock()
     mock_client.get_node = MagicMock(return_value=mock_node)
 
-    is_success, error_data = await opc_repository.write_data(
+    is_success, error_data = opc_repository.write_data(
-        'ns=2;s=TestNode', 42.0, 'invalid_type', opc_repository.logger, metadata['metadata']
+        'ns=2;s=TestNode', 42.0, 'invalid_type', metadata['metadata']
     )
 
     opc_repository.validate_connection.assert_called_once()
@@ -410,33 +385,28 @@ async def test_write_data_invalid_data_type(opc_repository, mock_client):
     assert error_data.get('attachment_content') is None
 
 
-@pytest.mark.asyncio
+def test_write_data(opc_repository, mock_client):
-async def test_write_data(opc_repository, mock_client):
+    opc_repository.validate_connection = MagicMock(return_value=(True, {}))
-    opc_repository.validate_connection = AsyncMock(return_value=(True, {}))
     opc_repository.client = mock_client
-    mock_node = AsyncMock()
+    mock_node = MagicMock()
     mock_client.get_node = MagicMock(return_value=mock_node)
 
-    result = await opc_repository.write_data(
+    result = opc_repository.write_data('ns=2;s=TestNode', 42.0, 'float', metadata['metadata'])
-        'ns=2;s=TestNode', 42.0, 'float', opc_repository.logger, metadata['metadata']
-    )
 
     mock_client.get_node.assert_called_once_with('ns=2;s=TestNode')
     mock_node.write_value.assert_called_once()
     assert result == (True, {'response_time': ANY})
 
 
-@pytest.mark.asyncio
+def test_write_data_write_value_failed(opc_repository, mock_client):
-async def test_write_data_write_value_failed(opc_repository, mock_client):
+    opc_repository.validate_connection = MagicMock(return_value=(True, {}))
-    opc_repository.validate_connection = AsyncMock(return_value=(True, {}))
     opc_repository.client = mock_client
-    mock_node = AsyncMock()
+    mock_node = MagicMock()
-    opc_repository.error_count = 0
     mock_client.get_node = MagicMock(return_value=mock_node)
     mock_node.write_value.side_effect = Exception('Test error')
 
-    is_success, error_data = await opc_repository.write_data(
+    is_success, error_data = opc_repository.write_data(
-        'ns=2;s=TestNode', 42.0, 'float', opc_repository.logger, metadata['metadata']
+        'ns=2;s=TestNode', 42.0, 'float', metadata['metadata']
     )
 
     opc_repository.validate_connection.assert_called_once()
@@ -451,3 +421,164 @@ async def test_write_data_write_value_failed(opc_repository, mock_client):
     assert error_data['block'] == 'opc_repository'
     assert error_data['level'] == NotificationLevel.ERROR
     assert error_data['attachment_content'] is not None
+
+
+def test_is_reconnectable_opcua_bad():
+    assert is_reconnectable_opcua_bad(BadSessionIdInvalid()) is True
+    assert is_reconnectable_opcua_bad(BadNodeIdUnknown()) is False
+    assert is_reconnectable_opcua_bad(Exception('other')) is False
+
+
+def test_write_data_bad_session_id_invalid_schedules_reconnect(opc_repository, mock_client):
+    opc_repository.validate_connection = MagicMock(return_value=(True, {}))
+    opc_repository.client = mock_client
+    opc_repository._start_reconnect = MagicMock()
+    mock_node = MagicMock()
+    mock_client.get_node = MagicMock(return_value=mock_node)
+    mock_node.write_value.side_effect = BadSessionIdInvalid()
+
+    is_success, error_data = opc_repository.write_data(
+        'ns=2;s=TestNode', 42.0, 'float', metadata['metadata']
+    )
+
+    mock_node.write_value.assert_called_once()
+    opc_repository._start_reconnect.assert_called_once()
+    assert is_success is False
+    assert error_data['opc_error_kind'] == 'session_bad'
+    assert error_data['opc_status'] == 'BadSessionIdInvalid'
+
+
+def test_write_data_reconnect_in_progress_immediate(opc_repository):
+    opc_repository._session_ready.clear()
+    opc_repository._reconnect_thread = MagicMock()
+    opc_repository._reconnect_thread.is_alive.return_value = True
+    opc_repository.validate_connection = MagicMock()
+
+    is_success, error_data = opc_repository.write_data(
+        'ns=2;s=TestNode', 42.0, 'float', metadata['metadata']
+    )
+
+    opc_repository.validate_connection.assert_not_called()
+    assert is_success is False
+    assert error_data['opc_error_kind'] == 'reconnect_in_progress'
+
+
+def test_start_reconnect_skips_within_interval(opc_repository):
+    opc_repository.last_reconnection_time = datetime.now()
+    opc_repository.reconnection_interval = 3600
+
+    opc_repository._start_reconnect('BadSessionIdInvalid', 'tok')
+
+    assert opc_repository._reconnect_thread is None
+
+
+def test_write_data_protocol_closed_schedules_reconnect(opc_repository):
+    opc_repository.client = MagicMock()
+    opc_repository.client.aio_obj.uaclient.protocol = MagicMock(state='closed')
+    opc_repository._start_reconnect = MagicMock()
+
+    is_success, error_data = opc_repository.write_data(
+        'ns=2;s=TestNode', 42.0, 'float', metadata['metadata']
+    )
+
+    opc_repository._start_reconnect.assert_called_once()
+    assert opc_repository._start_reconnect.call_args.args[0] == 'ProtocolClosed'
+    assert is_success is False
+    assert error_data['opc_error_kind'] == 'connection_lost'
+    assert error_data['opc_status'] == 'ProtocolClosed'
+
+
+def test_write_data_protocol_closed_skips_reconnect_within_interval(opc_repository):
+    opc_repository.client = MagicMock()
+    opc_repository.client.aio_obj.uaclient.protocol = MagicMock(state='closed')
+    opc_repository.last_reconnection_time = datetime.now()
+    opc_repository.reconnection_interval = 3600
+
+    is_success, error_data = opc_repository.write_data(
+        'ns=2;s=TestNode', 42.0, 'float', metadata['metadata']
+    )
+
+    assert opc_repository._reconnect_thread is None
+    assert is_success is False
+    assert error_data['opc_error_kind'] == 'connection_lost'
+
+
+def test_write_data_after_failed_reconnect_schedules_again(opc_repository):
+    opc_repository._session_ready.clear()
+    opc_repository.reconnection_interval = 0
+    opc_repository.last_reconnection_time = None
+    opc_repository._reconnect_locked = MagicMock(
+        return_value=(False, {'message': 'connect failed'})
+    )
+
+    opc_repository.write_data('ns=2;s=TestNode', 42.0, 'float', metadata['metadata'])
+    if opc_repository._reconnect_thread is not None:
+        opc_repository._reconnect_thread.join(timeout=2)
+    assert opc_repository._reconnect_locked.call_count == 1
+    assert not opc_repository._reconnect_thread_in_progress()
+
+    opc_repository.write_data('ns=2;s=TestNode', 42.0, 'float', metadata['metadata'])
+    if opc_repository._reconnect_thread is not None:
+        opc_repository._reconnect_thread.join(timeout=2)
+    assert opc_repository._reconnect_locked.call_count == 2
+
+
+def test_write_data_after_disconnect_does_not_schedule_reconnect(opc_repository, mock_client):
+    opc_repository.client = mock_client
+    proto = MagicMock()
+    proto.state = 'closed'
+    mock_client.aio_obj.uaclient.protocol = proto
+    opc_repository._disconnection_fallback = MagicMock(return_value=[])
+    opc_repository.disconnect()
+
+    is_success, error_data = opc_repository.write_data(
+        'ns=2;s=TestNode', 42.0, 'float', metadata['metadata']
+    )
+
+    assert opc_repository._reconnect_thread is None
+    assert is_success is False
+    assert error_data['opc_error_kind'] == 'connection_lost'
+
+
+def test_parallel_bad_writes_single_reconnect_task(opc_repository, mock_client):
+    opc_repository.validate_connection = MagicMock(return_value=(True, {}))
+    opc_repository.client = mock_client
+    opc_repository.reconnection_interval = 0
+    opc_repository.last_reconnection_time = None
+    mock_node = MagicMock()
+    mock_client.get_node = MagicMock(return_value=mock_node)
+    mock_node.write_value.side_effect = BadSessionIdInvalid()
+    opc_repository._start_reconnect = MagicMock()
+
+    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
+        futures = [
+            executor.submit(
+                opc_repository.write_data,
+                node,
+                value,
+                'float',
+                metadata['metadata'],
+            )
+            for node, value in (('ns=2;s=TestNode', 1.0), ('ns=2;s=TestNode2', 2.0))
+        ]
+        results = [future.result() for future in futures]
+
+    assert 1 <= opc_repository._start_reconnect.call_count <= 2
+    assert mock_node.write_value.call_count == 2
+    error_kinds = [r[1].get('opc_error_kind') for r in results]
+    assert error_kinds.count('session_bad') >= 1
+    assert all(k in ('session_bad', 'reconnect_in_progress') for k in error_kinds)
+
+
+@patch('laborious.utils.repository.opc_repository.datetime')
+def test_reconnect_locked_sets_last_reconnection_time(mock_datetime, opc_repository):
+    mock_datetime.now = MagicMock(return_value=datetime(2025, 1, 1, 12, 0, 0))
+    opc_repository._disconnect_locked = MagicMock()
+    opc_repository._connect_locked = MagicMock(return_value=(True, {}))
+
+    result = opc_repository._reconnect_locked()
+
+    opc_repository._disconnect_locked.assert_called_once()
+    opc_repository._connect_locked.assert_called_once()
+    assert result == (True, {})
+    assert opc_repository.last_reconnection_time == datetime(2025, 1, 1, 12, 0, 0)

tests/laborious/utils/test_connectors_config.py

@@ -4,13 +4,13 @@ from laborious.utils.connectors_config import (
     build_minio_config,
     build_mlflow_config,
     build_opc_config,
+    build_plugin_store_config,
 )
 
 
 def test_build_mlflow_config_with_env_vars():
     # Arrange
-    environ['MLFLOW_HOST'] = 'http://test-host'
+    environ['MLFLOW_URL'] = 'http://test-host:8080'
-    environ['MLFLOW_PORT'] = '8080'
     environ['MLFLOW_USERNAME'] = 'test-user'
     environ['MLFLOW_PASSWORD'] = 'test-pass'
 
@@ -18,17 +18,25 @@ def test_build_mlflow_config_with_env_vars():
     config = build_mlflow_config()
 
     # Assert
-    assert config['host'] == 'http://test-host'
+    assert config['url'] == 'http://test-host:8080'
-    assert config['port'] == 8080
     assert config['username'] == 'test-user'
     assert config['password'] == 'test-pass'
 
 
+def test_build_mlflow_config_host_already_has_port():
+    environ['MLFLOW_URL'] = 'http://tracker.example.com:443'
+    environ['MLFLOW_USERNAME'] = 'u'
+    environ['MLFLOW_PASSWORD'] = 'p'
+
+    config = build_mlflow_config()
+
+    assert config['url'] == 'http://tracker.example.com:443'
+
+
 def test_build_mlflow_config_with_defaults():
     # Arrange
     # Clear any existing env vars
-    environ.pop('MLFLOW_HOST', None)
+    environ.pop('MLFLOW_URL', None)
-    environ.pop('MLFLOW_PORT', None)
     environ.pop('MLFLOW_USERNAME', None)
     environ.pop('MLFLOW_PASSWORD', None)
 
@@ -36,12 +44,30 @@ def test_build_mlflow_config_with_defaults():
     config = build_mlflow_config()
 
     # Assert
-    assert config['host'] == 'http://localhost'
+    assert config['url'] == 'http://localhost:5080'
-    assert config['port'] == 5080
     assert config['username'] == 'aignosi'
     assert config['password'] == 'aignosi'
 
 
+def test_build_plugin_store_config_defaults():
+    environ.pop('STORE_BASE_URL', None)
+    environ.pop('STORE_OWNER', None)
+    environ.pop('STORE_REPO', None)
+    environ.pop('STORE_BRANCH', None)
+    environ.pop('STORE_USERNAME', None)
+    environ.pop('STORE_PASSWORD', None)
+    environ.pop('STORE_CACHE_TTL_SECONDS', None)
+    environ.pop('PYPI_SERVER', None)
+    environ.pop('PYPI_USERNAME', None)
+    environ.pop('PYPI_PASSWORD', None)
+
+    cfg = build_plugin_store_config()
+    assert cfg['base_url'] == 'http://localhost:3000'
+    assert cfg['owner'] == 'sientia'
+    assert cfg['repo'] == 'model-library-store'
+    assert cfg['pypi_index_url'] == 'http://localhost:5000'
+
+
 def test_build_opc_config_with_env_vars():
     # Arrange
     environ['OPC_CONFIG'] = '{"opc": {"name": "test-opc", "url": "opc.tcp://test:4840"}}'
@@ -96,6 +122,8 @@ def test_build_minio_config_with_env_vars():
     environ['MINIO_SECRET_KEY'] = 'test-secret'
     environ['MINIO_REGION_NAME'] = 'test-region'
     environ['MINIO_DEFAULT_BUCKET'] = 'test-bucket'
+    # Isolate from IDE/CI env (e.g. VS Code may export MINIO_SECURE=true).
+    environ['MINIO_SECURE'] = 'false'
     assert build_minio_config() == {
         'endpoint_url': 'http://test-host',
         'access_key': 'test-key',
@@ -112,6 +140,8 @@ def test_build_minio_config_with_defaults():
     environ.pop('MINIO_SECRET_KEY', None)
     environ.pop('MINIO_REGION_NAME', None)
     environ.pop('MINIO_DEFAULT_BUCKET', None)
+    environ.pop('MINIO_SECURE', None)
+    environ.pop('MINIO_RETENTION_HOURS', None)
     assert build_minio_config() == {
         'endpoint_url': 'http://localhost:9000',
         'access_key': 'minioadmin',

tests/laborious/utils/test_dataframe_debug.py

@@ -0,0 +1,12 @@
+from pandas import DataFrame
+
+from laborious.utils.dataframe_debug import build_dataframe_debug_message
+
+
+def test_build_dataframe_debug_message_skips_large_dataframe():
+    df = DataFrame({'a': [1, 2, 3]})
+
+    msg = build_dataframe_debug_message('payload', df, max_rows=1)
+
+    assert 'skipped because dataframe has 3 rows' in msg
+    assert '(max: 1)' in msg

tests/laborious/worker/test_worker.py

@@ -0,0 +1,243 @@
+from unittest.mock import AsyncMock, MagicMock, patch
+
+from pytest import mark, raises
+
+from laborious.worker import worker
+
+
+def _build_fake_activities():
+    inst = MagicMock()
+    inst.init_opc = MagicMock()
+    inst.shutdown = MagicMock()
+    inst.load_query_with_minio_offload = MagicMock()
+    inst.retrain_model = MagicMock()
+    inst.update_production_model = MagicMock()
+    inst.format_retrain_report = MagicMock()
+    inst.export_data_to_postgres = MagicMock()
+    inst.load_custom_query = MagicMock()
+    inst.calculate_simple_metrics = MagicMock()
+    inst.get_reference_data = MagicMock()
+    inst.calculate_drift = MagicMock()
+    inst.request_predict = MagicMock()
+    inst.request_transform = MagicMock()
+    inst.input_gate = MagicMock()
+    inst.mlflow_response_gate = MagicMock()
+    inst.mlflow_content_gate = MagicMock()
+    inst.format_transformed_data = MagicMock()
+    inst.format_prediction = MagicMock()
+    inst.format_default_prediction = MagicMock()
+    inst.write_opc_data = MagicMock()
+    inst.cleanup_minio_objects_expired = MagicMock()
+    inst.repeat_last_prediction = MagicMock()
+    inst.write_metrics = MagicMock()
+    inst.write_pi_web_api_data = MagicMock()
+    return inst
+
+
+def _build_fake_worker(async_result=None, async_error: Exception | None = None):
+    w = MagicMock()
+
+    async def _run():
+        if async_error is not None:
+            raise async_error
+        return async_result
+
+    w.run = MagicMock(side_effect=_run)
+    return w
+
+
+@patch('laborious.worker.worker.start_http_server')
+def test_start_prometheus_server_success(mock_start_http):
+    with patch.object(worker.metrics.APP_UP, 'labels') as labels:
+        gauge = MagicMock()
+        labels.return_value = gauge
+        with patch('laborious.worker.worker.os.getenv', return_value='9090'):
+            worker.start_prometheus_server()
+    mock_start_http.assert_called_once_with(9090)
+    gauge.set.assert_called_once_with(1)
+
+
+@patch('laborious.worker.worker.start_http_server', side_effect=RuntimeError('nope'))
+def test_start_prometheus_server_error_exits(_mock_start_http):
+    with patch('laborious.worker.worker.os._exit', side_effect=SystemExit(1)) as m_exit:
+        with raises(SystemExit):
+            worker.start_prometheus_server()
+    m_exit.assert_called_once_with(1)
+
+
+@mark.asyncio
+async def test_main_missing_runtime_exits_fast(monkeypatch):
+    monkeypatch.setenv('RUNTIME', '')
+    with (
+        patch('laborious.worker.worker.start_prometheus_server'),
+        patch('laborious.worker.worker.get_logger') as m_logger,
+        patch('laborious.worker.worker.NotificationHandler'),
+        patch('laborious.worker.worker.MetricsController'),
+        patch.object(worker.metrics.APP_UP, 'labels') as labels,
+        patch('laborious.worker.worker.sys.exit', side_effect=SystemExit(1)),
+    ):
+        labels.return_value = MagicMock()
+        with raises(SystemExit):
+            await worker.main()
+    assert m_logger.return_value.custom_critical.called
+
+
+@mark.asyncio
+async def test_main_plugin_install_failure(monkeypatch):
+    monkeypatch.setenv('RUNTIME', 'single')
+    fake_activities = _build_fake_activities()
+    fake_plugin = MagicMock()
+    fake_plugin.install_runtime = AsyncMock(side_effect=RuntimeError('install failed'))
+    with (
+        patch('laborious.worker.worker.start_prometheus_server'),
+        patch('laborious.worker.worker.get_logger'),
+        patch(
+            'laborious.worker.worker.build_mongodb_config',
+            return_value={'connection_string': 'cs', 'database_name': 'db'},
+        ),
+        patch('laborious.worker.worker.NotificationHandler'),
+        patch('laborious.worker.worker.MetricsController'),
+        patch(
+            'laborious.worker.worker.build_plugin_store_config',
+            return_value={
+                'base_url': '',
+                'owner': '',
+                'repo': '',
+                'username': None,
+                'password': None,
+                'branch': None,
+                'cache_ttl_seconds': None,
+                'pypi_index_url': '',
+                'pypi_username': None,
+                'pypi_password': None,
+            },
+        ),
+        patch('laborious.worker.worker.PluginStore', return_value=fake_plugin),
+        patch('laborious.worker.worker.Activities', return_value=fake_activities),
+        patch.object(worker.metrics.APP_UP, 'labels') as labels,
+        patch('laborious.worker.worker.sys.exit', side_effect=SystemExit(1)),
+    ):
+        labels.return_value = MagicMock()
+        with raises(SystemExit):
+            await worker.main()
+
+
+@mark.asyncio
+async def test_main_success_exit_zero(monkeypatch):
+    monkeypatch.setenv('RUNTIME', 'single')
+    fake_activities = _build_fake_activities()
+    fake_plugin = MagicMock()
+    fake_plugin.install_runtime = AsyncMock(return_value=None)
+    fake_workers = [_build_fake_worker() for _ in range(4)]
+
+    with (
+        patch('laborious.worker.worker.start_prometheus_server'),
+        patch('laborious.worker.worker.get_logger'),
+        patch(
+            'laborious.worker.worker.build_mongodb_config',
+            return_value={'connection_string': 'cs', 'database_name': 'db'},
+        ),
+        patch('laborious.worker.worker.NotificationHandler') as m_notif_cls,
+        patch('laborious.worker.worker.MetricsController'),
+        patch(
+            'laborious.worker.worker.build_plugin_store_config',
+            return_value={
+                'base_url': '',
+                'owner': '',
+                'repo': '',
+                'username': None,
+                'password': None,
+                'branch': None,
+                'cache_ttl_seconds': None,
+                'pypi_index_url': '',
+                'pypi_username': None,
+                'pypi_password': None,
+            },
+        ),
+        patch('laborious.worker.worker.PluginStore', return_value=fake_plugin),
+        patch('laborious.worker.worker.Activities', return_value=fake_activities),
+        patch('laborious.worker.worker.build_postgres_config', return_value={}),
+        patch('laborious.worker.worker.build_minio_config', return_value={}),
+        patch('laborious.worker.worker.build_opc_config', return_value={}),
+        patch('laborious.worker.worker.build_api_config', return_value={}),
+        patch('laborious.worker.worker.PrometheusConfig', return_value=MagicMock()),
+        patch('laborious.worker.worker.TelemetryConfig', return_value=MagicMock()),
+        patch('laborious.worker.worker.Runtime', return_value=MagicMock()),
+        patch(
+            'laborious.worker.worker.client.Client.connect', new=AsyncMock(return_value=MagicMock())
+        ),
+        patch('laborious.worker.worker.prepare_worker', side_effect=fake_workers) as m_prepare,
+        patch.object(worker.metrics.APP_UP, 'labels') as labels,
+        patch('laborious.worker.worker.sys.exit', side_effect=SystemExit(0)),
+    ):
+        labels.return_value = MagicMock()
+        with raises(SystemExit):
+            await worker.main()
+
+    notif = m_notif_cls.return_value
+    notif.shutdown.assert_called_once()
+    fake_activities.shutdown.assert_called_once()
+    assert m_prepare.call_count == 4
+    prepare_calls = m_prepare.call_args_list
+    assert prepare_calls[0].kwargs['runtime'] == 'single'
+    assert prepare_calls[3].kwargs['runtime'] == 'single'
+
+
+@mark.asyncio
+async def test_main_worker_gather_error_exits_one(monkeypatch):
+    monkeypatch.setenv('RUNTIME', 'single')
+    fake_activities = _build_fake_activities()
+    fake_plugin = MagicMock()
+    fake_plugin.install_runtime = AsyncMock(return_value=None)
+    fake_workers = [
+        _build_fake_worker(async_error=RuntimeError('boom')),
+        _build_fake_worker(),
+        _build_fake_worker(),
+        _build_fake_worker(),
+    ]
+
+    with (
+        patch('laborious.worker.worker.start_prometheus_server'),
+        patch('laborious.worker.worker.get_logger') as m_logger,
+        patch(
+            'laborious.worker.worker.build_mongodb_config',
+            return_value={'connection_string': 'cs', 'database_name': 'db'},
+        ),
+        patch('laborious.worker.worker.NotificationHandler'),
+        patch('laborious.worker.worker.MetricsController'),
+        patch(
+            'laborious.worker.worker.build_plugin_store_config',
+            return_value={
+                'base_url': '',
+                'owner': '',
+                'repo': '',
+                'username': None,
+                'password': None,
+                'branch': None,
+                'cache_ttl_seconds': None,
+                'pypi_index_url': '',
+                'pypi_username': None,
+                'pypi_password': None,
+            },
+        ),
+        patch('laborious.worker.worker.PluginStore', return_value=fake_plugin),
+        patch('laborious.worker.worker.Activities', return_value=fake_activities),
+        patch('laborious.worker.worker.build_postgres_config', return_value={}),
+        patch('laborious.worker.worker.build_minio_config', return_value={}),
+        patch('laborious.worker.worker.build_opc_config', return_value={}),
+        patch('laborious.worker.worker.build_api_config', return_value={}),
+        patch('laborious.worker.worker.PrometheusConfig', return_value=MagicMock()),
+        patch('laborious.worker.worker.TelemetryConfig', return_value=MagicMock()),
+        patch('laborious.worker.worker.Runtime', return_value=MagicMock()),
+        patch(
+            'laborious.worker.worker.client.Client.connect', new=AsyncMock(return_value=MagicMock())
+        ),
+        patch('laborious.worker.worker.prepare_worker', side_effect=fake_workers),
+        patch.object(worker.metrics.APP_UP, 'labels') as labels,
+        patch('laborious.worker.worker.sys.exit', side_effect=SystemExit(1)),
+    ):
+        labels.return_value = MagicMock()
+        with raises(SystemExit):
+            await worker.main()
+
+    assert m_logger.return_value.custom_error.called

tests/laborious/workflows/subworkflows/test_prediction_process.py

@@ -1,6 +1,6 @@
 from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
 
-from pytest import fixture, mark
+from pytest import fixture, mark, raises
 
 from laborious.activities.activities import Activities
 from laborious.workflows.sub_workflows.prediction_process import PredictionProcess
@@ -840,3 +840,27 @@ async def test_run_with_cleanup_prefixes(workflow_mock, prediction_process):
         retry_policy=ANY,
         start_to_close_timeout=ANY,
     )
+
+
+@mark.asyncio
+@patch('laborious.workflows.sub_workflows.prediction_process.workflow', new_callable=AsyncMock)
+async def test_run_always_cleans_up_on_pipeline_exception(workflow_mock, prediction_process):
+    input_data = {
+        'metadata': metadata,
+        'data': {'last_timestamp': '2024-01-01'},
+        'model_id': 1,
+        'model_name': 'm',
+        'model_config': {},
+        'save_transform': False,
+    }
+    prediction_process._run_prediction_pipeline = AsyncMock(side_effect=RuntimeError('boom'))
+
+    with raises(RuntimeError):
+        await prediction_process.run(input_data)
+
+    workflow_mock.execute_activity_method.assert_called_once_with(
+        Activities.cleanup_minio_objects_expired,
+        {**metadata, 'data': input_data['data']},
+        retry_policy=ANY,
+        start_to_close_timeout=ANY,
+    )

tests/laborious/workflows/test_minimal_retrain.py

@@ -34,8 +34,7 @@ async def test_run(workflow_mock: AsyncMock, minimal_retrain: MinimalRetrain):
         'table_name': 'test_table',
         'model_config': {
             'target': 'test_target',
-            'transform_flavor': 'test_transform_flavor',
+            'retention_minutes': 0,
-            'predict_flavor': 'test_predict_flavor',
         },
     }
 
@@ -165,8 +164,7 @@ async def test_run_storage_fail(workflow_mock: AsyncMock, minimal_retrain: Minim
         'table_name': 'test_table',
         'model_config': {
             'target': 'test_target',
-            'transform_flavor': 'test_transform_flavor',
+            'retention_minutes': 0,
-            'predict_flavor': 'test_predict_flavor',
         },
     }
 
@@ -227,8 +225,7 @@ async def test_run_fail_retrain(workflow_mock: AsyncMock, minimal_retrain: Minim
         'table_name': 'test_table',
         'model_config': {
             'target': 'test_target',
-            'transform_flavor': 'test_transform_flavor',
+            'retention_minutes': 0,
-            'predict_flavor': 'test_predict_flavor',
         },
     }
 

values.yaml

@@ -1,69 +1,34 @@
-# Default values for sientia-module.
+#
+# Default values for sientia-laborious-worker using the sientia-module chart (0.6.x).
 # This is a YAML-formatted file.
 # Declare variables to be passed into your templates.
+#
 
-# This will set the replicaset count more information can be found here: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
+projectName: &projectName "sientia-laborious-worker"
-replicaCount: 1
 
-# This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/
+# -----------------------------------------------------------------------------
-image:
+# Global configuration shared by all runtimes
-  repository: aignosi.azurecr.io/sientia-module
+# -----------------------------------------------------------------------------
-  # This sets the pull policy for images.
+global:
-  pullPolicy: Always
-  # Overrides the image tag whose default is the chart appVersion.
-  tag: "1.1.2"
-
-# This is for the secrets for pulling an image from a private repository more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
-imagePullSecrets:
-- name: docker-hub-secret
-# This is to override the chart name.
-nameOverride: "sientia-laborious-worker"
-fullnameOverride: "sientia-laborious-worker"
   namespace: sientia
 
-# This section builds out the service account more information can be found here: https://kubernetes.io/docs/concepts/security/service-accounts/
+  image:
-serviceAccount:
+    repository: aignosi.azurecr.io/sientia-module
-  # Specifies whether a service account should be created
+    pullPolicy: Always
-  create: true
+    tag: "1.2.0"
-  # Automatically mount a ServiceAccount's API credentials?
-  automount: true
-  # Annotations to add to the service account
-  annotations: {}
-  # The name of the service account to use.
-  # If not set and create is true, a name is generated using the fullname template
-  name: "sientia-laborious-worker"
-
-# This is for setting Kubernetes Annotations to a Pod.
-# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
-podAnnotations: {}
-# This is for setting Kubernetes Labels to a Pod.
-# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
-podLabels: {}
-
-podSecurityContext: {}
-  # fsGroup: 2000
-
-securityContext: {}
-  # capabilities:
-  #   drop:
-  #   - ALL
-  # readOnlyRootFilesystem: true
-  # runAsNonRoot: true
-  # runAsUser: 1000
 
+  commonLabels: {}
 
   resources:
     # Resource limits and requests are important for ResourceBasedTuner to work correctly.
     # The tuner monitors system CPU and memory usage, so proper resource limits must be set.
     limits:
-    cpu: 2000m  # 2 CPU cores
+      cpu: 2000m
-    memory: 20Gi  # 20 GB memory
+      memory: 20Gi
     requests:
-    cpu: 1000m  # 1 CPU core
+      cpu: 1000m
-    memory: 2Gi  # 2 GB memory
+      memory: 2Gi
 
-# This is to setup the liveness and readiness probes more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
-# This is to setup the liveness and readiness probes more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
   livenessProbe:
     exec:
       command:
@@ -88,9 +53,6 @@ readinessProbe:
     timeoutSeconds: 3
     failureThreshold: 2
 
-
-
-# This section is for setting up autoscaling more information can be found here: https://kubernetes.io/docs/concepts/workloads/autoscaling/
   autoscaling:
     enabled: false
     minReplicas: 1
@@ -98,68 +60,17 @@ autoscaling:
     targetCPUUtilizationPercentage: 80
     # targetMemoryUtilizationPercentage: 80
 
-# Additional volumes on the output Deployment definition.
+  # Environment variables shared by all runtimes.
-volumes: []
-# - name: foo
-#   secret:
-#     secretName: mysecret
-#     optional: false
-
-# Additional volumeMounts on the output Deployment definition.
-volumeMounts: []
-# - name: foo
-#   mountPath: "/etc/foo"
-#   readOnly: true
-
-nodeSelector: {}
-
-tolerations: []
-
-affinity: {}
-
-services:
-  sdk-metrics:
-    enabled: true
-    type: ClusterIP
-    port: 9091
-    targetPort: 9091
-    name: sdk-metrics
-
-  metrics:
-    enabled: true
-    type: ClusterIP
-    port: 9090
-    targetPort: 9090
-    name: metrics
-
-# Configuração do ServiceMonitor para o Prometheus Operator
-# ref: https://github.com/prometheus-operator/prometheus-operator
-serviceMonitor:
-  # Se true, um recurso ServiceMonitor será criado.
-  enabled: true
-  # O intervalo no qual as métricas devem ser coletadas (ex: 30s, 1m).
-  endpoints:
-  - port: metrics
-    path: /metrics
-    interval: 30s
-    relabelings: []
-  - port: sdk-metrics
-    path: /metrics
-    interval: 30s
-    relabelings: []
-
-  additionalLabels:
-    release: kube-prometheus-stack
-  
-
   env:
     # Entrypoint variables
     - name: GITHUB_REPO_URL
       value: "git@github.com:Aignosi/sientia-dataops-laborious_temporal.git"
     - name: GITHUB_BRANCH
-    value: "feature/SIENTIAPDE-1712"
+      value: "release/SIENTIAPDE-1646"
     - name: PYTHON_APP
       value: "laborious.worker.worker"
+    - name: PYPI_SERVER
+      value: "http://library-distribution-server.library.svc.cluster.local:5000"
 
     # Application variables
     - name: POSTGRES_HOST
@@ -179,15 +90,33 @@ env:
     - name: POSTGRES_MAX_CONNECTIONS
       value: "100"
 
-  - name: MLFLOW_HOST
+    - name: MLFLOW_URL
-    value: "http://sientia-tracker-mlflow-tracking.sientia-tracker.svc.cluster.local"
+      value: "http://sientia-tracker-mlflow-tracking.sientia-tracker.svc.cluster.local:80"
-  - name: MLFLOW_PORT
-    value: "80"
     - name: MLFLOW_USERNAME
       value: "aignosi"
     - name: MLFLOW_PASSWORD
       value: "1L0FP50j3ncp123"
 
+    # Plugin store (model-library-store Git + runtime packages).
+    - name: STORE_BASE_URL
+      value: "http://gitea-http.gitea.svc.cluster.local:3000"
+    - name: STORE_OWNER
+      value: "aignosi"
+    - name: STORE_REPO
+      value: "suse-model-store"
+    - name: STORE_USERNAME
+      valueFrom:
+        secretKeyRef:
+          name: sientia-plugin-store-credentials
+          key: username
+    - name: STORE_PASSWORD
+      valueFrom:
+        secretKeyRef:
+          name: sientia-plugin-store-credentials
+          key: password
+    - name: STORE_CACHE_TTL_SECONDS
+      value: "3600"
+
     - name: OPC_ID
       value: "1"
     - name: OPC_SERVER_NAME
@@ -195,7 +124,6 @@ env:
     - name: OPC_URL
       value: "opc.tcp://sientia-opc-simulator-opc.sientia.svc.cluster.local:4840"
 
-
     - name: LOG_LEVEL
       value: "DEBUG"
     - name: HTTP_METRICS_PORT
@@ -236,55 +164,49 @@ env:
 
     # Temporal worker tuning for PredictionsBatch.
     # IMPORTANT: prefix must be PREDICTIONSBATCH_ (from class name PredictionsBatch).
-  # Keep workflow-task concurrency moderate to reduce task completion races under load.
     - name: PREDICTIONSBATCH_MAX_CONCURRENT_WORKFLOW_TASKS
       value: "20"
-  # Allow higher activity parallelism because most activities are I/O-bound, but keep headroom.
     - name: PREDICTIONSBATCH_MAX_CONCURRENT_ACTIVITIES
       value: "60"
-  # Keep local activities controlled so they do not monopolize the event loop.
+    - name: PREDICTIONSBATCH_ACTIVITY_EXECUTOR_MAX_WORKERS
+      value: "10"
     - name: PREDICTIONSBATCH_MAX_CONCURRENT_LOCAL_ACTIVITIES
       value: "20"
-  # Cache enough workflows for reuse without excessive memory growth.
     - name: PREDICTIONSBATCH_MAX_CACHED_WORKFLOWS
       value: "200"
-  # Start with one workflow poller to avoid burst contention at startup.
     - name: PREDICTIONSBATCH_WORKFLOW_POLLER_BEHAVIOUR_MINIMUM
       value: "3"
-  # Small initial poller count warms up gradually instead of spiking task fetches.
     - name: PREDICTIONSBATCH_WORKFLOW_POLLER_BEHAVIOUR_INITIAL
       value: "5"
-  # Cap workflow pollers to limit scheduling pressure and avoid over-polling.
     - name: PREDICTIONSBATCH_WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM
       value: "15"
-  # Keep at least two activity pollers so activity queues do not starve during spikes.
     - name: PREDICTIONSBATCH_ACTIVITY_POLLER_BEHAVIOUR_MINIMUM
       value: "3"
-  # Moderate initial activity pollers for faster ramp-up with controlled pressure.
     - name: PREDICTIONSBATCH_ACTIVITY_POLLER_BEHAVIOUR_INITIAL
       value: "10"
-  # Limit max activity pollers to preserve CPU for workflow-task completion.
     - name: PREDICTIONSBATCH_ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM
       value: "30"
 
     - name: MINIMALRETRAIN_MAX_CONCURRENT_ACTIVITIES
-    value: "1"
+      value: "5"
+    - name: MINIMALRETRAIN_ACTIVITY_EXECUTOR_MAX_WORKERS
+      value: "5"
     - name: MINIMALRETRAIN_MAX_CONCURRENT_LOCAL_ACTIVITIES
-    value: "1"
+      value: "5"
     - name: MINIMALRETRAIN_MAX_CACHED_WORKFLOWS
-    value: "1"
+      value: "5"
     - name: MINIMALRETRAIN_WORKFLOW_POLLER_BEHAVIOUR_MINIMUM
-    value: "1"
+      value: "5"
     - name: MINIMALRETRAIN_WORKFLOW_POLLER_BEHAVIOUR_INITIAL
-    value: "1"
+      value: "5"
     - name: MINIMALRETRAIN_WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM
-    value: "1"
+      value: "5"
     - name: MINIMALRETRAIN_ACTIVITY_POLLER_BEHAVIOUR_MINIMUM
-    value: "1"
+      value: "5"
     - name: MINIMALRETRAIN_ACTIVITY_POLLER_BEHAVIOUR_INITIAL
-    value: "1"
+      value: "5"
     - name: MINIMALRETRAIN_ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM
-    value: "1"
+      value: "5"
 
     - name: PI_WEB_API_BASE_URL
       value: "https://pivision.votorantimcimentos.com/piwebapi"
@@ -296,8 +218,88 @@ env:
           name: pi-web-api-auth-token
           key: token
 
-  - name: PYPI_SERVER
+    # Thread-pool size for non-runtime workers that also use prepare_worker.
-    value: "http://library-distribution-server.library.svc.cluster.local:5000"
+    - name: SIMPLEMETRICS_ACTIVITY_EXECUTOR_MAX_WORKERS
+      value: "20"
+    - name: DRIFT_ACTIVITY_EXECUTOR_MAX_WORKERS
+      value: "20"
+
+# -----------------------------------------------------------------------------
+# Runtimes configuration
+# -----------------------------------------------------------------------------
+# IMPORTANT:
+# - The runtime name is used by the worker bootstrap to resolve plugins and task queues.
+# - Keep runtime names in sync with the plugin-store runtime names.
+runtimes:
+  basic:
+    replicas: 1
+  legacy:
+    replicas: 1
+    env:
+      - name: "GITHUB_BRANCH"
+        value: "main"
+      - name: "MLFLOW_HOST"
+        value: "http://sientia-tracker-mlflow-tracking.sientia-tracker.svc.cluster.local"
+      - name: "MLFLOW_PORT"
+        value: "80"
+
+# -----------------------------------------------------------------------------
+# Chart-level configuration (applies to all runtimes)
+# -----------------------------------------------------------------------------
+imagePullSecrets:
+  - name: docker-hub-secret
+
+nameOverride: *projectName
+fullnameOverride: *projectName
+
+serviceAccount:
+  create: true
+  automount: true
+  annotations: {}
+  name: *projectName
+
+podAnnotations: {}
+podLabels: {}
+
+podSecurityContext: {}
+securityContext: {}
+
+volumes: []
+volumeMounts: []
+
+nodeSelector: {}
+tolerations: []
+affinity: {}
+
+services:
+  sdk-metrics:
+    enabled: true
+    type: ClusterIP
+    port: 9091
+    targetPort: 9091
+    name: sdk-metrics
+  metrics:
+    enabled: true
+    type: ClusterIP
+    port: 9090
+    targetPort: 9090
+    name: metrics
+
+# Configuração do ServiceMonitor para o Prometheus Operator
+# ref: https://github.com/prometheus-operator/prometheus-operator
+serviceMonitor:
+  enabled: true
+  endpoints:
+    - port: metrics
+      path: /metrics
+      interval: 30s
+      relabelings: []
+    - port: sdk-metrics
+      path: /metrics
+      interval: 30s
+      relabelings: []
+  additionalLabels:
+    release: kube-prometheus-stack
 
 ssh:
   enabled: true
@@ -305,11 +307,13 @@ ssh:
   sshPath: /mnt/.ssh
   knownHostsPath: /mnt/known_hosts
 
-# kubectl create secret docker-registry docker-hub-secret --namespace sientia --docker-server=http://aignosi.azurecr.io --docker-username=aignosi --docker-password=5I5zpQ6sRaHqX1hD3dr+2mo647yO3FRc359/wu6gsP+ACRDRz5mp
+# kubectl create secret docker-registry docker-hub-secret --namespace sientia --docker-server=http://aignosi.azurecr.io --docker-username=aignosi --docker-password=<pwd>
-
+#
-# helm upgrade --install sientia-laborious-worker sientia/sientia-module -n sientia --create-namespace -f ./values.yaml --version 0.6.0
+# helm upgrade --install sientia-laborious-worker /home/grezewave/Documents/projects/sientia/sientia-core-applications/sientia-module -n sientia --create-namespace -f ./values.yaml
 
 # kubectl create secret generic git-ssh-key-sientia-laborious-worker \
 #   --namespace sientia \
 #   --from-file=ssh-privatekey=git_key \
 #   --type=kubernetes.io/ssh-auth
+
+# helm upgrade --install sientia-laborious-worker /home/grezewave/Documents/projects/sientia/sientia-core-applications/sientia-module -n sientia --create-namespace -f ./values.yaml