SIENTIAPDE-1646

SIENTIAPDE-1646 Merge origin/main into release/SIENTIAPDE-1646

Bring main up to date while keeping this branch's sync execution, PluginStore
runtime and sientia_model (SientiaMLflowRepository/SientiaModel) structure.

Resolution decisions:
- OPC: keep the sync asyncua.sync implementation; port main's server_uri
  validation, e2e race fix (lock held until released) and the new unit tests
  converted to sync.
- Model import workflow (.sientia bundles), joblib/transform flavors and the
  local MLFlowRepository (model_repository.py) are dropped, with their tests,
  docs, metrics, env vars and connectors config.
- Worker: drift/simple_metrics stay pinned to the 'core' runtime; port main's
  HOSTNAME pod id, logger-based prometheus startup and sys.exit after finally.
- values.yaml removed, following main (SIENTIAPDE-1945).
- Deps: keep sientia_model; take main's fsspec/s3fs/botocore/boto3 pins and
  sientia_do>=1.12.2; drop import-only deps (pynacl, cryptography, pymongo,
  python-dotenv, pycaret, joblib, gcsfs).
- e2e: keep sientia_data schema and scenario input files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

e2e/opc_test_server.py

@@ -147,8 +147,11 @@ class OpcE2ETestServer:
         value = await self._confidence_node.read_value()
         return float(value)
 
-    async def _pre_write_callback(self, _event, _service) -> None:
-        if self._session_bad_on_write:
+    async def _pre_write_callback(self, event, _service) -> None:
+        # Only client writes are faulted. asyncua's own clock task writes CurrentTime
+        # through this same PreWrite dispatch, and raising there kills its time_task;
+        # the stored exception then resurfaces as a teardown failure in server.stop().
+        if self._session_bad_on_write and getattr(event, 'is_external', False):
             raise ServiceError(ua.StatusCodes.BadSessionIdInvalid)
 
 

e2e/scenarios.md

@@ -7,10 +7,51 @@ It is a functional reference of scenario behavior, inputs, and expected outcomes
 
 ## Execution Context
 
+### One-time setup: disable the testcontainers reaper
+
+Do this once per machine, before running anything under `e2e/`:
+
+```bash
+echo "ryuk.disabled=true" >> ~/.testcontainers.properties
+```
+
+**Why.** Before starting any test container, testcontainers starts a reaper container ("Ryuk") whose
+job is to clean up containers if the test process dies without running its teardown. Ryuk bind-mounts
+the Docker socket. Under **Docker Desktop on Linux** the socket lives at
+`~/.docker/desktop/docker.sock`, which Docker Desktop's File Sharing does not expose, so the mount is
+denied and the reaper cannot start:
+
+```
+docker.errors.APIError: 500 Server Error ... /containers/<id>/start:
+Mounts denied: The path /socket_mnt/home/<user>/.docker/desktop/docker.sock
+is not shared from the host and is not known to Docker.
+```
+
+Every test then errors **at setup** — `ERROR`, not `FAILED`, and never inside an assertion. If you see
+that, this is why.
+
+**What you give up.** Nothing in a normal run: every fixture in `conftest.py` stops its own containers
+in teardown. Ryuk only covers the case where the process is killed without teardown (`kill -9`, a
+crash, a hard stop from the IDE). After one of those, clean up by hand:
+
+```bash
+docker ps -a --filter "label=org.testcontainers=true" -q | xargs -r docker rm -f
+```
+
+*Alternative, if you want to keep the reaper:* point Docker at the classic socket with
+`export DOCKER_HOST=unix:///var/run/docker.sock`. That only works if a native Docker Engine is
+running alongside Docker Desktop; if the Desktop daemon is the only one, disabling Ryuk is the way.
+
+
 - Tests run under `e2e/` and are marked with `@pytest.mark.integration`.
 - PostgreSQL and MinIO are provisioned with testcontainers.
 - `test_minio_offload.py` uses real MinIO I/O; other scenario suites may use stubs/mocks for optional outputs.
 - Real OPC UA scenarios use `@pytest.mark.opc` and an in-process asyncua server (`e2e/test_opc_real_server.py`).
+- **Nothing under `e2e/` is covered by `validate`**: that script lints `laborious/` and `tests/`
+  only, and its test step is `pytest tests/` (`testpaths = ["tests"]`). Lint these files by hand
+  (`ruff format --check e2e/ && ruff check e2e/`), and remember that `addopts` carries
+  `--strict-markers`: a test without `@pytest.mark.integration` is not collected, and a run that
+  collects nothing still exits 0. Check the collected count, not the exit code.
 
 ### Local validation
 

e2e/test_opc_real_server.py

@@ -7,13 +7,17 @@ Mock-based OPC tests remain in test_predictions_batch_format_export.py.
 
 import asyncio
 import threading
-import time
 
 import pytest
 from temporalio.testing import WorkflowEnvironment
 from temporalio.worker import Worker
 
-from e2e.helpers import assert_prediction, insert_sample_data, make_workflow_id, start_and_await_workflow
+from e2e.helpers import (
+    assert_prediction,
+    insert_sample_data,
+    make_workflow_id,
+    start_and_await_workflow,
+)
 from e2e.opc_test_server import UNKNOWN_NODE_ID, OpcE2ETestServer, build_opc_output_config
 from e2e.test_predictions_batch_format_export import get_base_input_data
 from laborious.activities.activities import Activities
@@ -22,16 +26,24 @@ from laborious.utils.repository.opc_repository import OpcRepository
 from laborious.workflows.predictions_batch import PredictionsBatch
 
 
-def _slow_reconnect_under_lock(repo: OpcRepository, hold_seconds: float = 0.75) -> None:
+def _reconnect_under_held_lock(
+    repo: OpcRepository, acquired: threading.Event, release: threading.Event
+) -> None:
     """
-    Hold the connection lock briefly so concurrent writes see reconnect_in_progress.
+    Hold the connection lock until released, then reconnect.
+
+    The lock must stay held for the whole workflow, and how long the workflow takes to
+    reach the OPC write is not bounded — a fixed sleep here lets `_reconnect_locked`
+    finish first, which sets `_session_ready` again and the write then succeeds.
 
     Args:
         repo (OpcRepository): Connected repository.
-        hold_seconds (float): Time to keep the lock before reconnecting.
+        acquired (threading.Event): Set once the lock is held, before any write can race.
+        release (threading.Event): Waited on while holding the lock; set by the caller.
     """
     with repo._connection_lock:
-        time.sleep(hold_seconds)
+        acquired.set()
+        release.wait()
         repo._reconnect_locked()
 
 
@@ -172,12 +184,15 @@ async def test_scenario_3_2_5_opc_write_blocked_during_reconnect_real_server(
 
     repo = test_activities_real_opc.opc_repository['1']
     repo._session_ready.clear()
+    acquired = threading.Event()
+    release = threading.Event()
     reconnect_thread = threading.Thread(
-        target=_slow_reconnect_under_lock,
-        args=(repo,),
+        target=_reconnect_under_held_lock,
+        args=(repo, acquired, release),
         daemon=True,
     )
     reconnect_thread.start()
+    acquired.wait(timeout=5.0)
 
     input_data = get_base_input_data(model_id)
     input_data['opc_output_config'] = build_opc_output_config(opc_e2e_server.node_ids)
@@ -191,6 +206,7 @@ async def test_scenario_3_2_5_opc_write_blocked_during_reconnect_real_server(
             make_workflow_id('test-opc-real-reconnect-block'),
         )
     finally:
+        release.set()
         reconnect_thread.join(timeout=5.0)
 
     assert_prediction(