sientia-dataops-laborious-legacy/e2e/scenarios.md

# Test Scenarios for Predictions Batch Workflow

This document describes all possible test scenarios for the `predictions_batch` workflow and its child workflows `prediction_process` and `format_and_export_prediction`.

## Running automated E2E tests (`e2e/`)

### One-time setup: disable the testcontainers reaper

Do this once per machine, before running anything under `e2e/`:

```bash
echo "ryuk.disabled=true" >> ~/.testcontainers.properties
```

**Why.** Before starting any test container, testcontainers starts a reaper container ("Ryuk") whose
job is to clean up containers if the test process dies without running its teardown. Ryuk bind-mounts
the Docker socket. Under **Docker Desktop on Linux** the socket lives at
`~/.docker/desktop/docker.sock`, which Docker Desktop's File Sharing does not expose, so the mount is
denied and the reaper cannot start:

```
docker.errors.APIError: 500 Server Error ... /containers/<id>/start:
Mounts denied: The path /socket_mnt/home/<user>/.docker/desktop/docker.sock
is not shared from the host and is not known to Docker.
```

Every test then errors **at setup** — `ERROR`, not `FAILED`, and never inside an assertion. If you see
that, this is why.

**What you give up.** Nothing in a normal run: every fixture in `conftest.py` and
`import_harness.py` stops its own containers in teardown. Ryuk only covers the case where the process
is killed without teardown (`kill -9`, a crash, a hard stop from the IDE). After one of those, clean
up by hand:

```bash
docker ps -a --filter "label=org.testcontainers=true" -q | xargs -r docker rm -f
```

The setting is read by `testcontainers/core/config.py`, which accepts either `ryuk.disabled` in
`~/.testcontainers.properties` or the `TESTCONTAINERS_RYUK_DISABLED` environment variable. The
properties file is preferred here because it is per-machine and applies to every project and every
shell, so nobody has to remember a prefix on the command line.

*Alternative, if you want to keep the reaper:* point Docker at the classic socket with
`export DOCKER_HOST=unix:///var/run/docker.sock`. That only works if a native Docker Engine is
running alongside Docker Desktop; if the Desktop daemon is the only one, disabling Ryuk is the way.

### Requirements and invocation

- **Runtime**: Docker (or a Docker-compatible daemon) must be available so [testcontainers](https://testcontainers.com/) can start **PostgreSQL** and **MinIO** containers.
- **Dependencies**: install dev requirements (includes `testcontainers[postgres,minio,mongodb]`).
- **Invocation**: run only integration-marked tests, for example: `pytest e2e/ -m integration`.
- **MinIO tests**: `e2e/test_minio_offload.py` exercises real S3 uploads; other E2E modules continue to mock MinIO on the worker used by most scenarios.
- **OPC tests (real server)**: `e2e/test_opc_real_server.py` uses an in-process **asyncua** server and real `OpcRepository` (`test_activities_real_opc`). Scenarios 3.1.2, 3.2.2, 3.2.4, and 3.2.5 are covered there. Other E2E modules keep the OPC mock.
- Run only OPC real-server tests: `pytest e2e/test_opc_real_server.py -m "integration and opc"`.
- **Model import tests**: the `import_model` suite is opt-in and has its own harness, containers and
  fidelity contract — see [§4.0](#40-running-the-import-e2e-tests). Run it with
  `pytest e2e -m import_e2e`.
- **joblib flavor tests**: `e2e/test_predictions_batch_joblib_flavor.py` covers reading a model
  through the explicit `joblib` flavor (predict and transform), and
  `e2e/test_retrain_joblib_flavor.py` covers writing one back on a retrain — see
  [§5](#5-joblib-flavor-scenarios). Neither `joblib.dump` nor `joblib.load` is mocked in either
  module: the artifact fixture holds a real `joblib.dump(..., compress=3)` pickle whose class ships
  inside the artifact's `code/` directory, so only the flavor's own `sys.path` handling makes it
  deserialize. Run them with
  `pytest e2e/test_predictions_batch_joblib_flavor.py e2e/test_retrain_joblib_flavor.py -m integration`.
- **Nothing under `e2e/` is covered by `validate`**: that script lints `laborious/` and `tests/`
  only, and its test step is `pytest tests/` (`testpaths = ["tests"]`). Lint these files by hand
  (`ruff format --check e2e/ && ruff check e2e/`), and remember that `addopts` carries
  `--strict-markers`: a test without `@pytest.mark.integration` is not collected, and a run that
  collects nothing still exits 0. Check the collected count, not the exit code.

## Workflow Overview

The `predictions_batch` workflow:
1. Loads data using a custom SQL query
2. Prepares prediction configuration
3. Delegates to `prediction_process` child workflow which:
   - Retrieves last timestamp for incremental processing
   - Applies input data quality gates
   - Executes MLFlow transform operation
   - Validates transform response
   - Executes MLFlow predict operation
   - Validates predict response
   - Delegates to `format_and_export_prediction` child workflow
4. The `format_and_export_prediction` workflow:
   - Formats prediction data (normal or default)
   - Exports to PI Web API (optional)
   - Exports to OPC server (optional)
   - Exports to PostgreSQL
   - Writes metrics

---

## 1. Predictions Batch - Main Workflow Scenarios

### 1.1 Success Scenarios

#### Scenario 1.1.1: Happy Path - Complete Success
**Description**: Workflow completes successfully with valid SQL query and all activities succeed

**Input**:
- Valid `schedule_name`, `model_name`, `model_id`
- Valid `query` returning non-empty DataFrame
- Valid `schema`, `table_name`, `transform_table_name`
- Optional `datetime_columns` for timestamp parsing
- Optional `input_filters`, `mlflow_transform_filters`, `mlflow_predict_filters`
- Optional `path_priority`, `opc_output_config`, `pi_web_api_output_config`

**Expected Behavior**:
- `load_custom_query` returns DataFrame with data
- Workflow prepares prediction input with all configurations
- `prediction_process` child workflow executes successfully
- All gates pass with no issues
- Transform and predict operations succeed
- Data exported to PostgreSQL
- Metrics written

**Assertions**:
- SQL query executed once
- `prediction_process` workflow called with correct parameters
- Data exists in PostgreSQL (predictions table)
- Metrics recorded
- No errors raised

---

### 1.2 Error Scenarios

#### Scenario 1.2.1: SQL Query Execution Error
**Description**: SQL query fails due to syntax error or connection issue

**Input**:
- Invalid SQL query (syntax error)
- Or database connection unavailable

**Expected Behavior**:
- `load_custom_query` raises exception (caught by Temporal retry policy)
- Notification sent with SQL error details
- After retries, activity may return empty data or workflow may fail
- If empty data returned, workflow completes with early exit via input gate

**Assertions**:
- Error notification sent
- Workflow completes (either fails or exits early)
- No data in predictions table

---

#### Scenario 1.2.2: Missing Required Parameters
**Description**: Essential parameters missing from input

**Input**:
- Missing `query` or `model_id` or `schema` or `table_name`

**Expected Behavior**:
- Workflow or activity raises KeyError or validation error
- Workflow fails immediately

**Assertions**:
- Workflow fails with parameter error
- Error notification sent
- No child workflow called

---

#### Scenario 1.2.3: Invalid Datetime Column Specification
**Description**: Datetime column specified doesn't exist in query results

**Input**:
- `datetime_columns: ['nonexistent_column']`
- Query results don't have this column

**Expected Behavior**:
- `load_custom_query` may raise KeyError or warning
- Depending on implementation, workflow may fail or continue
- Error notification sent

**Assertions**:
- Error raised or warning logged
- Workflow behavior depends on error handling policy

---

## 2. Prediction Process - Child Workflow Scenarios

### 2.1 Input gate Early Exit Scenarios

#### Scenario 2.1.1: Input Gate Triggers CONTINUE
**Description**: Input gate determines data should use previous prediction

**Input**:
- Data that should continue with input data as prediction
- `input_filters` configured with `POLICY: 'CONTINUE'`
- `path_priority` includes CONTINUE

**Expected Behavior**:
- `input_gate` returns `path_flag='CONTINUE'`
- `path_flag_handler` calls export workflow with input data directly
- MLFlow transform and predict skipped
- Data exported as-is

**Assertions**:
- `input_gate` called
- MLFlow operations NOT called
- Export workflow called with original data
- Workflow completes


#### Scenario 2.1.2: Input Gate Triggers STOP
**Description**: Input data quality gate fails with STOP policy

**Input**:
- Data with EMPTY_DATA or other critical issues
- `input_filters` configured with `POLICY: 'STOP'`

**Expected Behavior**:
- `input_gate` returns `path_flag='STOP'`
- `path_flag_handler` detects STOP
- Workflow returns early without calling MLFlow
- No prediction exported

**Assertions**:
- `input_gate` called
- `path_flag_handler` returns True (early exit)
- MLFlow transform NOT called
- Export workflow NOT called
- Workflow completes without error


#### Scenario 2.1.3: Input Gate Triggers REPEAT
**Description**: Input gate determines data should repeat last prediction

**Input**:
- Data with quality issues that require using previous prediction
- `input_filters` configured with `POLICY: 'REPEAT'`
- `path_priority` includes REPEAT

**Expected Behavior**:
- `input_gate` returns `path_flag='REPEAT'`
- `path_flag_handler` calls `repeat_last_prediction` activity
- MLFlow transform and predict skipped
- Last prediction repeated and exported

**Assertions**:
- `input_gate` called
- MLFlow operations NOT called
- `repeat_last_prediction` activity called
- Workflow completes

---

### 2.2 Transform gate Early Exit Scenarios

#### Scenario 2.2.1: Transform Gate Triggers CONTINUE
**Description**: Transform response gate determines data should continue despite issues

**Input**:
- Valid input data
- Transform response has quality issues but policy is CONTINUE
- `mlflow_transform_filters` configured with `POLICY: 'CONTINUE'`
- `path_priority` includes CONTINUE

**Expected Behavior**:
- `request_transform` succeeds
- `mlflow_response_gate` for transform returns `path_flag='CONTINUE'`
- `path_flag_handler` calls export workflow with transform data
- MLFlow predict skipped
- Transform data exported as-is

**Assertions**:
- Transform completed
- `mlflow_response_gate` called for transform
- MLFlow predict NOT called
- Export workflow called with transform data
- Workflow completes

---

#### Scenario 2.2.2: Transform Gate Triggers STOP
**Description**: Transform response validation fails with STOP policy

**Input**:
- Valid input data
- Transform response has critical errors
- `mlflow_transform_filters` configured with `POLICY: 'STOP'`

**Expected Behavior**:
- `request_transform` succeeds but response invalid
- `mlflow_response_gate` for transform returns `path_flag='STOP'`
- Workflow exits without calling predict or export

**Assertions**:
- Transform completed but validation failed
- `mlflow_response_gate` called for transform
- MLFlow predict NOT called
- Export workflow NOT called
- Workflow completes without error

---

#### Scenario 2.2.3: Transform Gate Triggers REPEAT
**Description**: Transform response gate determines data should repeat last prediction

**Input**:
- Valid input data
- Transform response has quality issues that require using previous prediction
- `mlflow_transform_filters` configured with `POLICY: 'REPEAT'`
- `path_priority` includes REPEAT

**Expected Behavior**:
- `request_transform` succeeds but response has issues
- `mlflow_response_gate` for transform returns `path_flag='REPEAT'`
- `path_flag_handler` calls `repeat_last_prediction` activity
- MLFlow predict skipped
- Last prediction repeated and exported

**Assertions**:
- Transform completed but validation triggered REPEAT
- `mlflow_response_gate` called for transform
- MLFlow predict NOT called
- `repeat_last_prediction` activity called
- Workflow completes

---

### 2.3 Predict gate Early Exit Scenarios

#### Scenario 2.3.1: Predict Gate Triggers CONTINUE
**Description**: Predict response gate determines data should continue despite issues

**Input**:
- Valid input and transform data
- Predict response has quality issues but policy is CONTINUE
- `mlflow_predict_filters` configured with `POLICY: 'CONTINUE'`
- `path_priority` includes CONTINUE

**Expected Behavior**:
- `request_predict` succeeds
- `mlflow_response_gate` for predict returns `path_flag='CONTINUE'`
- `path_flag_handler` calls export workflow with predict data
- Prediction exported despite quality issues

**Assertions**:
- Transform and predict completed
- `mlflow_response_gate` called for predict
- Export workflow called with predict data
- Workflow completes

---

#### Scenario 2.3.2: Predict Gate Triggers STOP
**Description**: Prediction validation fails with STOP policy

**Input**:
- Valid input and transform
- Predict response has critical errors
- `mlflow_predict_filters` configured with `POLICY: 'STOP'`

**Expected Behavior**:
- `request_predict` succeeds but response invalid
- `mlflow_response_gate` for predict returns `path_flag='STOP'`
- Workflow exits without export

**Assertions**:
- Transform completed
- Predict completed but validation failed
- Export workflow NOT called
- Workflow completes without error

---

#### Scenario 2.3.3: Predict Gate Triggers REPEAT
**Description**: Predict response gate determines data should repeat last prediction

**Input**:
- Valid input and transform data
- Predict response has quality issues that require using previous prediction
- `mlflow_predict_filters` configured with `POLICY: 'REPEAT'`
- `path_priority` includes REPEAT

**Expected Behavior**:
- `request_predict` succeeds but response has issues
- `mlflow_response_gate` for predict returns `path_flag='REPEAT'`
- `path_flag_handler` calls `repeat_last_prediction` activity
- Last prediction repeated and exported

**Assertions**:
- Transform and predict completed but validation triggered REPEAT
- `mlflow_response_gate` called for predict
- `repeat_last_prediction` activity called
- Export workflow NOT called with current prediction
- Workflow completes

---

## 3. Format and Export Prediction - Child Workflow Scenarios

### 3.1 Success Scenarios

#### Scenario 3.1.1: Default Prediction Export
**Description**: Error prediction path creates default prediction

**Input**:
- `path_flag: 'ERROR'` or other non-None value (not STOP/CONTINUE/REPEAT)
- `comment` provided with error details

**Expected Behavior**:
- `format_default_prediction` called instead of `format_prediction`
- Default prediction created with error metadata
- Exported to PostgreSQL only
- Transformed data NOT processed
- Metrics written

**Assertions**:
- `format_default_prediction` called
- `format_prediction` NOT called
- `format_transformed_data` NOT called
- One PostgreSQL export only
- Default values in prediction data
- Comment included

---

#### Scenario 3.1.2: Export with OPC only
**Description**: Export to PostgreSQL and OPC server only (no PI Web API)

**Input**:
- `path_flag: None`
- `opc_output_config` configured with valid OPC settings
- `pi_web_api_output_config: None` or `{}`

**Expected Behavior**:
- Normal formatting
- PostgreSQL export executed
- OPC export executed
- PI Web API activity skipped
- Metrics written with OPC metrics

**Assertions**:
- PI Web API activity NOT called
- OPC activity called
- PostgreSQL export called
- Metrics written with `opc_metrics` populated

---

#### Scenario 3.1.3: Export with PI Web API only
**Description**: Export to PostgreSQL and PI Web API only (no OPC)

**Input**:
- `path_flag: None`
- `pi_web_api_output_config` configured with valid PI Web API settings
- `opc_output_config: None` or `{}`

**Expected Behavior**:
- Normal formatting
- PostgreSQL export executed
- PI Web API export executed
- OPC activity skipped
- Metrics written without OPC metrics

**Assertions**:
- OPC activity NOT called
- PI Web API activity called
- PostgreSQL export called
- Metrics written with empty `opc_metrics`

---

#### Scenario 3.1.4: Export Without Optional Outputs
**Description**: Export only to PostgreSQL (no OPC or PI Web API)

**Input**:
- `path_flag: None`
- `opc_output_config: None` or `{}`
- `pi_web_api_output_config: None` or `{}`

**Expected Behavior**:
- Normal formatting
- Only PostgreSQL export executed
- OPC and PI Web API activities skipped
- Metrics written without OPC metrics

**Assertions**:
- PI Web API activity NOT called
- OPC activity NOT called
- PostgreSQL export called
- Metrics written with empty `opc_metrics`

---

#### Scenario 3.1.5: Export Without Transformed Data
**Description**: Only prediction exported, no transform table

**Input**:
- `path_flag: None`
- `transformed_data: None` or `save_transform: False`
- `opc_output_config: None` or `{}`
- `pi_web_api_output_config: None` or `{}`

**Expected Behavior**:
- Only prediction formatted and exported
- Transform export skipped
- Single PostgreSQL write

**Assertions**:
- `format_transformed_data` NOT called
- One PostgreSQL export
- Transform table remains empty

---

### 3.2 Error Scenarios

These paths do **not** rely on Temporal activity retries for export failures: the write activities run once, errors are handled inside the activity, and the **workflow completes successfully** with degraded metadata on the persisted prediction (`prediction_confidence` and `comments`).

#### Scenario 3.2.1: PI Web API Write Error
**Description**: PI Web API export fails

**Input**:
- Valid prediction
- PI Web API service unavailable or invalid config

**Expected Behavior**:
- `write_pi_web_api_data` surfaces the failure (exception handled in the activity layer)
- Notification may be sent
- Workflow **completes** (does not fail)
- Prediction row is still written to PostgreSQL with error confidence **13** and a comment describing the PI error
- Subsequent steps (e.g. OPC, Postgres) still run per workflow order with the updated prediction payload

**Assertions**:
- PI Web API error notification sent (when applicable)
- Workflow completes
- PostgreSQL contains the prediction with `prediction_confidence` 13 and expected `comments`

---

#### Scenario 3.2.2: OPC Write Error
**Description**: OPC server write fails

**Input**:
- Valid prediction
- OPC server unavailable or invalid configuration

**Expected Behavior**:
- `write_opc_data` reports failure without aborting the workflow
- Notification may be sent
- Workflow **completes** (does not fail)
- Prediction row is written to PostgreSQL with OPC error confidence **12** and a comment indicating OPC write issues

**Assertions**:
- OPC error notification sent (when applicable)
- Workflow completes
- PostgreSQL contains the prediction with `prediction_confidence` 12 and expected `comments`

---

#### Scenario 3.2.4: OPC Session / Channel Bad* (Tier-1)
**Description**: OPC write fails with a Tier-1 session or channel status (e.g. `BadSessionIdInvalid`) while transport may still appear open on the client

**Input**:
- Valid prediction and OPC output config
- Mock or server returning Tier-1 `UaStatusCodeError` on write (no write retry in the same activity)

**Expected Behavior**:
- `write_opc_data` fails forward for affected tags; background reconnect may be scheduled if `OPC_RECONNECTION_INTERVAL` allows
- Workflow **completes**
- PostgreSQL row uses **`prediction_confidence` 14** and comment prefix `OPC UA session/channel error:` (including OPC status name)
- `opc_write_attempts_total` records `result=BadSessionIdInvalid` (or matching status); no second write attempt in the same activity

**Assertions**:
- Workflow completes
- `prediction_confidence = 14`
- `comments` matches `OPC UA session/channel error:%`
- Generic OPC error confidence **12** is not used for this case

**Reference**: [docs/opc-communication.md](../docs/opc-communication.md), plan `.cursor/plans/opc_bad_reconnect_ac4c6045.plan.md`

---

#### Scenario 3.2.5: OPC Write Blocked During Reconnect
**Description**: A write is attempted while the repository is reconnecting (session not ready)

**Input**:
- Valid prediction
- Simulated slow reconnect (e.g. delayed `connect`) or concurrent writes where the first triggers reconnect

**Expected Behavior**:
- Second write (or parallel write) is rejected **immediately** when reconnect is in progress or `_session_ready` is cleared — **without** calling `write_value`
- No wait/sleep on the write path; no duplicate `connect` from parallel writers (connection lock)
- `prediction_confidence = 14`, `comments = OPC UA reconnect in progress` (distinguish from Tier-1 `Bad*` via comment prefix in SQL)

**Assertions**:
- At most one reconnect sequence (`disconnect` + `connect`) for the overlapping window
- No write retry after failure
- Tests in `test_opc_repository` (unit) and optional e2e in `test_predictions_batch_format_export.py`

---

#### Scenario 3.2.3: PI Web API Partial Write Error
**Description**: Two prediction tags attempt to be written to PI Web API, but only one succeeds

**Input**:
- Valid prediction
- Two prediction tags configured
- PI Web API returns partial success (one tag succeeds, one fails)

**Expected Behavior**:
- `write_pi_web_api_data` processes response
- `process_pi_web_api_response` detects partial failure
- Error confidence set (13)
- Notification sent for failed tag
- Workflow completes with error confidence (single activity attempt; no retry loop)

**Assertions**:
- One tag written successfully
- One tag failed
- Error confidence set in prediction
- Error notification sent
- Workflow completes

---
## 4. Model Import Workflow Scenarios (`import_model`)

The `import_model` workflow turns an encrypted `.sientia` bundle already in MinIO into a registered
MLflow model plus one MongoDB `models` document — or into a durable error on the row the BFF
inserted. It shares no code path with the predictions workflows above and has its own harness.

### 4.0 Running the import E2E tests

- **Marker**: every test below carries `import_e2e`; the real-clock retry tests also carry `slow`.
  `testpaths` stays `["tests"]`, so none of this is collected by the unit gate.
- **Invocation**: `pytest e2e -m import_e2e`.
- **Runtime**: Docker for three containers (the shared MinIO, a dedicated Postgres for
  `sientia-core-mlops-bff`, MongoDB) **plus** an `mlflow server` subprocess started from this
  repository's venv. Dev requirements install `testcontainers[postgres,minio,mongodb]`.
- **Fidelity**: on the import path nothing is mocked — real MinIO, real Postgres, real MongoDB with a
  unique index, a real MLflow tracking server with a working registry, real Argon2id and real
  `secretstream`. Only `CoreNotificationHandler`'s Mongo client stays mocked, because notifications
  are a side channel this suite counts rather than a dependency it tests.
- **Fixtures**: `e2e/import_harness.py`, registered as a pytest plugin from `e2e/conftest.py`. Every
  fixture is `import_`-prefixed.
- **Prerequisite**: the one-time reaper setup at the top of this document
  ([Running automated E2E tests](#one-time-setup-disable-the-testcontainers-reaper)). Without it every
  test here errors at setup with a Docker mount error and never reaches an assertion.

### 4.1 Success Scenarios

---

#### Scenario 4.1.1: Producer-Emitted Bundle Imports End to End
**Description**: The committed golden `.sientia` file — bytes the producer's own code wrote — is
imported at the producer's real Argon2id cost (256 MiB, 3 passes)

**Input**:
- `tests/fixtures/bundle/golden_model_v3.sientia` uploaded under `imported_models/`
- One `PENDING` / `run_type='IMPORT'` row carrying the object key and the uploader's digest
- The workflow input: `import_run_id`, `bucket`, `object_key`, `expected_digest`, `password_envelope`

**Expected Behavior**:
- The row is claimed, the seven gates pass, the experiment and a new run are created
- The artifact tree is logged with `prediction_model` and `data_model` at the run's root
- A registry version is created and left unstaged; the model document is written inactive
- One terminal write sets `IMPORT_SUCCESS`

**Assertions**:
- Exactly one run named `import-fixture_model-v3`, status `FINISHED`
- `prediction_model` contains `MLmodel` and `model.pkl`
- Bundle parameters logged, origin logged under `source.*`
- Destination version is `1`, independent of the bundle's `model_version` of `3`
- Model document `id` is the seeded maximum plus one; `active` is `false`
- Row: `IMPORT_SUCCESS`, `error_message` null, both names filled

---

#### Scenario 4.1.2: Factory-Built Bundle Imports End to End
**Description**: The same assertions over a synthesised tree with names unique to the test

**Input**: A bundle from `tests/helpers/bundle_factory.py` at the cheap KDF cost

**Expected Behavior**: As 4.1.1

**Assertions**:
- As 4.1.1, plus `model_config` holds exactly `transform_flavor`, `predict_flavor`,
  `retention_minutes` and `target`
- `import.workflow_id` in the row's detail JSON matches the run

---

#### Scenario 4.1.3: The Registry Source Resolves Through the Legacy Loader
**Description**: The `source` the importer writes must be one the *existing* loader can parse

**Input**: A successful import's registered version

**Expected Behavior**: `MLFlowRepository.get_model_run_id` extracts the run id from the source

**Assertions**:
- The source ends in `/prediction_model`
- `get_model_run_id` returns the run the import created
- Both artifact paths are present under that run
- Called with the stage the version is actually in, because an import promotes nothing

---

#### Scenario 4.1.4: Second Import of the Same Model
**Description**: The same model imported twice, with different declared `model_version`s

**Input**: Two bundles declaring `model_version` `2` then `7`, same model and experiment names

**Expected Behavior**: Additive — a new run and a new registry version, nothing promoted, the
existing model document untouched

**Assertions**:
- Two versions (`1`, `2`) of one registered model; destination numbering ignores the bundle's
- Two runs in the same experiment, each logging its own `source.model_version`
- No version in `Production`
- The model document is byte-identical to what the first import wrote

---

#### Scenario 4.1.5: A Model Name Already in the Collection
**Description**: An operator-owned document already carries the bundle's model name

**Input**: A document with `active: True`, `predict_flavor: pyfunc` and a hand-set target

**Expected Behavior**: The import notices the name and leaves the document alone; the import still
succeeds

**Assertions**:
- The collection is unchanged, exactly one document carries that name, no duplicate
- Row: `IMPORT_SUCCESS` — a document that is already there is not a failure

---

### 4.2 Gate Rejection Scenarios

Every case asserts the same five things: terminal status `ORCHESTRATOR_VALIDATION_ERROR`, the
recorded step, the recorded code, `error_message` **equal** to the catalog sentence for that step,
and that no experiment, run, artifact, registry version, model document or extracted file remains.

---

#### Scenario 4.2.1: Object Above the Size Ceiling (gate 1)
**Description**: An object larger than `IMPORT_MAX_OBJECT_BYTES`, rejected from `stat_object` before
its body is fetched

**Input**: A factory bundle against a ceiling narrowed to 1 KiB for this test class

**Expected Behavior**: Rejected at `download`

**Assertions**: The five shared assertions

---

#### Scenario 4.2.2: Malformed Header (gate 2)
**Description**: Five variants — wrong magic, unsupported format version, KDF memory above the
ceiling, KDF passes above the ceiling, a header digest that does not describe the chunk stream

**Expected Behavior**: Each rejected at `download`, sharing gate 1's step

**Assertions**:
- The five shared assertions
- The recorded sentence does **not** mention the password: gates 1 and 2 must not implicate a
  password that is fine

---

#### Scenario 4.2.3: Decryption Failures (gate 3)
**Description**: Three variants — a wrong bundle password inside a valid envelope, a tampered chunk
with the header digest repaired, and a stream whose every chunk is framed `MESSAGE`

**Expected Behavior**: Each rejected at `decryption`

**Assertions**:
- The five shared assertions
- The tampered case proves the file is self-consistent and only the AEAD can tell
- The framing case proves the digest cannot see an incomplete stream

---

#### Scenario 4.2.4: Hostile Archive (gate 4)
**Description**: Four variants — an entry escaping the tree, an absolute entry path, a symbolic
link, and a declared expansion over the ceiling

**Expected Behavior**: Each rejected at `archive_inspection`, from the central directory alone

**Assertions**:
- The five shared assertions
- **Nothing extracted**: the worker work directory is empty, not merely free of provisioning

---

#### Scenario 4.2.5: A Member Extraction Cannot Place (gate 5)
**Description**: An archive declaring the same member name twice — passes gate 4, which checks names
and modes but not duplicates

**Expected Behavior**: Rejected at `extraction` on the second member

**Assertions**: The five shared assertions

---

#### Scenario 4.2.6: Incomplete Structure (gate 6)
**Description**: Five variants — no `metadata.json`, metadata missing `run_id`, parameters missing
`target_variable`, no `prediction_model` directory, and a 60-character experiment name

**Expected Behavior**: Each rejected at `structure_validation`

**Assertions**:
- The five shared assertions
- For the over-long name: the row's `experiment_name` is still the inserted placeholder, so no write
  reached the `VARCHAR(50)` column, and that column is confirmed still 50 wide

---

#### Scenario 4.2.7: Forbidden File Type (gate 7)
**Description**: An `artifacts/payload.exe` member with a suffix outside the allow-list

**Expected Behavior**: Rejected at `content_policy`

**Assertions**: The five shared assertions

---

### 4.3 Digest Enforcement Scenarios

---

#### Scenario 4.3.1: A Digest Computed Over Other Bytes
**Description**: A perfectly valid bundle whose `expected_digest` belongs to a different file

**Expected Behavior**: Fails at `download`, before any header is parsed

**Assertions**:
- Terminal status, step, code and catalog sentence
- The failure names the digest, not the magic or the version
- `run_name` is still null and `experiment_name` still the placeholder — the bundle was never opened

---

#### Scenario 4.3.2: The Object Is Replaced After Upload
**Description**: The same key, different bytes — also a valid bundle, so nothing but the digest can
tell

**Expected Behavior**: Fails at `download`

**Assertions**: As 4.3.1; only a real object store can show this at all

---

#### Scenario 4.3.3: No Digest at All
**Description**: `expected_digest` absent from the input

**Expected Behavior**: Fails at `received`, before anything is downloaded

**Assertions**: The request-invalid code, and the work directory was never created

---

### 4.4 Provisioning Failure Scenarios

Each asserts the platform terminal status `IMPORT_ERROR` — not the validation status, because a
provisioning failure is not the user's to fix — its own step and code, the workflow identifier in
the detail JSON, and `error_message` **equal** to the catalog sentence.

---

#### Scenario 4.4.1: The Tracking Server Is Stopped
**Description**: The `mlflow server` process is genuinely terminated for the duration

**Expected Behavior**: Fails at `experiment_creation`

**Assertions**:
- The shared assertions
- The names are already on the row: the bundle was opened before MLflow was reached
- Nothing in Mongo; nothing provisioned

---

#### Scenario 4.4.2: Artifact Upload Fails
**Description**: `log_artifacts` is patched — the one step where neither hostile input nor a broken
dependency can isolate the failure

**Expected Behavior**: Fails at `artifact_upload`

**Assertions**:
- The shared assertions
- The experiment exists (the previous step succeeded); no version, no document

---

#### Scenario 4.4.3: Registration Fails
**Description**: `create_model_version` is patched. MLflow rejects only an empty registered-model
name, and one process serves both tracking and the registry, so nothing else isolates this step

**Expected Behavior**: Fails at `registration`

**Assertions**:
- The shared assertions
- The run and both artifact paths survive — "the files were stored, the model was not registered"

---

#### Scenario 4.4.4: The Model Document Is Refused
**Description**: A real unique index on `model_config.target` refuses the activity's own insert.
(A duplicate `id` cannot be constructed: the activity derives `max(int(id)) + 1` from the same
collection, so the id it computes can never already exist)

**Expected Behavior**: Fails at `model_document`

**Assertions**:
- The shared assertions
- One registry version exists — registered, and unusable

---

### 4.5 Status Record Scenarios

---

#### Scenario 4.5.1: The Claim Precedes Every Gate
**Description**: The row is observed from inside `stat_object`, the first call gate 1 makes

**Assertions**: Running status and `import.workflow_id` already present at that moment

---

#### Scenario 4.5.2: The Names Are Visible Mid-Flight
**Description**: The row is observed from inside the first call experiment creation makes, during an
import that then succeeds

**Assertions**: `experiment_name` and `run_name` already populated before anything exists in MLflow

---

#### Scenario 4.5.3: The Inserting Side's Columns Are Never Rewritten
**Description**: After a success and after a failure

**Assertions**: `username`, `request_data`, `bucket_name`, `file_name`, `run_type`, `created_at`
unchanged; `status` and `updated_at` did change

---

#### Scenario 4.5.4: Unclaimable Rows
**Description**: Three variants — an unknown id, a `run_type='TRAIN'` row, and a row already held by
another workflow id

**Expected Behavior**: The Temporal run ends failed with the not-claimable error type, and **nothing
is written to any row**

**Assertions**: The error type from the history, the whole table byte-identical, nothing provisioned

---

#### Scenario 4.5.5: The Row Names a Different Object
**Description**: `file_name` disagrees with the input's `object_key`

**Expected Behavior**: Fails at `received` **with the error recorded on the row** — unlike 4.5.4,
this row was successfully claimed

**Assertions**: The received code and sentence on the row; nothing provisioned

---

#### Scenario 4.5.6: Constraint Conformance
**Description**: The real constraints, exercised directly

**Assertions**:
- Every status a normal import writes is accepted on an `IMPORT` row
- All four terminal values are refused across run types, naming
  `chk_experiment_run_terminal_status_run_type`; the shared stage values are accepted on both
- A training row still walks its own vocabulary
- The table has exactly the 13 columns `V1`…`V12` leave (no `ALTER TABLE` by this feature)

---

### 4.6 Absence Scenarios

---

#### Scenario 4.6.1: One Table, One Database
**Description**: A full import bracketed by marker statements on both Postgres servers, read from
`log_statement=all`

**Assertions**:
- Every statement touching a user relation names `experiment_run`
- No `project`, `models`, `import_run`, `model_project` or `model_registry` reference (with quoted
  literals stripped, so the bundle's own `model_project` *value* is not mistaken for a table)
- The predictions database logs **no** statement at all, though the import holds working credentials
  for it
- A rejected import — which writes more status rows than a successful one — behaves the same

---

### 4.7 Cleanup, Retention and History Scenarios

---

#### Scenario 4.7.1: The Worker Filesystem Is Clean
**Description**: After a success and after rejections at two different steps (decryption and
structure validation, which unwind on different code paths)

**Assertions**: No extracted tree, no downloaded object, no temporary archive

---

#### Scenario 4.7.2: The Retention Rule Is Present
**Assertions**: `sientia-import-bundle-expiry` on the import bucket, enabled, filtered on
`imported_models/`, expiring in 7 days — and the uploaded object still there, because the import
never deletes it

---

#### Scenario 4.7.3: The Record Outlives the Object
**Description**: The uploaded object is deleted, as retention will eventually do

**Assertions**: The row still reads `IMPORT_SUCCESS` with its names

---

#### Scenario 4.7.4: No Plaintext Secret in the Event History
**Assertions**:
- The bundle password, the base64 envelope key and the key's hex form appear nowhere in the history
- `password_envelope` **is** in the history, so the negative assertions above cannot pass vacuously

---

### 4.8 Retry Envelope and Missing-`V12` Scenarios

These run on a **real clock** (`WorkflowEnvironment.start_local()`) with the status retry intervals
shortened; the production values stay asserted by the unit suite. All are marked `slow`.

---

#### Scenario 4.8.1: A Database That Comes Back
**Description**: The status database is stopped for two seconds after provisioning and restarted
inside the retry envelope

**Assertions**: The terminal write lands `IMPORT_SUCCESS` with the correct workflow id

---

#### Scenario 4.8.2: A Broken Progress Update
**Description**: A brief early outage that only the progress hints see

**Assertions**: The import completes with the correct verdict and does not spend the terminal
write's envelope on a hint nobody reads

---

#### Scenario 4.8.3: A Database That Stays Down After a Successful Import
**Description**: The database is taken away as soon as the row is claimed and never comes back

**Expected Behavior**: The model is fully provisioned and the workflow **completes** — reporting
failure would tell somebody to re-import a model that is already on the platform

**Assertions**:
- The run returns its result; the model, version and document all exist
- The failure channel carries `IMPORT_STATUS_NOT_RECORDED`, `step=status_write` and
  `verdict_not_recorded=`
- The row is left in the running status

---

#### Scenario 4.8.4: A Failed Import Whose Failure Cannot Be Written Either
**Description**: The import fails at `artifact_upload` and the terminal write is then refused

**Assertions**:
- The reported cause is the artifact-upload failure, not the status write
- The status write reports itself only through the channel
- The row is left running

---

#### Scenario 4.8.5: A Table Without `V12`
**Description**: The evidence for why the migration gates enabling the import worker

**Expected Behavior**: The claim succeeds (`V11` gave the table `run_type`; `PENDING` and the running
status are pre-`V12` values), the model is fully provisioned, and only the terminal value is new — so
the last write of the import is the first thing that fails, non-retryably

**Assertions**:
- The run completes; the model, version and document all exist
- The row is left in the running status, with both names filled
- The channel carries the not-recorded code and the verdict
- Elapsed time shows the refusal was **not** retried
- `pg_constraint` confirms the fixture's `chk_experiment_run_status` really does not allow
  `IMPORT_SUCCESS`, so the test cannot pass vacuously

---

## 5. joblib Flavor Scenarios

`joblib` is an explicit model flavor, declared in `model_config.predict_flavor` /
`model_config.transform_flavor` like any other. It reads the artifact directly — download the
`.pkl`, put the artifact's `code/` on `sys.path`, `joblib.load` — and **never** calls an MLflow
flavor loader. There is no fallback in either direction: a failing loader propagates.

Covered by `e2e/test_predictions_batch_joblib_flavor.py` (read) and
`e2e/test_retrain_joblib_flavor.py` (write). Both use the `joblib_flavor_artifacts` fixture, which
builds one artifact directory per model type holding a real compressed `model.pkl` plus a `code/`
directory carrying the module where the model's class lives. The fixture takes that directory off
`sys.path` again before yielding, so the pickle is **unloadable** until the flavor puts it back —
that is what makes these tests prove module resolution rather than assume it. The fixture also
restores `sys.path` in teardown, because the flavor deliberately never removes what it inserted.

### 5.1 Read Scenarios

---

#### Scenario 5.1.1: Predict Through the joblib Flavor
**Description**: `predict_flavor: 'joblib'` on a normal `predictions_batch` run

**Input**:
- `model_config.predict_flavor = 'joblib'`, `transform_flavor = 'sklearn'`
- `client.download_artifacts` serving the real `prediction_model` artifact

**Expected Behavior**:
- The artifact is downloaded once and deserialized with `joblib.load`
- No `models:/<name>/production` URI is built, so no flavor loader ever sees this model
- The prediction the artifact's own object produces is what lands in Postgres

**Assertions**:
- Exactly one prediction row, `prediction_status='Good'`, value matching the artifact's model
- `download_artifacts` called once, with `'prediction_model'`
- No `mlflow.{sklearn,pyfunc}.load_model` call names this model

---

#### Scenario 5.1.2: Transform Through the joblib Flavor
**Description**: `transform_flavor: 'joblib'`, which had no E2E coverage before

**Input**:
- `model_config.transform_flavor = 'joblib'`, `predict_flavor = 'sklearn'`
- `client.download_artifacts` serving the real `data_model` artifact

**Expected Behavior**:
- `data_model` is downloaded and deserialized; the prediction model stays on its MLflow flavor
- No artifact URI is resolved for the transform, so `mlflow.get_run` is never called

**Assertions**:
- The prediction lands with the mocked prediction model's value
- `download_artifacts` called once, with `'data_model'`
- `mlflow.get_run` not called; no flavor loader was given a `/data_model` URI

---

#### Scenario 5.1.3: An MLflow Flavor Is Unaffected
**Description**: A model declaring `sklearn` behaves exactly as before

**Input**:
- `client.list_artifacts` answering that the run's artifacts carry no `code` directory

**Assertions**:
- The prediction lands with the mocked model's value
- `download_artifacts` NOT called — the joblib path is never entered as a safety net, and with no
  `code` directory to fetch there is nothing else to download either

---

#### Scenario 5.1.4: Only the Artifact's `code` Directory Is Downloaded
**Description**: A model on an MLflow flavor whose artifact **does** carry `code/`. The flavor
loaders read the model URI themselves and fetch what they need, so the `code` directory — the one
thing MLflow will not put on `sys.path` unless the `MLmodel` manifest declares it — is the whole of
what this workflow downloads through the client

**Input**:
- `model_config` on the default `sklearn` flavors
- `client.list_artifacts` answering that both artifacts carry `code`

**Expected Behavior**:
- The `code` directory of each artifact is downloaded before its model is loaded, and goes on
  `sys.path`
- Neither model's weights travel through `client.download_artifacts`

**Assertions**:
- The prediction lands with the mocked model's value
- `download_artifacts` called exactly twice, with `'data_model/code'` and `'prediction_model/code'`

---

### 5.2 Error Scenarios

---

#### Scenario 5.2.1: joblib Flavor Failure Has a Single Cause
**Description**: The downloaded artifact has no top-level `.pkl`

**Expected Behavior**: The workflow stops. The reported cause is the missing pickle and nothing
else — with no fallback there is no second exception chained to a first

**Assertions**:
- The run stops (`assert_stop`)
- The log carries `No .pkl file found in prediction_model artifact`
- The log does **not** carry `Original error`, the marker of a chained cause

---

### 5.3 Write Scenarios

---

#### Scenario 5.3.1: Retraining a joblib Model Round-Trips
**Description**: `retrain_model` on a model whose `predict_flavor` and `transform_flavor` are both
`joblib` — write and read in one test

**Input**:
- Training frame indexed by `DATETIME_FORMAT_WITH_TZ` timestamps, carrying the target column
- MLflow's artifact I/O replaced by a fake store that copies into an inspectable run directory

**Expected Behavior**:
- Both models load through the joblib flavor; no flavor loader runs
- `log_model` writes `prediction_model/model.pkl` (a `joblib.dump(..., compress=3)` pickle) and
  re-logs `prediction_model/code/`, without an `MLmodel` manifest
- Loading the **written** artifact through the joblib flavor deserializes it and predicts, after
  the module the first load imported has been dropped from `sys.modules`

**Assertions**:
- `retrain_model` returns `success: True`
- `download_artifacts` was asked for both `data_model` and `prediction_model`
- `sklearn`/`pyfunc` loaders have zero calls
- The written run has `prediction_model/model.pkl`, `prediction_model/code/<module>.py` and
  `data_model/model.pkl`, and **no** `prediction_model/MLmodel`
- The reloaded model's prediction equals the original's, and the written `code/` is on `sys.path`

---