sientia-dataops-laborious-legacy/e2e/scenarios.md

Test Scenarios for Predictions Batch Workflow

This document describes all possible test scenarios for the predictions_batch workflow and its child workflows prediction_process and format_and_export_prediction.

Running automated E2E tests (e2e/)

One-time setup: disable the testcontainers reaper

Do this once per machine, before running anything under e2e/:

echo "ryuk.disabled=true" >> ~/.testcontainers.properties

Why. Before starting any test container, testcontainers starts a reaper container ("Ryuk") whose job is to clean up containers if the test process dies without running its teardown. Ryuk bind-mounts the Docker socket. Under Docker Desktop on Linux the socket lives at ~/.docker/desktop/docker.sock, which Docker Desktop's File Sharing does not expose, so the mount is denied and the reaper cannot start:

docker.errors.APIError: 500 Server Error ... /containers/<id>/start:
Mounts denied: The path /socket_mnt/home/<user>/.docker/desktop/docker.sock
is not shared from the host and is not known to Docker.

Every test then errors at setup — ERROR, not FAILED, and never inside an assertion. If you see that, this is why.

What you give up. Nothing in a normal run: every fixture in conftest.py and import_harness.py stops its own containers in teardown. Ryuk only covers the case where the process is killed without teardown (kill -9, a crash, a hard stop from the IDE). After one of those, clean up by hand:

docker ps -a --filter "label=org.testcontainers=true" -q | xargs -r docker rm -f

The setting is read by testcontainers/core/config.py, which accepts either ryuk.disabled in ~/.testcontainers.properties or the TESTCONTAINERS_RYUK_DISABLED environment variable. The properties file is preferred here because it is per-machine and applies to every project and every shell, so nobody has to remember a prefix on the command line.

Alternative, if you want to keep the reaper: point Docker at the classic socket with export DOCKER_HOST=unix:///var/run/docker.sock. That only works if a native Docker Engine is running alongside Docker Desktop; if the Desktop daemon is the only one, disabling Ryuk is the way.

Requirements and invocation

• Runtime: Docker (or a Docker-compatible daemon) must be available so testcontainers can start PostgreSQL and MinIO containers.
• Dependencies: install dev requirements (includes testcontainers[postgres,minio,mongodb]).
• Invocation: run only integration-marked tests, for example: pytest e2e/ -m integration.
• MinIO tests: e2e/test_minio_offload.py exercises real S3 uploads; other E2E modules continue to mock MinIO on the worker used by most scenarios.
• OPC tests (real server): e2e/test_opc_real_server.py uses an in-process asyncua server and real OpcRepository (test_activities_real_opc). Scenarios 3.1.2, 3.2.2, 3.2.4, and 3.2.5 are covered there. Other E2E modules keep the OPC mock.
• Run only OPC real-server tests: pytest e2e/test_opc_real_server.py -m "integration and opc".
• Model import tests: the import_model suite is opt-in and has its own harness, containers and fidelity contract — see §4.0. Run it with pytest e2e -m import_e2e.
• joblib flavor tests: e2e/test_predictions_batch_joblib_flavor.py covers reading a model through the explicit joblib flavor (predict and transform), and e2e/test_retrain_joblib_flavor.py covers writing one back on a retrain — see §5. Neither joblib.dump nor joblib.load is mocked in either module: the artifact fixture holds a real joblib.dump(..., compress=3) pickle whose class ships inside the artifact's code/ directory, so only the flavor's own sys.path handling makes it deserialize. Run them with pytest e2e/test_predictions_batch_joblib_flavor.py e2e/test_retrain_joblib_flavor.py -m integration.
• Nothing under e2e/ is covered by validate: that script lints laborious/ and tests/ only, and its test step is pytest tests/ (testpaths = ["tests"]). Lint these files by hand (ruff format --check e2e/ && ruff check e2e/), and remember that addopts carries --strict-markers: a test without @pytest.mark.integration is not collected, and a run that collects nothing still exits 0. Check the collected count, not the exit code.

Workflow Overview

The predictions_batch workflow:

1. Loads data using a custom SQL query
2. Prepares prediction configuration
3. Delegates to prediction_process child workflow which:
   • Retrieves last timestamp for incremental processing
   • Applies input data quality gates
   • Executes MLFlow transform operation
   • Validates transform response
   • Executes MLFlow predict operation
   • Validates predict response
   • Delegates to format_and_export_prediction child workflow
4. The format_and_export_prediction workflow:
   • Formats prediction data (normal or default)
   • Exports to PI Web API (optional)
   • Exports to OPC server (optional)
   • Exports to PostgreSQL
   • Writes metrics

---

1. Predictions Batch - Main Workflow Scenarios

1.1 Success Scenarios

Scenario 1.1.1: Happy Path - Complete Success

Description: Workflow completes successfully with valid SQL query and all activities succeed

Input:

• Valid schedule_name, model_name, model_id
• Valid query returning non-empty DataFrame
• Valid schema, table_name, transform_table_name
• Optional datetime_columns for timestamp parsing
• Optional input_filters, mlflow_transform_filters, mlflow_predict_filters
• Optional path_priority, opc_output_config, pi_web_api_output_config

Expected Behavior:

• load_custom_query returns DataFrame with data
• Workflow prepares prediction input with all configurations
• prediction_process child workflow executes successfully
• All gates pass with no issues
• Transform and predict operations succeed
• Data exported to PostgreSQL
• Metrics written

Assertions:

• SQL query executed once
• prediction_process workflow called with correct parameters
• Data exists in PostgreSQL (predictions table)
• Metrics recorded
• No errors raised

---

1.2 Error Scenarios

Scenario 1.2.1: SQL Query Execution Error

Description: SQL query fails due to syntax error or connection issue

Input:

• Invalid SQL query (syntax error)
• Or database connection unavailable

Expected Behavior:

• load_custom_query raises exception (caught by Temporal retry policy)
• Notification sent with SQL error details
• After retries, activity may return empty data or workflow may fail
• If empty data returned, workflow completes with early exit via input gate

Assertions:

• Error notification sent
• Workflow completes (either fails or exits early)
• No data in predictions table

---

Scenario 1.2.2: Missing Required Parameters

Description: Essential parameters missing from input

Input:

• Missing query or model_id or schema or table_name

Expected Behavior:

• Workflow or activity raises KeyError or validation error
• Workflow fails immediately

Assertions:

• Workflow fails with parameter error
• Error notification sent
• No child workflow called

---

Scenario 1.2.3: Invalid Datetime Column Specification

Description: Datetime column specified doesn't exist in query results

Input:

• datetime_columns: ['nonexistent_column']
• Query results don't have this column

Expected Behavior:

• load_custom_query may raise KeyError or warning
• Depending on implementation, workflow may fail or continue
• Error notification sent

Assertions:

• Error raised or warning logged
• Workflow behavior depends on error handling policy

---

2. Prediction Process - Child Workflow Scenarios

2.1 Input gate Early Exit Scenarios

Scenario 2.1.1: Input Gate Triggers CONTINUE

Description: Input gate determines data should use previous prediction

Input:

• Data that should continue with input data as prediction
• input_filters configured with POLICY: 'CONTINUE'
• path_priority includes CONTINUE

Expected Behavior:

• input_gate returns path_flag='CONTINUE'
• path_flag_handler calls export workflow with input data directly
• MLFlow transform and predict skipped
• Data exported as-is

Assertions:

• input_gate called
• MLFlow operations NOT called
• Export workflow called with original data
• Workflow completes

Scenario 2.1.2: Input Gate Triggers STOP

Description: Input data quality gate fails with STOP policy

Input:

• Data with EMPTY_DATA or other critical issues
• input_filters configured with POLICY: 'STOP'

Expected Behavior:

• input_gate returns path_flag='STOP'
• path_flag_handler detects STOP
• Workflow returns early without calling MLFlow
• No prediction exported

Assertions:

• input_gate called
• path_flag_handler returns True (early exit)
• MLFlow transform NOT called
• Export workflow NOT called
• Workflow completes without error

Scenario 2.1.3: Input Gate Triggers REPEAT

Description: Input gate determines data should repeat last prediction

Input:

• Data with quality issues that require using previous prediction
• input_filters configured with POLICY: 'REPEAT'
• path_priority includes REPEAT

Expected Behavior:

• input_gate returns path_flag='REPEAT'
• path_flag_handler calls repeat_last_prediction activity
• MLFlow transform and predict skipped
• Last prediction repeated and exported

Assertions:

• input_gate called
• MLFlow operations NOT called
• repeat_last_prediction activity called
• Workflow completes

---

2.2 Transform gate Early Exit Scenarios

Scenario 2.2.1: Transform Gate Triggers CONTINUE

Description: Transform response gate determines data should continue despite issues

Input:

• Valid input data
• Transform response has quality issues but policy is CONTINUE
• mlflow_transform_filters configured with POLICY: 'CONTINUE'
• path_priority includes CONTINUE

Expected Behavior:

• request_transform succeeds
• mlflow_response_gate for transform returns path_flag='CONTINUE'
• path_flag_handler calls export workflow with transform data
• MLFlow predict skipped
• Transform data exported as-is

Assertions:

• Transform completed
• mlflow_response_gate called for transform
• MLFlow predict NOT called
• Export workflow called with transform data
• Workflow completes

---

Scenario 2.2.2: Transform Gate Triggers STOP

Description: Transform response validation fails with STOP policy

Input:

• Valid input data
• Transform response has critical errors
• mlflow_transform_filters configured with POLICY: 'STOP'

Expected Behavior:

• request_transform succeeds but response invalid
• mlflow_response_gate for transform returns path_flag='STOP'
• Workflow exits without calling predict or export

Assertions:

• Transform completed but validation failed
• mlflow_response_gate called for transform
• MLFlow predict NOT called
• Export workflow NOT called
• Workflow completes without error

---

Scenario 2.2.3: Transform Gate Triggers REPEAT

Description: Transform response gate determines data should repeat last prediction

Input:

• Valid input data
• Transform response has quality issues that require using previous prediction
• mlflow_transform_filters configured with POLICY: 'REPEAT'
• path_priority includes REPEAT

Expected Behavior:

• request_transform succeeds but response has issues
• mlflow_response_gate for transform returns path_flag='REPEAT'
• path_flag_handler calls repeat_last_prediction activity
• MLFlow predict skipped
• Last prediction repeated and exported

Assertions:

• Transform completed but validation triggered REPEAT
• mlflow_response_gate called for transform
• MLFlow predict NOT called
• repeat_last_prediction activity called
• Workflow completes

---

2.3 Predict gate Early Exit Scenarios

Scenario 2.3.1: Predict Gate Triggers CONTINUE

Description: Predict response gate determines data should continue despite issues

Input:

• Valid input and transform data
• Predict response has quality issues but policy is CONTINUE
• mlflow_predict_filters configured with POLICY: 'CONTINUE'
• path_priority includes CONTINUE

Expected Behavior:

• request_predict succeeds
• mlflow_response_gate for predict returns path_flag='CONTINUE'
• path_flag_handler calls export workflow with predict data
• Prediction exported despite quality issues

Assertions:

• Transform and predict completed
• mlflow_response_gate called for predict
• Export workflow called with predict data
• Workflow completes

---

Scenario 2.3.2: Predict Gate Triggers STOP

Description: Prediction validation fails with STOP policy

Input:

• Valid input and transform
• Predict response has critical errors
• mlflow_predict_filters configured with POLICY: 'STOP'

Expected Behavior:

• request_predict succeeds but response invalid
• mlflow_response_gate for predict returns path_flag='STOP'
• Workflow exits without export

Assertions:

• Transform completed
• Predict completed but validation failed
• Export workflow NOT called
• Workflow completes without error

---

Scenario 2.3.3: Predict Gate Triggers REPEAT

Description: Predict response gate determines data should repeat last prediction

Input:

• Valid input and transform data
• Predict response has quality issues that require using previous prediction
• mlflow_predict_filters configured with POLICY: 'REPEAT'
• path_priority includes REPEAT

Expected Behavior:

• request_predict succeeds but response has issues
• mlflow_response_gate for predict returns path_flag='REPEAT'
• path_flag_handler calls repeat_last_prediction activity
• Last prediction repeated and exported

Assertions:

• Transform and predict completed but validation triggered REPEAT
• mlflow_response_gate called for predict
• repeat_last_prediction activity called
• Export workflow NOT called with current prediction
• Workflow completes

---

3. Format and Export Prediction - Child Workflow Scenarios

3.1 Success Scenarios

Scenario 3.1.1: Default Prediction Export

Description: Error prediction path creates default prediction

Input:

• path_flag: 'ERROR' or other non-None value (not STOP/CONTINUE/REPEAT)
• comment provided with error details

Expected Behavior:

• format_default_prediction called instead of format_prediction
• Default prediction created with error metadata
• Exported to PostgreSQL only
• Transformed data NOT processed
• Metrics written

Assertions:

• format_default_prediction called
• format_prediction NOT called
• format_transformed_data NOT called
• One PostgreSQL export only
• Default values in prediction data
• Comment included

---

Scenario 3.1.2: Export with OPC only

Description: Export to PostgreSQL and OPC server only (no PI Web API)

Input:

• path_flag: None
• opc_output_config configured with valid OPC settings
• pi_web_api_output_config: None or {}

Expected Behavior:

• Normal formatting
• PostgreSQL export executed
• OPC export executed
• PI Web API activity skipped
• Metrics written with OPC metrics

Assertions:

• PI Web API activity NOT called
• OPC activity called
• PostgreSQL export called
• Metrics written with opc_metrics populated

---

Scenario 3.1.3: Export with PI Web API only

Description: Export to PostgreSQL and PI Web API only (no OPC)

Input:

• path_flag: None
• pi_web_api_output_config configured with valid PI Web API settings
• opc_output_config: None or {}

Expected Behavior:

• Normal formatting
• PostgreSQL export executed
• PI Web API export executed
• OPC activity skipped
• Metrics written without OPC metrics

Assertions:

• OPC activity NOT called
• PI Web API activity called
• PostgreSQL export called
• Metrics written with empty opc_metrics

---

Scenario 3.1.4: Export Without Optional Outputs

Description: Export only to PostgreSQL (no OPC or PI Web API)

Input:

• path_flag: None
• opc_output_config: None or {}
• pi_web_api_output_config: None or {}

Expected Behavior:

• Normal formatting
• Only PostgreSQL export executed
• OPC and PI Web API activities skipped
• Metrics written without OPC metrics

Assertions:

• PI Web API activity NOT called
• OPC activity NOT called
• PostgreSQL export called
• Metrics written with empty opc_metrics

---

Scenario 3.1.5: Export Without Transformed Data

Description: Only prediction exported, no transform table

Input:

• path_flag: None
• transformed_data: None or save_transform: False
• opc_output_config: None or {}
• pi_web_api_output_config: None or {}

Expected Behavior:

• Only prediction formatted and exported
• Transform export skipped
• Single PostgreSQL write

Assertions:

• format_transformed_data NOT called
• One PostgreSQL export
• Transform table remains empty

---

3.2 Error Scenarios

These paths do not rely on Temporal activity retries for export failures: the write activities run once, errors are handled inside the activity, and the workflow completes successfully with degraded metadata on the persisted prediction (prediction_confidence and comments).

Scenario 3.2.1: PI Web API Write Error

Description: PI Web API export fails

Input:

• Valid prediction
• PI Web API service unavailable or invalid config

Expected Behavior:

• write_pi_web_api_data surfaces the failure (exception handled in the activity layer)
• Notification may be sent
• Workflow completes (does not fail)
• Prediction row is still written to PostgreSQL with error confidence 13 and a comment describing the PI error
• Subsequent steps (e.g. OPC, Postgres) still run per workflow order with the updated prediction payload

Assertions:

• PI Web API error notification sent (when applicable)
• Workflow completes
• PostgreSQL contains the prediction with prediction_confidence 13 and expected comments

---

Scenario 3.2.2: OPC Write Error

Description: OPC server write fails

Input:

• Valid prediction
• OPC server unavailable or invalid configuration

Expected Behavior:

• write_opc_data reports failure without aborting the workflow
• Notification may be sent
• Workflow completes (does not fail)
• Prediction row is written to PostgreSQL with OPC error confidence 12 and a comment indicating OPC write issues

Assertions:

• OPC error notification sent (when applicable)
• Workflow completes
• PostgreSQL contains the prediction with prediction_confidence 12 and expected comments

---

Scenario 3.2.4: OPC Session / Channel Bad* (Tier-1)

Description: OPC write fails with a Tier-1 session or channel status (e.g. BadSessionIdInvalid) while transport may still appear open on the client

Input:

• Valid prediction and OPC output config
• Mock or server returning Tier-1 UaStatusCodeError on write (no write retry in the same activity)

Expected Behavior:

• write_opc_data fails forward for affected tags; background reconnect may be scheduled if OPC_RECONNECTION_INTERVAL allows
• Workflow completes
• PostgreSQL row uses prediction_confidence 14 and comment prefix OPC UA session/channel error: (including OPC status name)
• opc_write_attempts_total records result=BadSessionIdInvalid (or matching status); no second write attempt in the same activity

Assertions:

• Workflow completes
• prediction_confidence = 14
• comments matches OPC UA session/channel error:%
• Generic OPC error confidence 12 is not used for this case

Reference: docs/opc-communication.md, plan .cursor/plans/opc_bad_reconnect_ac4c6045.plan.md

---

Scenario 3.2.5: OPC Write Blocked During Reconnect

Description: A write is attempted while the repository is reconnecting (session not ready)

Input:

• Valid prediction
• Simulated slow reconnect (e.g. delayed connect) or concurrent writes where the first triggers reconnect

Expected Behavior:

• Second write (or parallel write) is rejected immediately when reconnect is in progress or _session_ready is cleared — without calling write_value
• No wait/sleep on the write path; no duplicate connect from parallel writers (connection lock)
• prediction_confidence = 14, comments = OPC UA reconnect in progress (distinguish from Tier-1 Bad* via comment prefix in SQL)

Assertions:

• At most one reconnect sequence (disconnect + connect) for the overlapping window
• No write retry after failure
• Tests in test_opc_repository (unit) and optional e2e in test_predictions_batch_format_export.py

---

Scenario 3.2.3: PI Web API Partial Write Error

Description: Two prediction tags attempt to be written to PI Web API, but only one succeeds

Input:

• Valid prediction
• Two prediction tags configured
• PI Web API returns partial success (one tag succeeds, one fails)

Expected Behavior:

• write_pi_web_api_data processes response
• process_pi_web_api_response detects partial failure
• Error confidence set (13)
• Notification sent for failed tag
• Workflow completes with error confidence (single activity attempt; no retry loop)

Assertions:

• One tag written successfully
• One tag failed
• Error confidence set in prediction
• Error notification sent
• Workflow completes

---

4. Model Import Workflow Scenarios (import_model)

The import_model workflow turns an encrypted .sientia bundle already in MinIO into a registered MLflow model plus one MongoDB models document — or into a durable error on the row the BFF inserted. It shares no code path with the predictions workflows above and has its own harness.

4.0 Running the import E2E tests

• Marker: every test below carries import_e2e; the real-clock retry tests also carry slow. testpaths stays ["tests"], so none of this is collected by the unit gate.
• Invocation: pytest e2e -m import_e2e.
• Runtime: Docker for three containers (the shared MinIO, a dedicated Postgres for sientia-core-mlops-bff, MongoDB) plus an mlflow server subprocess started from this repository's venv. Dev requirements install testcontainers[postgres,minio,mongodb].
• Fidelity: on the import path nothing is mocked — real MinIO, real Postgres, real MongoDB with a unique index, a real MLflow tracking server with a working registry, real Argon2id and real secretstream. Only CoreNotificationHandler's Mongo client stays mocked, because notifications are a side channel this suite counts rather than a dependency it tests.
• Fixtures: e2e/import_harness.py, registered as a pytest plugin from e2e/conftest.py. Every fixture is import_-prefixed.
• Prerequisite: the one-time reaper setup at the top of this document (Running automated E2E tests). Without it every test here errors at setup with a Docker mount error and never reaches an assertion.

4.1 Success Scenarios

---

Scenario 4.1.1: Producer-Emitted Bundle Imports End to End

Description: The committed golden .sientia file — bytes the producer's own code wrote — is imported at the producer's real Argon2id cost (256 MiB, 3 passes)

Input:

• tests/fixtures/bundle/golden_model_v3.sientia uploaded under imported_models/
• One PENDING / run_type='IMPORT' row carrying the object key and the uploader's digest
• The workflow input: import_run_id, bucket, object_key, expected_digest, password_envelope

Expected Behavior:

• The row is claimed, the seven gates pass, the experiment and a new run are created
• The artifact tree is logged with prediction_model and data_model at the run's root
• A registry version is created and left unstaged; the model document is written inactive
• One terminal write sets IMPORT_SUCCESS

Assertions:

• Exactly one run named import-fixture_model-v3, status FINISHED
• prediction_model contains MLmodel and model.pkl
• Bundle parameters logged, origin logged under source.*
• Destination version is 1, independent of the bundle's model_version of 3
• Model document id is the seeded maximum plus one; active is false
• Row: IMPORT_SUCCESS, error_message null, both names filled

---

Scenario 4.1.2: Factory-Built Bundle Imports End to End

Description: The same assertions over a synthesised tree with names unique to the test

Input: A bundle from tests/helpers/bundle_factory.py at the cheap KDF cost

Expected Behavior: As 4.1.1

Assertions:

• As 4.1.1, plus model_config holds exactly transform_flavor, predict_flavor, retention_minutes and target
• import.workflow_id in the row's detail JSON matches the run

---

Scenario 4.1.3: The Registry Source Resolves Through the Legacy Loader

Description: The source the importer writes must be one the existing loader can parse

Input: A successful import's registered version

Expected Behavior: MLFlowRepository.get_model_run_id extracts the run id from the source

Assertions:

• The source ends in /prediction_model
• get_model_run_id returns the run the import created
• Both artifact paths are present under that run
• Called with the stage the version is actually in, because an import promotes nothing

---

Scenario 4.1.4: Second Import of the Same Model

Description: The same model imported twice, with different declared model_versions

Input: Two bundles declaring model_version 2 then 7, same model and experiment names

Expected Behavior: Additive — a new run and a new registry version, nothing promoted, the existing model document untouched

Assertions:

• Two versions (1, 2) of one registered model; destination numbering ignores the bundle's
• Two runs in the same experiment, each logging its own source.model_version
• No version in Production
• The model document is byte-identical to what the first import wrote

---

Scenario 4.1.5: A Model Name Already in the Collection

Description: An operator-owned document already carries the bundle's model name

Input: A document with active: True, predict_flavor: pyfunc and a hand-set target

Expected Behavior: The import notices the name and leaves the document alone; the import still succeeds

Assertions:

• The collection is unchanged, exactly one document carries that name, no duplicate
• Row: IMPORT_SUCCESS — a document that is already there is not a failure

---

4.2 Gate Rejection Scenarios

Every case asserts the same five things: terminal status ORCHESTRATOR_VALIDATION_ERROR, the recorded step, the recorded code, error_message equal to the catalog sentence for that step, and that no experiment, run, artifact, registry version, model document or extracted file remains.

---

Scenario 4.2.1: Object Above the Size Ceiling (gate 1)

Description: An object larger than IMPORT_MAX_OBJECT_BYTES, rejected from stat_object before its body is fetched

Input: A factory bundle against a ceiling narrowed to 1 KiB for this test class

Expected Behavior: Rejected at download

Assertions: The five shared assertions

---

Scenario 4.2.2: Malformed Header (gate 2)

Description: Five variants — wrong magic, unsupported format version, KDF memory above the ceiling, KDF passes above the ceiling, a header digest that does not describe the chunk stream

Expected Behavior: Each rejected at download, sharing gate 1's step

Assertions:

• The five shared assertions
• The recorded sentence does not mention the password: gates 1 and 2 must not implicate a password that is fine

---

Scenario 4.2.3: Decryption Failures (gate 3)

Description: Three variants — a wrong bundle password inside a valid envelope, a tampered chunk with the header digest repaired, and a stream whose every chunk is framed MESSAGE

Expected Behavior: Each rejected at decryption

Assertions:

• The five shared assertions
• The tampered case proves the file is self-consistent and only the AEAD can tell
• The framing case proves the digest cannot see an incomplete stream

---

Scenario 4.2.4: Hostile Archive (gate 4)

Description: Four variants — an entry escaping the tree, an absolute entry path, a symbolic link, and a declared expansion over the ceiling

Expected Behavior: Each rejected at archive_inspection, from the central directory alone

Assertions:

• The five shared assertions
• Nothing extracted: the worker work directory is empty, not merely free of provisioning

---

Scenario 4.2.5: A Member Extraction Cannot Place (gate 5)

Description: An archive declaring the same member name twice — passes gate 4, which checks names and modes but not duplicates

Expected Behavior: Rejected at extraction on the second member

Assertions: The five shared assertions

---

Scenario 4.2.6: Incomplete Structure (gate 6)

Description: Five variants — no metadata.json, metadata missing run_id, parameters missing target_variable, no prediction_model directory, and a 60-character experiment name

Expected Behavior: Each rejected at structure_validation

Assertions:

• The five shared assertions
• For the over-long name: the row's experiment_name is still the inserted placeholder, so no write reached the VARCHAR(50) column, and that column is confirmed still 50 wide

---

Scenario 4.2.7: Forbidden File Type (gate 7)

Description: An artifacts/payload.exe member with a suffix outside the allow-list

Expected Behavior: Rejected at content_policy

Assertions: The five shared assertions

---

4.3 Digest Enforcement Scenarios

---

Scenario 4.3.1: A Digest Computed Over Other Bytes

Description: A perfectly valid bundle whose expected_digest belongs to a different file

Expected Behavior: Fails at download, before any header is parsed

Assertions:

• Terminal status, step, code and catalog sentence
• The failure names the digest, not the magic or the version
• run_name is still null and experiment_name still the placeholder — the bundle was never opened

---

Scenario 4.3.2: The Object Is Replaced After Upload

Description: The same key, different bytes — also a valid bundle, so nothing but the digest can tell

Expected Behavior: Fails at download

Assertions: As 4.3.1; only a real object store can show this at all

---

Scenario 4.3.3: No Digest at All

Description: expected_digest absent from the input

Expected Behavior: Fails at received, before anything is downloaded

Assertions: The request-invalid code, and the work directory was never created

---

4.4 Provisioning Failure Scenarios

Each asserts the platform terminal status IMPORT_ERROR — not the validation status, because a provisioning failure is not the user's to fix — its own step and code, the workflow identifier in the detail JSON, and error_message equal to the catalog sentence.

---

Scenario 4.4.1: The Tracking Server Is Stopped

Description: The mlflow server process is genuinely terminated for the duration

Expected Behavior: Fails at experiment_creation

Assertions:

• The shared assertions
• The names are already on the row: the bundle was opened before MLflow was reached
• Nothing in Mongo; nothing provisioned

---

Scenario 4.4.2: Artifact Upload Fails

Description: log_artifacts is patched — the one step where neither hostile input nor a broken dependency can isolate the failure

Expected Behavior: Fails at artifact_upload

Assertions:

• The shared assertions
• The experiment exists (the previous step succeeded); no version, no document

---

Scenario 4.4.3: Registration Fails

Description: create_model_version is patched. MLflow rejects only an empty registered-model name, and one process serves both tracking and the registry, so nothing else isolates this step

Expected Behavior: Fails at registration

Assertions:

• The shared assertions
• The run and both artifact paths survive — "the files were stored, the model was not registered"

---

Scenario 4.4.4: The Model Document Is Refused

Description: A real unique index on model_config.target refuses the activity's own insert. (A duplicate id cannot be constructed: the activity derives max(int(id)) + 1 from the same collection, so the id it computes can never already exist)

Expected Behavior: Fails at model_document

Assertions:

• The shared assertions
• One registry version exists — registered, and unusable

---

4.5 Status Record Scenarios

---

Scenario 4.5.1: The Claim Precedes Every Gate

Description: The row is observed from inside stat_object, the first call gate 1 makes

Assertions: Running status and import.workflow_id already present at that moment

---

Scenario 4.5.2: The Names Are Visible Mid-Flight

Description: The row is observed from inside the first call experiment creation makes, during an import that then succeeds

Assertions: experiment_name and run_name already populated before anything exists in MLflow

---

Scenario 4.5.3: The Inserting Side's Columns Are Never Rewritten

Description: After a success and after a failure

Assertions: username, request_data, bucket_name, file_name, run_type, created_at unchanged; status and updated_at did change

---

Scenario 4.5.4: Unclaimable Rows

Description: Three variants — an unknown id, a run_type='TRAIN' row, and a row already held by another workflow id

Expected Behavior: The Temporal run ends failed with the not-claimable error type, and nothing is written to any row

Assertions: The error type from the history, the whole table byte-identical, nothing provisioned

---

Scenario 4.5.5: The Row Names a Different Object

Description: file_name disagrees with the input's object_key

Expected Behavior: Fails at received with the error recorded on the row — unlike 4.5.4, this row was successfully claimed

Assertions: The received code and sentence on the row; nothing provisioned

---

Scenario 4.5.6: Constraint Conformance

Description: The real constraints, exercised directly

Assertions:

• Every status a normal import writes is accepted on an IMPORT row
• All four terminal values are refused across run types, naming chk_experiment_run_terminal_status_run_type; the shared stage values are accepted on both
• A training row still walks its own vocabulary
• The table has exactly the 13 columns V1…V12 leave (no ALTER TABLE by this feature)

---

4.6 Absence Scenarios

---

Scenario 4.6.1: One Table, One Database

Description: A full import bracketed by marker statements on both Postgres servers, read from log_statement=all

Assertions:

• Every statement touching a user relation names experiment_run
• No project, models, import_run, model_project or model_registry reference (with quoted literals stripped, so the bundle's own model_project value is not mistaken for a table)
• The predictions database logs no statement at all, though the import holds working credentials for it
• A rejected import — which writes more status rows than a successful one — behaves the same

---

4.7 Cleanup, Retention and History Scenarios

---

Scenario 4.7.1: The Worker Filesystem Is Clean

Description: After a success and after rejections at two different steps (decryption and structure validation, which unwind on different code paths)

Assertions: No extracted tree, no downloaded object, no temporary archive

---

Scenario 4.7.2: The Retention Rule Is Present

Assertions: sientia-import-bundle-expiry on the import bucket, enabled, filtered on imported_models/, expiring in 7 days — and the uploaded object still there, because the import never deletes it

---

Scenario 4.7.3: The Record Outlives the Object

Description: The uploaded object is deleted, as retention will eventually do

Assertions: The row still reads IMPORT_SUCCESS with its names

---

Scenario 4.7.4: No Plaintext Secret in the Event History

Assertions:

• The bundle password, the base64 envelope key and the key's hex form appear nowhere in the history
• password_envelope is in the history, so the negative assertions above cannot pass vacuously

---

4.8 Retry Envelope and Missing-V12 Scenarios

These run on a real clock (WorkflowEnvironment.start_local()) with the status retry intervals shortened; the production values stay asserted by the unit suite. All are marked slow.

---

Scenario 4.8.1: A Database That Comes Back

Description: The status database is stopped for two seconds after provisioning and restarted inside the retry envelope

Assertions: The terminal write lands IMPORT_SUCCESS with the correct workflow id

---

Scenario 4.8.2: A Broken Progress Update

Description: A brief early outage that only the progress hints see

Assertions: The import completes with the correct verdict and does not spend the terminal write's envelope on a hint nobody reads

---

Scenario 4.8.3: A Database That Stays Down After a Successful Import

Description: The database is taken away as soon as the row is claimed and never comes back

Expected Behavior: The model is fully provisioned and the workflow completes — reporting failure would tell somebody to re-import a model that is already on the platform

Assertions:

• The run returns its result; the model, version and document all exist
• The failure channel carries IMPORT_STATUS_NOT_RECORDED, step=status_write and verdict_not_recorded=
• The row is left in the running status

---

Scenario 4.8.4: A Failed Import Whose Failure Cannot Be Written Either

Description: The import fails at artifact_upload and the terminal write is then refused

Assertions:

• The reported cause is the artifact-upload failure, not the status write
• The status write reports itself only through the channel
• The row is left running

---

Scenario 4.8.5: A Table Without V12

Description: The evidence for why the migration gates enabling the import worker

Expected Behavior: The claim succeeds (V11 gave the table run_type; PENDING and the running status are pre-V12 values), the model is fully provisioned, and only the terminal value is new — so the last write of the import is the first thing that fails, non-retryably

Assertions:

• The run completes; the model, version and document all exist
• The row is left in the running status, with both names filled
• The channel carries the not-recorded code and the verdict
• Elapsed time shows the refusal was not retried
• pg_constraint confirms the fixture's chk_experiment_run_status really does not allow IMPORT_SUCCESS, so the test cannot pass vacuously

---

5. joblib Flavor Scenarios

joblib is an explicit model flavor, declared in model_config.predict_flavor / model_config.transform_flavor like any other. It reads the artifact directly — download the .pkl, put the artifact's code/ on sys.path, joblib.load — and never calls an MLflow flavor loader. There is no fallback in either direction: a failing loader propagates.

Covered by e2e/test_predictions_batch_joblib_flavor.py (read) and e2e/test_retrain_joblib_flavor.py (write). Both use the joblib_flavor_artifacts fixture, which builds one artifact directory per model type holding a real compressed model.pkl plus a code/ directory carrying the module where the model's class lives. The fixture takes that directory off sys.path again before yielding, so the pickle is unloadable until the flavor puts it back — that is what makes these tests prove module resolution rather than assume it. The fixture also restores sys.path in teardown, because the flavor deliberately never removes what it inserted.

5.1 Read Scenarios

---

Scenario 5.1.1: Predict Through the joblib Flavor

Description: predict_flavor: 'joblib' on a normal predictions_batch run

Input:

• model_config.predict_flavor = 'joblib', transform_flavor = 'sklearn'
• client.download_artifacts serving the real prediction_model artifact

Expected Behavior:

• The artifact is downloaded once and deserialized with joblib.load
• No models:/<name>/production URI is built, so no flavor loader ever sees this model
• The prediction the artifact's own object produces is what lands in Postgres

Assertions:

• Exactly one prediction row, prediction_status='Good', value matching the artifact's model
• download_artifacts called once, with 'prediction_model'
• No mlflow.{sklearn,pyfunc,pytorch}.load_model call names this model

---

Scenario 5.1.2: Transform Through the joblib Flavor

Description: transform_flavor: 'joblib', which had no E2E coverage before

Input:

• model_config.transform_flavor = 'joblib', predict_flavor = 'sklearn'
• client.download_artifacts serving the real data_model artifact

Expected Behavior:

• data_model is downloaded and deserialized; the prediction model stays on its MLflow flavor
• No artifact URI is resolved for the transform, so mlflow.get_run is never called

Assertions:

• The prediction lands with the mocked prediction model's value
• download_artifacts called once, with 'data_model'
• mlflow.get_run not called; no flavor loader was given a /data_model URI

---

Scenario 5.1.3: An MLflow Flavor Is Unaffected

Description: A model declaring sklearn behaves exactly as before

Assertions:

• The prediction lands with the mocked model's value
• download_artifacts NOT called — the joblib path is never entered as a safety net

---

5.2 Error Scenarios

---

Scenario 5.2.1: joblib Flavor Failure Has a Single Cause

Description: The downloaded artifact has no top-level .pkl

Expected Behavior: The workflow stops. The reported cause is the missing pickle and nothing else — with no fallback there is no second exception chained to a first

Assertions:

• The run stops (assert_stop)
• The log carries No .pkl file found in prediction_model artifact
• The log does not carry Original error, the marker of a chained cause

---

5.3 Write Scenarios

---

Scenario 5.3.1: Retraining a joblib Model Round-Trips

Description: retrain_model on a model whose predict_flavor and transform_flavor are both joblib — write and read in one test

Input:

• Training frame indexed by DATETIME_FORMAT_WITH_TZ timestamps, carrying the target column
• MLflow's artifact I/O replaced by a fake store that copies into an inspectable run directory

Expected Behavior:

• Both models load through the joblib flavor; no flavor loader runs
• log_model writes prediction_model/model.pkl (a joblib.dump(..., compress=3) pickle) and re-logs prediction_model/code/, without an MLmodel manifest
• Loading the written artifact through the joblib flavor deserializes it and predicts, after the module the first load imported has been dropped from sys.modules

Assertions:

• retrain_model returns success: True
• download_artifacts was asked for both data_model and prediction_model
• sklearn/pyfunc loaders have zero calls
• The written run has prediction_model/model.pkl, prediction_model/code/<module>.py and data_model/model.pkl, and no prediction_model/MLmodel
• The reloaded model's prediction equals the original's, and the written code/ is on sys.path

---