gitea_admin/sientia-dataops-laborious-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

QTZPOC-20

Browse Source 
QTZPOC-20: Implement explicit joblib model flavor and enhance model import workflow with extensive E2E tests.

This commit introduces a first-class joblib model flavor for direct loading and saving of Python pickle artifacts, including custom code within the artifact's code/ directory, bypassing MLflow's flavor loaders. The model import workflow's status database configuration is now streamlined to default to the POSTGRES_* server, with explicit IMPORT_STATUS_DB_* overrides.

Comprehensive end-to-end tests have been added for the model import workflow, covering success paths, gate rejections, digest enforcement, provisioning failures, status record integrity, cleanup, and retry/recovery scenarios using a real-clock environment. New E2
This commit is contained in:
Vitor Pimentel dos Santos
2026-08-26 15:45:33 -03:00
parent dac4f1718f
commit e95e65b72b
32 changed files with 4247 additions and 1040 deletions
Download Patch File Download Diff File Expand all files Collapse all files

642
e2e/scenarios.md
View File

@@ -4,12 +4,71 @@ This document describes all possible test scenarios for the `predictions_batch`
## Running automated E2E tests (`e2e/`)
### One-time setup: disable the testcontainers reaper
Do this once per machine, before running anything under `e2e/`:
```bash
echo "ryuk.disabled=true" >> ~/.testcontainers.properties
```
**Why.** Before starting any test container, testcontainers starts a reaper container ("Ryuk") whose
job is to clean up containers if the test process dies without running its teardown. Ryuk bind-mounts
the Docker socket. Under **Docker Desktop on Linux** the socket lives at
`~/.docker/desktop/docker.sock`, which Docker Desktop's File Sharing does not expose, so the mount is
denied and the reaper cannot start:
```
docker.errors.APIError: 500 Server Error ... /containers/<id>/start:
Mounts denied: The path /socket_mnt/home/<user>/.docker/desktop/docker.sock
is not shared from the host and is not known to Docker.
```
Every test then errors **at setup** — `ERROR`, not `FAILED`, and never inside an assertion. If you see
that, this is why.
**What you give up.** Nothing in a normal run: every fixture in `conftest.py` and
`import_harness.py` stops its own containers in teardown. Ryuk only covers the case where the process
is killed without teardown (`kill -9`, a crash, a hard stop from the IDE). After one of those, clean
up by hand:
```bash
docker ps -a --filter "label=org.testcontainers=true" -q | xargs -r docker rm -f
```
The setting is read by `testcontainers/core/config.py`, which accepts either `ryuk.disabled` in
`~/.testcontainers.properties` or the `TESTCONTAINERS_RYUK_DISABLED` environment variable. The
properties file is preferred here because it is per-machine and applies to every project and every
shell, so nobody has to remember a prefix on the command line.
*Alternative, if you want to keep the reaper:* point Docker at the classic socket with
`export DOCKER_HOST=unix:///var/run/docker.sock`. That only works if a native Docker Engine is
running alongside Docker Desktop; if the Desktop daemon is the only one, disabling Ryuk is the way.
### Requirements and invocation
- **Runtime**: Docker (or a Docker-compatible daemon) must be available so [testcontainers](https://testcontainers.com/) can start **PostgreSQL** and **MinIO** containers.
- **Dependencies**: install dev requirements (includes `testcontainers[postgres,minio]`).
- **Dependencies**: install dev requirements (includes `testcontainers[postgres,minio,mongodb]`).
- **Invocation**: run only integration-marked tests, for example: `pytest e2e/ -m integration`.
- **MinIO tests**: `e2e/test_minio_offload.py` exercises real S3 uploads; other E2E modules continue to mock MinIO on the worker used by most scenarios.
- **OPC tests (real server)**: `e2e/test_opc_real_server.py` uses an in-process **asyncua** server and real `OpcRepository` (`test_activities_real_opc`). Scenarios 3.1.2, 3.2.2, 3.2.4, and 3.2.5 are covered there. Other E2E modules keep the OPC mock.
- Run only OPC real-server tests: `pytest e2e/test_opc_real_server.py -m "integration and opc"`.
- **Model import tests**: the `import_model` suite is opt-in and has its own harness, containers and
fidelity contract — see [§4.0](#40-running-the-import-e2e-tests). Run it with
`pytest e2e -m import_e2e`.
- **joblib flavor tests**: `e2e/test_predictions_batch_joblib_flavor.py` covers reading a model
through the explicit `joblib` flavor (predict and transform), and
`e2e/test_retrain_joblib_flavor.py` covers writing one back on a retrain — see
[§5](#5-joblib-flavor-scenarios). Neither `joblib.dump` nor `joblib.load` is mocked in either
module: the artifact fixture holds a real `joblib.dump(..., compress=3)` pickle whose class ships
inside the artifact's `code/` directory, so only the flavor's own `sys.path` handling makes it
deserialize. Run them with
`pytest e2e/test_predictions_batch_joblib_flavor.py e2e/test_retrain_joblib_flavor.py -m integration`.
- **Nothing under `e2e/` is covered by `validate`**: that script lints `laborious/` and `tests/`
only, and its test step is `pytest tests/` (`testpaths = ["tests"]`). Lint these files by hand
(`ruff format --check e2e/ && ruff check e2e/`), and remember that `addopts` carries
`--strict-markers`: a test without `@pytest.mark.integration` is not collected, and a run that
collects nothing still exits 0. Check the collected count, not the exit code.
## Workflow Overview
@@ -561,4 +620,583 @@ These paths do **not** rely on Temporal activity retries for export failures: th
- Error notification sent
- Workflow completes
---
---
## 4. Model Import Workflow Scenarios (`import_model`)
The `import_model` workflow turns an encrypted `.sientia` bundle already in MinIO into a registered
MLflow model plus one MongoDB `models` document — or into a durable error on the row the BFF
inserted. It shares no code path with the predictions workflows above and has its own harness.
### 4.0 Running the import E2E tests
- **Marker**: every test below carries `import_e2e`; the real-clock retry tests also carry `slow`.
`testpaths` stays `["tests"]`, so none of this is collected by the unit gate.
- **Invocation**: `pytest e2e -m import_e2e`.
- **Runtime**: Docker for three containers (the shared MinIO, a dedicated Postgres for
`sientia-core-mlops-bff`, MongoDB) **plus** an `mlflow server` subprocess started from this
repository's venv. Dev requirements install `testcontainers[postgres,minio,mongodb]`.
- **Fidelity**: on the import path nothing is mocked — real MinIO, real Postgres, real MongoDB with a
unique index, a real MLflow tracking server with a working registry, real Argon2id and real
`secretstream`. Only `CoreNotificationHandler`'s Mongo client stays mocked, because notifications
are a side channel this suite counts rather than a dependency it tests.
- **Fixtures**: `e2e/import_harness.py`, registered as a pytest plugin from `e2e/conftest.py`. Every
fixture is `import_`-prefixed.
- **Prerequisite**: the one-time reaper setup at the top of this document
([Running automated E2E tests](#one-time-setup-disable-the-testcontainers-reaper)). Without it every
test here errors at setup with a Docker mount error and never reaches an assertion.
### 4.1 Success Scenarios
---
#### Scenario 4.1.1: Producer-Emitted Bundle Imports End to End
**Description**: The committed golden `.sientia` file — bytes the producer's own code wrote — is
imported at the producer's real Argon2id cost (256 MiB, 3 passes)
**Input**:
- `tests/fixtures/bundle/golden_model_v3.sientia` uploaded under `imported_models/`
- One `PENDING` / `run_type='IMPORT'` row carrying the object key and the uploader's digest
- The workflow input: `import_run_id`, `bucket`, `object_key`, `expected_digest`, `password_envelope`
**Expected Behavior**:
- The row is claimed, the seven gates pass, the experiment and a new run are created
- The artifact tree is logged with `prediction_model` and `data_model` at the run's root
- A registry version is created and left unstaged; the model document is written inactive
- One terminal write sets `IMPORT_SUCCESS`
**Assertions**:
- Exactly one run named `import-fixture_model-v3`, status `FINISHED`
- `prediction_model` contains `MLmodel` and `model.pkl`
- Bundle parameters logged, origin logged under `source.*`
- Destination version is `1`, independent of the bundle's `model_version` of `3`
- Model document `id` is the seeded maximum plus one; `active` is `false`
- Row: `IMPORT_SUCCESS`, `error_message` null, both names filled
---
#### Scenario 4.1.2: Factory-Built Bundle Imports End to End
**Description**: The same assertions over a synthesised tree with names unique to the test
**Input**: A bundle from `tests/helpers/bundle_factory.py` at the cheap KDF cost
**Expected Behavior**: As 4.1.1
**Assertions**:
- As 4.1.1, plus `model_config` holds exactly `transform_flavor`, `predict_flavor`,
`retention_minutes` and `target`
- `import.workflow_id` in the row's detail JSON matches the run
---
#### Scenario 4.1.3: The Registry Source Resolves Through the Legacy Loader
**Description**: The `source` the importer writes must be one the *existing* loader can parse
**Input**: A successful import's registered version
**Expected Behavior**: `MLFlowRepository.get_model_run_id` extracts the run id from the source
**Assertions**:
- The source ends in `/prediction_model`
- `get_model_run_id` returns the run the import created
- Both artifact paths are present under that run
- Called with the stage the version is actually in, because an import promotes nothing
---
#### Scenario 4.1.4: Second Import of the Same Model
**Description**: The same model imported twice, with different declared `model_version`s
**Input**: Two bundles declaring `model_version` `2` then `7`, same model and experiment names
**Expected Behavior**: Additive — a new run and a new registry version, nothing promoted, the
existing model document untouched
**Assertions**:
- Two versions (`1`, `2`) of one registered model; destination numbering ignores the bundle's
- Two runs in the same experiment, each logging its own `source.model_version`
- No version in `Production`
- The model document is byte-identical to what the first import wrote
---
#### Scenario 4.1.5: A Model Name Already in the Collection
**Description**: An operator-owned document already carries the bundle's model name
**Input**: A document with `active: True`, `predict_flavor: pyfunc` and a hand-set target
**Expected Behavior**: The import notices the name and leaves the document alone; the import still
succeeds
**Assertions**:
- The collection is unchanged, exactly one document carries that name, no duplicate
- Row: `IMPORT_SUCCESS` — a document that is already there is not a failure
---
### 4.2 Gate Rejection Scenarios
Every case asserts the same five things: terminal status `ORCHESTRATOR_VALIDATION_ERROR`, the
recorded step, the recorded code, `error_message` **equal** to the catalog sentence for that step,
and that no experiment, run, artifact, registry version, model document or extracted file remains.
---
#### Scenario 4.2.1: Object Above the Size Ceiling (gate 1)
**Description**: An object larger than `IMPORT_MAX_OBJECT_BYTES`, rejected from `stat_object` before
its body is fetched
**Input**: A factory bundle against a ceiling narrowed to 1 KiB for this test class
**Expected Behavior**: Rejected at `download`
**Assertions**: The five shared assertions
---
#### Scenario 4.2.2: Malformed Header (gate 2)
**Description**: Five variants — wrong magic, unsupported format version, KDF memory above the
ceiling, KDF passes above the ceiling, a header digest that does not describe the chunk stream
**Expected Behavior**: Each rejected at `download`, sharing gate 1's step
**Assertions**:
- The five shared assertions
- The recorded sentence does **not** mention the password: gates 1 and 2 must not implicate a
password that is fine
---
#### Scenario 4.2.3: Decryption Failures (gate 3)
**Description**: Three variants — a wrong bundle password inside a valid envelope, a tampered chunk
with the header digest repaired, and a stream whose every chunk is framed `MESSAGE`
**Expected Behavior**: Each rejected at `decryption`
**Assertions**:
- The five shared assertions
- The tampered case proves the file is self-consistent and only the AEAD can tell
- The framing case proves the digest cannot see an incomplete stream
---
#### Scenario 4.2.4: Hostile Archive (gate 4)
**Description**: Four variants — an entry escaping the tree, an absolute entry path, a symbolic
link, and a declared expansion over the ceiling
**Expected Behavior**: Each rejected at `archive_inspection`, from the central directory alone
**Assertions**:
- The five shared assertions
- **Nothing extracted**: the worker work directory is empty, not merely free of provisioning
---
#### Scenario 4.2.5: A Member Extraction Cannot Place (gate 5)
**Description**: An archive declaring the same member name twice — passes gate 4, which checks names
and modes but not duplicates
**Expected Behavior**: Rejected at `extraction` on the second member
**Assertions**: The five shared assertions
---
#### Scenario 4.2.6: Incomplete Structure (gate 6)
**Description**: Five variants — no `metadata.json`, metadata missing `run_id`, parameters missing
`target_variable`, no `prediction_model` directory, and a 60-character experiment name
**Expected Behavior**: Each rejected at `structure_validation`
**Assertions**:
- The five shared assertions
- For the over-long name: the row's `experiment_name` is still the inserted placeholder, so no write
reached the `VARCHAR(50)` column, and that column is confirmed still 50 wide
---
#### Scenario 4.2.7: Forbidden File Type (gate 7)
**Description**: An `artifacts/payload.exe` member with a suffix outside the allow-list
**Expected Behavior**: Rejected at `content_policy`
**Assertions**: The five shared assertions
---
### 4.3 Digest Enforcement Scenarios
---
#### Scenario 4.3.1: A Digest Computed Over Other Bytes
**Description**: A perfectly valid bundle whose `expected_digest` belongs to a different file
**Expected Behavior**: Fails at `download`, before any header is parsed
**Assertions**:
- Terminal status, step, code and catalog sentence
- The failure names the digest, not the magic or the version
- `run_name` is still null and `experiment_name` still the placeholder — the bundle was never opened
---
#### Scenario 4.3.2: The Object Is Replaced After Upload
**Description**: The same key, different bytes — also a valid bundle, so nothing but the digest can
tell
**Expected Behavior**: Fails at `download`
**Assertions**: As 4.3.1; only a real object store can show this at all
---
#### Scenario 4.3.3: No Digest at All
**Description**: `expected_digest` absent from the input
**Expected Behavior**: Fails at `received`, before anything is downloaded
**Assertions**: The request-invalid code, and the work directory was never created
---
### 4.4 Provisioning Failure Scenarios
Each asserts the platform terminal status `IMPORT_ERROR` — not the validation status, because a
provisioning failure is not the user's to fix — its own step and code, the workflow identifier in
the detail JSON, and `error_message` **equal** to the catalog sentence.
---
#### Scenario 4.4.1: The Tracking Server Is Stopped
**Description**: The `mlflow server` process is genuinely terminated for the duration
**Expected Behavior**: Fails at `experiment_creation`
**Assertions**:
- The shared assertions
- The names are already on the row: the bundle was opened before MLflow was reached
- Nothing in Mongo; nothing provisioned
---
#### Scenario 4.4.2: Artifact Upload Fails
**Description**: `log_artifacts` is patched — the one step where neither hostile input nor a broken
dependency can isolate the failure
**Expected Behavior**: Fails at `artifact_upload`
**Assertions**:
- The shared assertions
- The experiment exists (the previous step succeeded); no version, no document
---
#### Scenario 4.4.3: Registration Fails
**Description**: `create_model_version` is patched. MLflow rejects only an empty registered-model
name, and one process serves both tracking and the registry, so nothing else isolates this step
**Expected Behavior**: Fails at `registration`
**Assertions**:
- The shared assertions
- The run and both artifact paths survive — "the files were stored, the model was not registered"
---
#### Scenario 4.4.4: The Model Document Is Refused
**Description**: A real unique index on `model_config.target` refuses the activity's own insert.
(A duplicate `id` cannot be constructed: the activity derives `max(int(id)) + 1` from the same
collection, so the id it computes can never already exist)
**Expected Behavior**: Fails at `model_document`
**Assertions**:
- The shared assertions
- One registry version exists — registered, and unusable
---
### 4.5 Status Record Scenarios
---
#### Scenario 4.5.1: The Claim Precedes Every Gate
**Description**: The row is observed from inside `stat_object`, the first call gate 1 makes
**Assertions**: Running status and `import.workflow_id` already present at that moment
---
#### Scenario 4.5.2: The Names Are Visible Mid-Flight
**Description**: The row is observed from inside the first call experiment creation makes, during an
import that then succeeds
**Assertions**: `experiment_name` and `run_name` already populated before anything exists in MLflow
---
#### Scenario 4.5.3: The Inserting Side's Columns Are Never Rewritten
**Description**: After a success and after a failure
**Assertions**: `username`, `request_data`, `bucket_name`, `file_name`, `run_type`, `created_at`
unchanged; `status` and `updated_at` did change
---
#### Scenario 4.5.4: Unclaimable Rows
**Description**: Three variants — an unknown id, a `run_type='TRAIN'` row, and a row already held by
another workflow id
**Expected Behavior**: The Temporal run ends failed with the not-claimable error type, and **nothing
is written to any row**
**Assertions**: The error type from the history, the whole table byte-identical, nothing provisioned
---
#### Scenario 4.5.5: The Row Names a Different Object
**Description**: `file_name` disagrees with the input's `object_key`
**Expected Behavior**: Fails at `received` **with the error recorded on the row** — unlike 4.5.4,
this row was successfully claimed
**Assertions**: The received code and sentence on the row; nothing provisioned
---
#### Scenario 4.5.6: Constraint Conformance
**Description**: The real constraints, exercised directly
**Assertions**:
- Every status a normal import writes is accepted on an `IMPORT` row
- All four terminal values are refused across run types, naming
`chk_experiment_run_terminal_status_run_type`; the shared stage values are accepted on both
- A training row still walks its own vocabulary
- The table has exactly the 13 columns `V1`…`V12` leave (no `ALTER TABLE` by this feature)
---
### 4.6 Absence Scenarios
---
#### Scenario 4.6.1: One Table, One Database
**Description**: A full import bracketed by marker statements on both Postgres servers, read from
`log_statement=all`
**Assertions**:
- Every statement touching a user relation names `experiment_run`
- No `project`, `models`, `import_run`, `model_project` or `model_registry` reference (with quoted
literals stripped, so the bundle's own `model_project` *value* is not mistaken for a table)
- The predictions database logs **no** statement at all, though the import holds working credentials
for it
- A rejected import — which writes more status rows than a successful one — behaves the same
---
### 4.7 Cleanup, Retention and History Scenarios
---
#### Scenario 4.7.1: The Worker Filesystem Is Clean
**Description**: After a success and after rejections at two different steps (decryption and
structure validation, which unwind on different code paths)
**Assertions**: No extracted tree, no downloaded object, no temporary archive
---
#### Scenario 4.7.2: The Retention Rule Is Present
**Assertions**: `sientia-import-bundle-expiry` on the import bucket, enabled, filtered on
`imported_models/`, expiring in 7 days — and the uploaded object still there, because the import
never deletes it
---
#### Scenario 4.7.3: The Record Outlives the Object
**Description**: The uploaded object is deleted, as retention will eventually do
**Assertions**: The row still reads `IMPORT_SUCCESS` with its names
---
#### Scenario 4.7.4: No Plaintext Secret in the Event History
**Assertions**:
- The bundle password, the base64 envelope key and the key's hex form appear nowhere in the history
- `password_envelope` **is** in the history, so the negative assertions above cannot pass vacuously
---
### 4.8 Retry Envelope and Missing-`V12` Scenarios
These run on a **real clock** (`WorkflowEnvironment.start_local()`) with the status retry intervals
shortened; the production values stay asserted by the unit suite. All are marked `slow`.
---
#### Scenario 4.8.1: A Database That Comes Back
**Description**: The status database is stopped for two seconds after provisioning and restarted
inside the retry envelope
**Assertions**: The terminal write lands `IMPORT_SUCCESS` with the correct workflow id
---
#### Scenario 4.8.2: A Broken Progress Update
**Description**: A brief early outage that only the progress hints see
**Assertions**: The import completes with the correct verdict and does not spend the terminal
write's envelope on a hint nobody reads
---
#### Scenario 4.8.3: A Database That Stays Down After a Successful Import
**Description**: The database is taken away as soon as the row is claimed and never comes back
**Expected Behavior**: The model is fully provisioned and the workflow **completes** — reporting
failure would tell somebody to re-import a model that is already on the platform
**Assertions**:
- The run returns its result; the model, version and document all exist
- The failure channel carries `IMPORT_STATUS_NOT_RECORDED`, `step=status_write` and
`verdict_not_recorded=`
- The row is left in the running status
---
#### Scenario 4.8.4: A Failed Import Whose Failure Cannot Be Written Either
**Description**: The import fails at `artifact_upload` and the terminal write is then refused
**Assertions**:
- The reported cause is the artifact-upload failure, not the status write
- The status write reports itself only through the channel
- The row is left running
---
#### Scenario 4.8.5: A Table Without `V12`
**Description**: The evidence for why the migration gates enabling the import worker
**Expected Behavior**: The claim succeeds (`V11` gave the table `run_type`; `PENDING` and the running
status are pre-`V12` values), the model is fully provisioned, and only the terminal value is new — so
the last write of the import is the first thing that fails, non-retryably
**Assertions**:
- The run completes; the model, version and document all exist
- The row is left in the running status, with both names filled
- The channel carries the not-recorded code and the verdict
- Elapsed time shows the refusal was **not** retried
- `pg_constraint` confirms the fixture's `chk_experiment_run_status` really does not allow
`IMPORT_SUCCESS`, so the test cannot pass vacuously
---
## 5. joblib Flavor Scenarios
`joblib` is an explicit model flavor, declared in `model_config.predict_flavor` /
`model_config.transform_flavor` like any other. It reads the artifact directly — download the
`.pkl`, put the artifact's `code/` on `sys.path`, `joblib.load` — and **never** calls an MLflow
flavor loader. There is no fallback in either direction: a failing loader propagates.
Covered by `e2e/test_predictions_batch_joblib_flavor.py` (read) and
`e2e/test_retrain_joblib_flavor.py` (write). Both use the `joblib_flavor_artifacts` fixture, which
builds one artifact directory per model type holding a real compressed `model.pkl` plus a `code/`
directory carrying the module where the model's class lives. The fixture takes that directory off
`sys.path` again before yielding, so the pickle is **unloadable** until the flavor puts it back —
that is what makes these tests prove module resolution rather than assume it. The fixture also
restores `sys.path` in teardown, because the flavor deliberately never removes what it inserted.
### 5.1 Read Scenarios
---
#### Scenario 5.1.1: Predict Through the joblib Flavor
**Description**: `predict_flavor: 'joblib'` on a normal `predictions_batch` run
**Input**:
- `model_config.predict_flavor = 'joblib'`, `transform_flavor = 'sklearn'`
- `client.download_artifacts` serving the real `prediction_model` artifact
**Expected Behavior**:
- The artifact is downloaded once and deserialized with `joblib.load`
- No `models:/<name>/production` URI is built, so no flavor loader ever sees this model
- The prediction the artifact's own object produces is what lands in Postgres
**Assertions**:
- Exactly one prediction row, `prediction_status='Good'`, value matching the artifact's model
- `download_artifacts` called once, with `'prediction_model'`
- No `mlflow.{sklearn,pyfunc,pytorch}.load_model` call names this model
---
#### Scenario 5.1.2: Transform Through the joblib Flavor
**Description**: `transform_flavor: 'joblib'`, which had no E2E coverage before
**Input**:
- `model_config.transform_flavor = 'joblib'`, `predict_flavor = 'sklearn'`
- `client.download_artifacts` serving the real `data_model` artifact
**Expected Behavior**:
- `data_model` is downloaded and deserialized; the prediction model stays on its MLflow flavor
- No artifact URI is resolved for the transform, so `mlflow.get_run` is never called
**Assertions**:
- The prediction lands with the mocked prediction model's value
- `download_artifacts` called once, with `'data_model'`
- `mlflow.get_run` not called; no flavor loader was given a `/data_model` URI
---
#### Scenario 5.1.3: An MLflow Flavor Is Unaffected
**Description**: A model declaring `sklearn` behaves exactly as before
**Assertions**:
- The prediction lands with the mocked model's value
- `download_artifacts` NOT called — the joblib path is never entered as a safety net
---
### 5.2 Error Scenarios
---
#### Scenario 5.2.1: joblib Flavor Failure Has a Single Cause
**Description**: The downloaded artifact has no top-level `.pkl`
**Expected Behavior**: The workflow stops. The reported cause is the missing pickle and nothing
else — with no fallback there is no second exception chained to a first
**Assertions**:
- The run stops (`assert_stop`)
- The log carries `No .pkl file found in prediction_model artifact`
- The log does **not** carry `Original error`, the marker of a chained cause
---
### 5.3 Write Scenarios
---
#### Scenario 5.3.1: Retraining a joblib Model Round-Trips
**Description**: `retrain_model` on a model whose `predict_flavor` and `transform_flavor` are both
`joblib` — write and read in one test
**Input**:
- Training frame indexed by `DATETIME_FORMAT_WITH_TZ` timestamps, carrying the target column
- MLflow's artifact I/O replaced by a fake store that copies into an inspectable run directory
**Expected Behavior**:
- Both models load through the joblib flavor; no flavor loader runs
- `log_model` writes `prediction_model/model.pkl` (a `joblib.dump(..., compress=3)` pickle) and
re-logs `prediction_model/code/`, without an `MLmodel` manifest
- Loading the **written** artifact through the joblib flavor deserializes it and predicts, after
the module the first load imported has been dropped from `sys.modules`
**Assertions**:
- `retrain_model` returns `success: True`
- `download_artifacts` was asked for both `data_model` and `prediction_model`
- `sklearn`/`pyfunc` loaders have zero calls
- The written run has `prediction_model/model.pkl`, `prediction_model/code/<module>.py` and
`data_model/model.pkl`, and **no** `prediction_model/MLmodel`
- The reloaded model's prediction equals the original's, and the written `code/` is on `sys.path`
---