gitea_admin/sientia-dataops-laborious-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

QTZPOC-20

Browse Source 
QTZPOC-20: Implement explicit joblib model flavor and enhance model import workflow with extensive E2E tests.

This commit introduces a first-class joblib model flavor for direct loading and saving of Python pickle artifacts, including custom code within the artifact's code/ directory, bypassing MLflow's flavor loaders. The model import workflow's status database configuration is now streamlined to default to the POSTGRES_* server, with explicit IMPORT_STATUS_DB_* overrides.

Comprehensive end-to-end tests have been added for the model import workflow, covering success paths, gate rejections, digest enforcement, provisioning failures, status record integrity, cleanup, and retry/recovery scenarios using a real-clock environment. New E2
This commit is contained in:
Vitor Pimentel dos Santos
2026-08-26 15:45:33 -03:00
parent dac4f1718f
commit e95e65b72b
32 changed files with 4247 additions and 1040 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
docs/model-import.md
View File

@@ -88,14 +88,18 @@ is at fault.
| `IMPORT_MAX_UNCOMPRESSED_BYTES` | 4 GiB | gate 4 |
| `IMPORT_MAX_COMPRESSION_RATIO` | `200` | gate 4 |
| `IMPORT_MODELS_COLLECTION` | `models` | the MongoDB collection that *is* the model listing |
| `IMPORT_STATUS_DB_HOST` | `localhost` | the BFF database holding the import log |
| `IMPORT_STATUS_DB_PORT` | `5432` | |
| `IMPORT_STATUS_DB_NAME` | `sientia-core-mlops-bff` | |
| `IMPORT_STATUS_DB_USER` | `mlops_bff_user` | |
| `IMPORT_STATUS_DB_PASSWORD` | empty | |
| `IMPORT_STATUS_DB_NAME` | `sientia-core-mlops-bff` | the BFF database holding the import log |
| `IMPORT_STATUS_DB_HOST` | `POSTGRES_HOST` | override: the log is on another server |
| `IMPORT_STATUS_DB_PORT` | `POSTGRES_PORT` | override |
| `IMPORT_STATUS_DB_USER` | `POSTGRES_USER` | override |
| `IMPORT_STATUS_DB_PASSWORD` | `POSTGRES_PASSWORD` | override |
`IMPORT_STATUS_DB_*` is a **second** Postgres connection, independent of the one Laborious already
holds for predictions. The import writes to that database and no other, and to one table in it.
The import log is a **second connection but the same server**: host, port, user and password come
from `POSTGRES_*`, and only the database name is import configuration, because the log lives in the
BFF's database rather than in the one Laborious holds for predictions. The import writes to that
database and no other, and to one table in it. The four `IMPORT_STATUS_DB_*` connection variables
are overrides for a deployment whose log genuinely sits elsewhere — the e2e suite, which runs the
two databases as two containers, is exactly that case.
A missing or malformed `IMPORT_PASSWORD_KEY` is an import failure at the decryption step, not a boot
failure: the worker starts either way.
@@ -324,10 +328,11 @@ A `COMPLETED` import is **not** a servable model. Stated plainly:
So an operator must promote the version to `Production` and set `active: true`. This workflow never
promotes and never activates — that runbook is QTZPOC-21's.
One more thing to know before serving an imported model: `predict_flavor: 'joblib'` is a **read-only**
flavor. It is what the artifact actually is, and it is enough to load and predict, but nothing in this
platform writes that flavor — so an imported model cannot be retrained until a write flavor is
configured for it.
One more thing to know before serving an imported model: `predict_flavor: 'joblib'` is what the
artifact actually is, and it loads, predicts **and** retrains — `log_model` writes that flavor too,
as `model.pkl` + `code/`. The artifact it writes carries no `MLmodel` manifest, so an imported model
stays readable only by this runtime, before and after a retrain. Full contract:
[`model-flavors.md`](model-flavors.md).
## 14. Retry, and what does not retry
@@ -347,3 +352,35 @@ change.
There is no rollback, no compensation, no resume and no reconciliation. A failed import stays failed;
trying again is a new upload and a new row.
## 15. How this contract is verified
Two suites, and they prove different things.
**Unit** — `pytest tests/`, part of the standard gate (`validate --project-name=laborious`). Covers
the bundle reader against committed golden bytes, every gate, the failure catalog, the status SQL and
the workflow's ordering, all against fakes.
**End to end** — `pytest e2e -m import_e2e`, opt-in and outside `testpaths`. Runs the real workflow on
a real worker against real MinIO, a real Postgres holding this document's `public.experiment_run`
(with and without `V12`), a real MongoDB with the unique index on `models.id`, and a real
`mlflow server`. Nothing on the import path is mocked. This is what proves the claims a fake cannot:
that the `CHECK` constraints of § 6 accept every status written, that the registry source of § 11
resolves through the existing loader, that the `V12` failure shape of § 7 is what it says, and that no
plaintext password reaches the Temporal event history.
Requires Docker plus one one-time setup per machine:
```bash
echo "ryuk.disabled=true" >> ~/.testcontainers.properties
```
Without it every e2e test fails in fixture setup with a Docker mount error — see
[`README.md` § End-to-End Tests](../README.md#end-to-end-tests-e2e) for why and what it costs.
Scenario catalogue: [`e2e/scenarios.md`](../e2e/scenarios.md) § 4.
Requirement-to-test mapping: `openspec/changes/import-model-e2e-suite/traceability.md`.
Doing one import by hand, standing in for the frontend that does not exist yet:
[`scripts/import_model_manual_run.py`](../scripts/import_model_manual_run.py) — see
[`README.md` § Running One Model Import by Hand](../README.md#running-one-model-import-by-hand).