gitea_admin/sientia-dataops-laborious-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

QTZPOC-20

Browse Source 
QTZPOC-20: Implement model import workflow for .sientia bundles

Introduces the import_model workflow and its associated activities to enable importing .sientia bundles into MLflow and MongoDB. This includes:

*   New core workflow and activity logic for handling bundle download, decryption, validation, MLflow provisioning (experiments, runs, registered versions), and MongoDB model document creation.
*   Comprehensive E2E and unit test suites, including a dedicated test harness and a manual run script for development.
*   Updated .env.example and README.md with new configurations and usage instructions.
*   Ensures correct MLflow model source resolution for legacy loader compatibility and robust MongoDB document management.
*   Refactors existing utility scripts into a scripts/ directory.
This commit is contained in:
Vitor Pimentel dos Santos
2026-08-25 17:07:42 -03:00
parent 0e445de386
commit dac4f1718f
41 changed files with 8168 additions and 1045 deletions
Download Patch File Download Diff File Expand all files Collapse all files

349
docs/model-import.md Normal file
View File

@@ -0,0 +1,349 @@
# Importing a model (`import_model`)
The `import_model` workflow turns an encrypted `.sientia` object in MinIO into a registered MLflow
model plus one document in the MongoDB `models` collection — or into a row that says, in a sentence a
person can read, why it did not.
This document is the contract for the two sides that talk to it: the **frontend**, which uploads the
object and builds the request, and the **BFF**, which owns the import log row. Everything here is
enforced by code in this repository; where a statement is about a column or a constraint, the
authority is the migration named in §7.
Format of the file itself: [`sientia-bundle-format.md`](sientia-bundle-format.md).
---
## 1. What the caller does, in order
1. **Upload** the `.sientia` object to the import bucket, under the configured prefix (default
`imported_models/`). The key must end in `.sientia`.
2. **Compute `expected_digest`** — the SHA-256, lowercase hex, of the **exact bytes that were
uploaded**. Not of a re-read, not of a re-serialisation: of what went over the wire.
3. **Encrypt the bundle password** into a `password_envelope` (§3).
4. **Insert the import log row** (§5) and take its `id`.
5. **Start the workflow** on `import_model-{RUNTIME}-queue` with the input of §2.
The workflow claims the row, opens the bundle, provisions MLflow, writes the model document, cleans
up and records one terminal verdict. It never inserts a row, never promotes a version and never
activates a model.
## 2. The workflow input
```json
{
"import_run_id": 41,
"object_key": "imported_models/sales_forecast_v3.sientia",
"expected_digest": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
"password_envelope": "Base64OfNonceCiphertextTag==",
"bucket": "sientia"
}
```
| Field | Required | Meaning |
|---|---|---|
| `import_run_id` | yes | primary key of the `public.experiment_run` row the BFF inserted. Must be a positive integer |
| `object_key` | yes | the uploaded object's key. Must match the row's `file_name` when the row carries one |
| `expected_digest` | **yes** | SHA-256 of the uploaded bytes, 64 lowercase hex characters. There is no path that skips the comparison: a missing value fails at `received`, a mismatching one at `download`, before the header is parsed |
| `password_envelope` | yes | the bundle password, encrypted (§3). A plaintext `password` field is **refused** |
| `bucket` | no | defaults to `IMPORT_BUNDLE_BUCKET` |
**Fields that are refused rather than ignored**: `project_id`, `project_name`,
`provisioning_target`, `schema`, `table_name`, `status_table`, `password`. An input carrying any of
them fails at `received`, so no caller can believe it recorded something the import does not record
(see §10).
## 3. The password envelope
AES-256-GCM, and the ciphertext layout is what the WebCrypto API returns natively:
```
envelope = base64( nonce(12 bytes) || ciphertext || tag(16 bytes) )
```
`crypto.subtle.encrypt({name: 'AES-GCM', iv: nonce}, key, plaintext)` returns `ciphertext || tag`
already concatenated, so the browser side is: generate 12 random bytes, encrypt, concatenate the
nonce in front, base64 the result.
The key is a 32-byte AES-256 key shared with the worker through `IMPORT_PASSWORD_KEY` (base64). The
worker decrypts the envelope **inside the activity that uses the password**, so no plaintext password
is ever an activity input, an activity result or a workflow variable — and therefore never appears in
the Temporal event history. A test asserts that for every failure path.
A failure to open the envelope is recorded as a `decryption` failure, distinct in the logs from a
wrong *bundle* password but sharing its step, because the step says where the import stopped, not who
is at fault.
## 4. Environment
| Variable | Default | What it is |
|---|---|---|
| `IMPORT_MODEL_ENABLED` | unset | registers the fifth (import) worker. Truthy values: `1`, `true`, `yes`, `on` |
| `IMPORT_PASSWORD_KEY` | — | base64 of the 32-byte AES-256-GCM key that wraps bundle passwords |
| `IMPORT_BUNDLE_BUCKET` | `sientia` | bucket holding the uploaded bundles |
| `IMPORT_BUNDLE_PREFIX` | `imported_models/` | key prefix the uploads must sit under |
| `IMPORT_BUNDLE_RETENTION_DAYS` | `7` | lifecycle expiry the importer applies to that prefix |
| `IMPORT_WORK_DIR` | `<tempdir>/sientia-import` | worker scratch directory |
| `IMPORT_MAX_OBJECT_BYTES` | 1 GiB | gate 1 |
| `IMPORT_MAX_ARCHIVE_ENTRIES` | `5000` | gate 4 |
| `IMPORT_MAX_UNCOMPRESSED_BYTES` | 4 GiB | gate 4 |
| `IMPORT_MAX_COMPRESSION_RATIO` | `200` | gate 4 |
| `IMPORT_MODELS_COLLECTION` | `models` | the MongoDB collection that *is* the model listing |
| `IMPORT_STATUS_DB_HOST` | `localhost` | the BFF database holding the import log |
| `IMPORT_STATUS_DB_PORT` | `5432` | |
| `IMPORT_STATUS_DB_NAME` | `sientia-core-mlops-bff` | |
| `IMPORT_STATUS_DB_USER` | `mlops_bff_user` | |
| `IMPORT_STATUS_DB_PASSWORD` | empty | |
`IMPORT_STATUS_DB_*` is a **second** Postgres connection, independent of the one Laborious already
holds for predictions. The import writes to that database and no other, and to one table in it.
A missing or malformed `IMPORT_PASSWORD_KEY` is an import failure at the decryption step, not a boot
failure: the worker starts either way.
## 5. The import log row — the BFF's half
The row is `public.experiment_run` in `sientia-core-mlops-bff`. **The workflow never inserts it.**
The inserting side writes:
| Column | Value |
|---|---|
| `run_type` | `'IMPORT'` |
| `status` | `'PENDING'` |
| `username` | whoever asked for the import |
| `bucket_name` | the bucket the object was uploaded to |
| `file_name` | the object key |
| `request_data` | the request payload. If it carries a digest under `expected_digest`, `digest`, `sha256` or `file_digest`, the workflow checks it against the input's |
Then it passes the returned `id` as `import_run_id`.
The workflow's first act is to **claim** that row:
```sql
UPDATE public.experiment_run
SET status = 'ORCHESTRATOR_WAITING_PROC', updated_at = :now,
orchestrator_response_data = :detail
WHERE id = :import_run_id
AND run_type = 'IMPORT'
AND (status = 'PENDING'
OR (status = 'ORCHESTRATOR_WAITING_PROC'
AND orchestrator_response_data -> 'import' ->> 'workflow_id' = :workflow_id))
```
The second disjunct makes a re-execution of the same workflow idempotent instead of a conflict. Every
later write keys on `id` alone, because ownership was settled here.
**Columns the workflow never writes**: `username`, `request_data`, `bucket_name`, `file_name`,
`run_type`, `created_at`. **Columns it does write**: `status`, `error_message`, `updated_at`,
`orchestrator_response_data`, and — as soon as the bundle is open, before anything exists in MLflow —
`experiment_name` (the bundle's own, verbatim) and `run_name` (`import-{model_name}-v{model_version}`,
truncated inside the name segment to fit 50 characters). Those two are **overwritten** by the
workflow; whatever the inserting side put there is replaced.
### When the row cannot be claimed
If the claim matches nothing — no such id, a `run_type='TRAIN'` row, or a row already running under
another workflow — the import **writes nothing anywhere**, because the row belongs to someone else.
Instead it:
- fails the Temporal run with error type `ImportLogRowNotClaimableError` and code
`IMPORT_LOG_ROW_NOT_CLAIMABLE`, non-retryable;
- increments the import error metric and fires a notification;
- logs the id and the observed state (which of the three causes it was).
So a row left `PENDING` with no `workflow_id` in its detail JSON means the workflow never owned it.
**Closing that row out belongs to whichever side inserted it** — by timeout, by a sweeper, by a person
— however that side chooses. The importer deliberately does not guess.
## 6. Status vocabulary
| Status | Written when | New in `V12`? |
|---|---|---|
| `ORCHESTRATOR_WAITING_PROC` | claimed, and while running | no — **reused** from the training flow |
| `ORCHESTRATOR_VALIDATION_ERROR` | failure at `received` … `content_policy` — the user can act on it | no — **reused** |
| `IMPORT_SUCCESS` | every step succeeded | yes |
| `IMPORT_ERROR` | failure at `experiment_creation` … `model_document`, or unclassified — only support can act on it | yes |
The two reused values describe a *stage*, not a flow: what says the row is an import is `run_type`,
not the status. There is deliberately no `IMPORT_RUNNING`; it would carry no information `run_type`
does not already carry, and it would widen both the pairing `CHECK` and the frontend's switch for
nothing.
`error_message` is the user-facing sentence, taken from the catalog in §8. On success it is written as
`NULL`, never `''` — the column carries a `LENGTH(error_message) >= 3` check.
## 7. The migration this depends on
`sientia-dataops-database-migrations`,
`postgres/sientia-core-mlops-bff/migrations/V12__add_import_statuses_to_experiment_run.sql`:
- adds `IMPORT_SUCCESS` and `IMPORT_ERROR` to `chk_experiment_run_status`;
- adds `chk_experiment_run_terminal_status_run_type`, pairing only the **terminal** values with their
`run_type` (`IMPORT_*` on `'IMPORT'`, `TRAINING_*` on `'TRAIN'`) and leaving the shared
`ORCHESTRATOR_*` values out of the rule;
- **adds no columns at all.**
`V12` must be merged and applied before `IMPORT_MODEL_ENABLED` is set anywhere. Without it, an import
provisions the model and then cannot record its verdict: the claim still succeeds (no new columns are
needed for it), the old `CHECK` refuses `IMPORT_SUCCESS`, and the failure surfaces as
`IMPORT_STATUS_NOT_RECORDED` on the first attempt — no retry wait — leaving the row in
`ORCHESTRATOR_WAITING_PROC` with a fully provisioned model behind it.
## 8. `orchestrator_response_data`, and the error catalog
There is no column for the error code and none for the workflow identifier (decision D21). Both live
at fixed keys in the JSON column the workflow already owns, and the whole `import` object is rewritten
on every write, so the identifier the claim wrote is never dropped:
```json
{
"import": {
"workflow_id": "import_model-41-01HW…",
"code": "IMPORT_BUNDLE_INCOMPLETE",
"step": "structure_validation",
"gate": 6,
"source": { "model_name": "sales_forecast", "experiment_name": "…", "…": "…" },
"cleanup_failed": false
}
}
```
- `workflow_id` — what a user quotes to support, and what the claim's idempotency disjunct reads.
- `code` — the frontend's i18n key. **Absent on success.**
- `step` — where the import stopped (the values of §9).
- `gate` — which of the seven gates rejected it, when one did.
- `source` — the bundle's own metadata block, once it is known.
- `cleanup_failed` — temporary files were left on the worker. Present on a **successful** import too:
it never changes the status, because the user's model was imported.
### Code → reason
`error_message` is exactly the sentence in this table; nothing is interpolated into it, and the
exception's text — paths, table names, exception classes, tracebacks — reaches only the logs.
| Code | Step | Sentence written to `error_message` |
|---|---|---|
| `IMPORT_REQUEST_INVALID` | `received` | The import could not be started because the request was incomplete or contradictory. Nothing was created. Submit the import again. |
| `IMPORT_FILE_UNREADABLE` | `download` | The uploaded file could not be read, or it is not the file that was sent. Upload it again. |
| `IMPORT_BUNDLE_CANNOT_BE_OPENED` | `decryption` | This file could not be opened: the password is wrong, or the file is damaged or was altered. Confirm the password with whoever exported the model, then try again. |
| `IMPORT_BUNDLE_REJECTED` | `archive_inspection` | The file's contents did not pass the platform's safety checks, so it was not opened. Export the model again from the origin platform. |
| `IMPORT_BUNDLE_UNPACK_FAILED` | `extraction` | The file could not be unpacked safely and nothing from it was kept. Export the model again from the origin platform. |
| `IMPORT_BUNDLE_INCOMPLETE` | `structure_validation` | This file is not a complete model export — part of what the platform needs is missing from it. Export the model again from the origin platform. |
| `IMPORT_BUNDLE_UNEXPECTED_CONTENT` | `content_policy` | The file contains something a model export should not contain, so it was rejected. Export the model again from the origin platform. |
| `IMPORT_STORAGE_PREPARATION_FAILED` | `experiment_creation` | The platform could not prepare a place to keep this model. The import stopped and no model was created. Contact support with this import's identifier. |
| `IMPORT_MODEL_FILES_NOT_STORED` | `artifact_upload` | The model's files could not be stored on this platform. The import stopped and the model is not available. Contact support with this import's identifier. |
| `IMPORT_MODEL_NOT_REGISTERED` | `registration` | The model's files were stored, but the model itself could not be registered, so it cannot be used. Contact support with this import's identifier. |
| `IMPORT_MODEL_SETTINGS_NOT_SAVED` | `model_document` | The model was registered, but the settings that tell the platform how to run it could not be saved. Contact support with this import's identifier. |
| `IMPORT_UNEXPECTED_ERROR` | — | The import stopped for an unexpected reason and no model was created. Contact support with this import's identifier. |
Three codes never reach a row's `error_message`, because there is no row to write or no failure to
report to a user. They exist in the logs, the metric and the notification only:
| Code | When |
|---|---|
| `IMPORT_LOG_ROW_NOT_CLAIMABLE` | the row is not this workflow's (§5) |
| `IMPORT_STATUS_NOT_RECORDED` | the verdict itself could not be written; the log line carries the verdict it could not record |
| `IMPORT_CLEANUP_INCOMPLETE` | temporary files were left behind; also flagged as `cleanup_failed` in the detail |
## 9. Steps
`received`, `download`, `decryption`, `archive_inspection`, `extraction`, `structure_validation`,
`content_policy`, `experiment_creation`, `artifact_upload`, `registration`, `model_document`,
`cleanup`. The first seven are the gates (see the format document); a failure in any of them is a
`ORCHESTRATOR_VALIDATION_ERROR`. The next four are provisioning; a failure there is an
`IMPORT_ERROR`. `cleanup` is never a reported failure.
## 10. The import knows nothing about projects
The model listing **is** the MongoDB `models` document. No migrated relational database in this
platform holds a project table or a model table, nothing in this repository reads a project-to-model
link, and the workflow input carries no project target — which is why naming one is refused rather
than accepted and ignored. Whatever the import screen does with projects is the frontend's business.
The bundle's own `metadata.model_project` is logged as an MLflow parameter (`source.model_project`),
as a record of where the model came from. It is not a link.
## 11. What MLflow ends up holding
| Thing | Value |
|---|---|
| Experiment | `metadata.experiment_name` from the bundle, created if absent, reused if present |
| Run | always **new**, created with the `run_name` already written on the record |
| Artifacts | the bundle's whole `artifacts/` tree, with `prediction_model` and `data_model` at the run's artifact root |
| Parameters | the bundle's `parameters`, verbatim, plus the origin metadata as `source.*` (including `source.run_id`, which is never dereferenced) |
| Registered version | created on demand; source is the run's **resolved artifact URI** with `/prediction_model` appended, e.g. `mlflow-artifacts:/17/<run_id>/artifacts/prediction_model` |
| Stage | **untouched** |
The source shape matters and is easy to get wrong: the legacy loader extracts the run id as
`source.split('/')[2]`, which is the run id in the resolved artifact URI but the *artifact path* in a
literal `runs:/<run_id>/prediction_model`. Registering with the unresolved URI breaks loading
silently — nothing raises, the loader just returns the wrong string. This is the same value
`mlflow.register_model` stores for the training flow, because it resolves the `runs:/` URI before
writing it; `MlflowClient.create_model_version` does not, so the import resolves it itself.
## 12. The model document
Written **last**, into the collection named by `IMPORT_MODELS_COLLECTION` (default `models`):
```json
{
"id": "31",
"name": "sales_forecast",
"active": false,
"model_config": {
"transform_flavor": "sklearn",
"predict_flavor": "joblib",
"retention_minutes": 60,
"target": "SE"
}
}
```
- `id` is `str(max(int(id) for id in collection) + 1)` — computed in Python, from the ids that are
numbers. Never a `count`, never a lexicographic sort (which would answer `"9" > "30"`). Ids that
are not decimal numbers are ignored for the maximum and counted in the log.
- `name` is `metadata.model_name` from the bundle, verbatim.
- `active` is `false`, always. No code path in the import writes or updates a document with
`active: true`.
- `target` comes from `parameters.target_variable`; nothing else supplies it.
- The `models` collection carries a **unique sparse index on `id`**, created by the MongoDB baseline
migration. A duplicate-key insert is reported as a `model_document` failure — no retry, no id
recomputation.
**A name already in the collection is left exactly as it is.** The import writes nothing, modifies
nothing, logs the skip and succeeds: `active`, `target` and the flavors of a model already on the
platform are operator-owned.
## 13. After a successful import — the manual steps
A `COMPLETED` import is **not** a servable model. Stated plainly:
- the registered version is **unstaged**, and `load_predict_model` only sees a version once it is in
stage `Production`;
- the model document is **inactive** (`active: false`).
So an operator must promote the version to `Production` and set `active: true`. This workflow never
promotes and never activates — that runbook is QTZPOC-21's.
One more thing to know before serving an imported model: `predict_flavor: 'joblib'` is a **read-only**
flavor. It is what the artifact actually is, and it is enough to load and predict, but nothing in this
platform writes that flavor — so an imported model cannot be retrained until a write flavor is
configured for it.
## 14. Retry, and what does not retry
Only the status writes retry, and only because `UPDATE ... SET <fixed values> WHERE id` leaves the row
the same however many times it runs:
| Call | Attempts | Why |
|---|---|---|
| claim, terminal write | 10, 1 s → 30 s backoff (≈151 s envelope) | the verdict must land; the database being briefly unavailable is exactly what a retry fixes |
| progress writes, names | 3, capped at 2 s | a hint nobody reads — the terminal write rewrites it — must not stall a working import |
| everything else | 1 | a repeated registry call or Mongo insert can write twice |
`ImportLogRowNotClaimableError` and `ImportStatusRejectedError` are non-retryable: the first would
read the same rows and reach the same conclusion, and the second means the statement itself was
refused (a `CHECK` violation, an undefined column, a value too long), which a second attempt cannot
change.
There is no rollback, no compensation, no resume and no reconciliation. A failed import stays failed;
trying again is a new upload and a new row.

196
docs/sientia-bundle-format.md Normal file
View File

@@ -0,0 +1,196 @@
# The `.sientia` bundle format
A `.sientia` file is one exported MLflow model: its whole artifact tree, its run parameters and six
fields of origin metadata, zipped and then encrypted under a password the exporter chose. It is
written by the Streamlit platform (`sientia-projects-templates`) and read by this repository's
importer. Two programs in two repositories agree on these bytes, so the constants below are
duplicated by necessity — the drift is held in place by tests on both sides, not by a shared library.
This document covers both directions: **§1–§3 how a bundle is generated**, so the producer can be
audited against it, and **§4–§7 how it is read**, so a rejection can be explained.
Normative implementations:
| Side | Where |
|---|---|
| Writer | `sientia-projects-templates`, `app/src/operations/model_export/{crypto,bundle,exporter}.py` |
| Reader | this repository, `laborious/utils/bundle/{format,reader}.py` |
| Test writer | this repository, `tests/helpers/bundle_factory.py` — mirrors the producer field for field, for building hostile variants |
| Golden fixture | `tests/fixtures/bundle/` — a real producer artefact, with its password and producer commit recorded in that directory's `README.md` |
---
## 1. Generation, step by step
Entry point: `exporter.export_model_bundle(*, client, run_id, model_name, experiment_name,
model_version, model_project, password) -> bytes`.
1. **Password strength** — `crypto.validate_password_strength`: at least 12 characters, one
uppercase, one lowercase, one digit, one symbol. **Export-side only.** The importer cannot
re-check it and does not try: by the time a bundle exists, the password is whatever it was.
2. **Exportability** — `bundle.validate_run_is_exportable`: the run's top-level artifacts must
include **both** `data_model` and `prediction_model`, checked with `client.list_artifacts(run_id)`
before anything is downloaded.
3. **Whole-tree download** — `client.download_artifacts(run_id, '', staging)` into a
`tempfile.mkdtemp(prefix='sientia_export_')` directory. The entire artifact tree, not a subset.
4. **Run parameters** — `dict(run.data.params)`, copied **verbatim**. Whatever the run logged is
what the bundle carries, including `target_variable` (see §3).
5. **Metadata document** — `bundle.build_metadata_json`, exactly two top-level keys:
- `parameters`: the run parameters, verbatim;
- `metadata`: `model_name`, `experiment_name`, `run_id`, `model_version`, `model_project` and
`export_timestamp = datetime.now(UTC).isoformat()`. Six fields, no others.
6. **Zip** — `bundle.build_zip`, `zipfile.ZIP_DEFLATED`. `metadata.json` is written **first**, with
`json.dumps(...)` and no indent. Every file under the staging directory follows as
`artifacts/<relpath>`, with `\` replaced by `/`. **No directory entries are stored**, so a reader
must derive directories from file paths.
7. **Encryption** — `crypto.encrypt_archive`: a fresh 16-byte `os.urandom` salt, Argon2id key
derivation, then a libsodium `secretstream` push in 64 KiB plaintext chunks, the last chunk tagged
`FINAL` and the rest `MESSAGE`.
8. **Download name** — `{model_name}_v{model_version}.sientia`. The staging directory is removed in a
`finally`, and the password is popped from the Streamlit session state.
Producer pins: `pynacl==1.5.0`, `mlflow==2.10.1`.
## 2. Constants
Copied from `crypto.py` unless noted. The reader's names are the ones in
`laborious/utils/bundle/format.py`.
| Producer | Value | Reader |
|---|---|---|
| `MAGIC` | `b'SIENTIA1'` | `MAGIC` |
| `_HEADER_STRUCT` | `struct.Struct('>8sBBB16sQQB32s64s')` | `HEADER_STRUCT` |
| `HEADER_STRUCT_SIZE` | `140` | `HEADER_SIZE` |
| `FORMAT_VERSION` | `1` | `FORMAT_VERSION` |
| `AEAD_ID_XCHACHA20POLY1305_SECRETSTREAM` | `1` | same name |
| `KDF_ID_ARGON2ID` | `1` | same name |
| `SALT_SIZE` | `16` | `SALT_SIZE` |
| `KDF_MEMLIMIT_BYTES` | `268435456` (256 MiB) | `MAX_KDF_MEMLIMIT` — a **ceiling** on the reader's side |
| `KDF_OPSLIMIT` | `3` | `MAX_KDF_OPSLIMIT` — likewise a ceiling |
| `KDF_PARALLELISM_RESERVED` | `1` | `PARALLELISM` |
| `DIGEST_SIZE` | `32` | `DIGEST_SIZE` |
| `RESERVED_SIGNATURE_SIZE` | `64` | `RESERVED_SIGNATURE_SIZE` |
| `CHUNK_SIZE` | `65536` (64 KiB) | `CHUNK_SIZE` |
| KDF algorithm | `crypto_pwhash_ALG_ARGON2ID13` | `KDF_ALG` |
| Key length | `crypto_secretstream_xchacha20poly1305_KEYBYTES` (32) | `KEY_SIZE` |
Fixed by libsodium rather than by the producer, but needed to frame the stream:
- `crypto_secretstream_xchacha20poly1305_HEADERBYTES == 24` → `STREAM_HEADER_SIZE`
- `crypto_secretstream_xchacha20poly1305_ABYTES == 17` → `ABYTES`
- pre-upload prefix `140 + 24 + (65536 + 17) = 65717` → `PREFIX_SIZE`, the smallest number of leading
bytes from which a header and one full chunk can be checked
**Password normalisation.** `_derive_key` applies `unicodedata.normalize('NFC', password)` and
encodes UTF-8 — no trimming, no case folding, no padding. The reader normalises identically, so a
password typed in decomposed form opens a bundle written from the composed form.
## 3. `target_variable`, and where it does not come from
Gate 6 requires `parameters.target_variable`: it is the only source of `model_config.target` in the
model document, and every workflow that serves a model indexes that key directly. On the producer
side:
| Export path | Logs `target_variable`? | Evidence |
|---|---|---|
| Time-series training template | yes | `app/pages/template_01.py` L1212-1214 |
| Model manager, "Save Experiment" | yes, if the session carried it | `app/pages/model_manager.py` L791-796 re-logs `st.session_state.parameters` verbatim |
| Model ensemble | same condition | `app/pages/model_ensemble.py` L695-700 |
| Pipeline / timeseries template | **no** | `app/pages/timeseries_template.py` L1761-1762, L1811-1812 log `run_metadata` only |
| AutoML | **no** | `automl_logging_operations.py` L454-491; the target appears as `"target"` inside the model card JSON, never as a run parameter |
So a gate-6 rejection naming `target_variable` means the origin run came from the pipeline or AutoML
path. The remedy is to re-export from a run trained by the time-series template, which is what the
importer's `IMPORT_BUNDLE_INCOMPLETE` sentence already asks for.
---
## 4. The file on disk
```
offset width field constraint the reader enforces
------ ----- -------------------- ------------------------------------------------------
0 8 magic == b'SIENTIA1'
8 1 format_version == 1
9 1 aead_id == 1 (XChaCha20-Poly1305 secretstream)
10 1 kdf_id == 1 (Argon2id)
11 16 salt passed to the KDF as-is
27 8 kdf_memlimit_bytes <= 268435456
35 8 kdf_opslimit <= 3
43 1 kdf_parallelism == 1
44 32 ciphertext_digest == sha256(file[164:])
76 64 reserved_signature all zero
------ -----
140 24 secretstream header libsodium HEADERBYTES
164 ... chunk stream 64 KiB plaintext chunks, each +17 bytes of AEAD tag
```
All integers are big-endian; the format string starts with `>`, so there is no alignment padding and
the offsets above are exact.
**The digest boundary.** The producer computes `hashlib.sha256(ciphertext)` where `ciphertext` is the
concatenation of the *pushed chunks only*, and then writes `header + stream_header + ciphertext`. On
disk, therefore, the digest covers everything after offset **164**, not after 140:
> `ciphertext_digest == sha256(file[HEADER_SIZE + STREAM_HEADER_SIZE:])`
A reader that hashes from 140 will reject every valid bundle. The fixture test that recomputes both
values and asserts the first matches while the second does not is what pins this, and is the reason
the golden fixture exists.
**Version policy.** `format_version` is checked for equality, not for a range: a version this reader
does not know is refused at gate 2 rather than parsed optimistically. `reserved_signature` is
reserved for a future signature and must be all zero today, so a signed bundle is a new format
version and not a silent change of meaning.
## 5. The seven gates
`BundleReader.open()` runs them in this order, in one function, so the order cannot be changed by a
caller. Each raises `BundleGateError(gate=n)`, which the activity layer maps to an import step.
| Gate | Step recorded | What it checks | Written to disk |
|---|---|---|---|
| 1 | `download` | object size ceiling, key prefix, `.sientia` suffix — from `stat_object`, before the body is fetched; then the SHA-256 of the downloaded bytes against the digest the uploader computed | the downloaded object only |
| 2 | `download` | the whole header table of §4, including the chunk-stream digest | nothing |
| 3 | `decryption` | Argon2id derivation with the header's parameters, then `secretstream` pull. The AEAD is the integrity check: a wrong password and an altered file fail identically, and the message says so | the decrypted zip, in the work directory |
| 4 | `archive_inspection` | central directory only: entry count, total uncompressed size, compression ratio, absolute names, `..` components, symlink and non-regular modes | nothing |
| 5 | `extraction` | extraction into a **fresh** directory (a pre-existing one is a failure), every member's `realpath` contained in it, no overwrite of a pre-existing file; a rejection removes its own partial tree | the extracted tree |
| 6 | `structure_validation` | `metadata.json` + `artifacts/` layout, exactly two top-level keys, all six metadata fields present and non-blank, `experiment_name` between 3 and 50 characters, `parameters.target_variable` present and non-blank, and `prediction_model`/`data_model` each holding `MLmodel` and `model.pkl` | nothing |
| 7 | `content_policy` | every file is either a known name (`MLmodel`, `metadata.json`, `model.pkl`, `conda.yaml`, `python_env.yaml`, `requirements.txt`, `model_card.json`, `model_card.svg`) or a known suffix (`.csv .json .yaml .yml .txt .pkl .svg .md`) | nothing |
Gates 1 and 2 share the `download` step deliberately: an oversized object, a wrong magic, an
unsupported version and a digest that disagrees all say *these are not the bytes that were sent*, and
none of them may borrow the `decryption` sentence, which implicates the password.
The `experiment_name` bound in gate 6 is not cosmetic: the import record's `experiment_name` column is
`VARCHAR(50)`, and the name is written onto the record before provisioning starts. Enforcing the
column's limit at the gate is what keeps it from surfacing three activities later as a constraint
violation.
## 6. Ceilings and where they come from
Gate 1 and gate 4 read their limits from `build_import_config()`, so a deployment can tighten them:
| Variable | Default | Gate |
|---|---|---|
| `IMPORT_MAX_OBJECT_BYTES` | 1 GiB | 1 |
| `IMPORT_MAX_ARCHIVE_ENTRIES` | 5000 | 4 |
| `IMPORT_MAX_UNCOMPRESSED_BYTES` | 4 GiB | 4 |
| `IMPORT_MAX_COMPRESSION_RATIO` | 200 | 4 |
| `IMPORT_BUNDLE_PREFIX` | `imported_models/` | 1 |
The KDF ceilings are **not** configurable: they are the producer's own cost parameters, and a header
declaring more is a resource-exhaustion attempt, refused before any memory is allocated.
## 7. Fixtures
- `tests/fixtures/bundle/golden_model_v3.sientia` — emitted by the producer's own code in its venv,
with the password and producer commit recorded in `tests/fixtures/bundle/README.md`. Regenerate it
with `tests/fixtures/bundle/regenerate.py`. It pays the producer's real Argon2id cost (256 MiB,
3 passes), so it is used where the real thing matters and not in every test.
- `tests/helpers/bundle_factory.py` — the test-only writer. Same header fields at the same offsets,
but 8 MiB and 1 pass, and with a knob for every hostile variant: wrong magic, unknown version, KDF
parameters above the ceiling, digest mismatch, tampered ciphertext, truncated final chunk, non-zero
reserved signature, zip bomb, `..` entry, absolute entry, symlink, missing `data_model`, missing
`MLmodel`, missing `model.pkl`, malformed metadata, missing `target_variable`, extra `.sh` file. A
test asserts its header bytes are structurally identical to the golden fixture's.