vitor-aignosi
c1a44a9d7a
QTZPOC-20: Unify artifact code directory handling for all model flavors and standardize joblib pickle path Implemented a robust mechanism to ensure an artifact's code/ directory is prepended to sys.path before loading for sklearn, pyfunc, and joblib flavors. sklearn and pyfunc now resolve MLflow URIs to local directories first to facilitate this sys.path modification. Standardized the joblib flavor to explicitly read and write model.pkl within the artifact directory. Updated documentation and added comprehensive tests to reflect these changes.
101 lines
5.3 KiB
Markdown
101 lines
5.3 KiB
Markdown
# The `.sientia` bundle format
|
|
|
|
A `.sientia` file is one exported MLflow model: its whole artifact tree, its run parameters and its
|
|
origin metadata, zipped and then encrypted under a password the exporter chose.
|
|
|
|
**It is written by the Streamlit platform, `sientia-projects-templates`** (`app/src/operations/
|
|
model_export/`). That application owns the export — its screen, its flow, its code — and documents
|
|
it. This page is the short version: the parameters both sides have to agree on, and what this
|
|
repository's reader accepts.
|
|
|
|
## How it is generated
|
|
|
|
The producer downloads the run's artifact tree whole, copies `run.data.params` **verbatim** (with
|
|
`target_variable` among them, or the import fails at gate 6), writes `metadata.json` first into a
|
|
`ZIP_DEFLATED` archive with the artifacts following as `artifacts/<relpath>` and no directory
|
|
entries, encrypts that zip, and offers it as `{model_name}_v{model_version}.sientia`.
|
|
|
|
`metadata.json` has exactly two keys: `parameters` — the run's parameters verbatim — and `metadata`,
|
|
six fields: `model_name`, `experiment_name`, `run_id`, `model_version`, `model_project`,
|
|
`export_timestamp` (UTC ISO 8601).
|
|
|
|
## Encryption parameters
|
|
|
|
The reader derives the key with the values the header carries, but refuses anything above the
|
|
producer's own cost — a header declaring more is a resource-exhaustion attempt, refused before any
|
|
memory is allocated.
|
|
|
|
| | Value |
|
|
|---|---|
|
|
| KDF | Argon2id (`crypto_pwhash_ALG_ARGON2ID13`) |
|
|
| memlimit | `268435456` (256 MiB) — a **ceiling** on the reader's side |
|
|
| opslimit | `3` — likewise a ceiling |
|
|
| parallelism | `1` (reserved, not configurable) |
|
|
| salt | 16 bytes, fresh per bundle |
|
|
| key | 32 bytes |
|
|
| AEAD | XChaCha20-Poly1305 `secretstream`, 64 KiB plaintext chunks, last tagged `FINAL` |
|
|
| digest | SHA-256 of the **pushed chunks only** — on disk, `sha256(file[164:])`, past the 140-byte header and the 24-byte stream header. Hashing from 140 rejects every valid bundle |
|
|
| password | normalised `NFC`, encoded UTF-8 — no trimming, no case folding |
|
|
|
|
The export-side password rule (12+ characters, mixed case, digit, symbol) is enforced there only: by
|
|
the time a bundle exists, the password is whatever it was, and the reader cannot re-check it.
|
|
|
|
## Shape of the file
|
|
|
|
`HEADER_STRUCT` is `>8sBBB16sQQB32s64s`. Big-endian, no alignment padding, so the offsets are exact:
|
|
|
|
```
|
|
offset width field constraint the reader enforces
|
|
------ ----- -------------------- ------------------------------------------------------
|
|
0 8 magic == b'SIENTIA1'
|
|
8 1 format_version == 1
|
|
9 1 aead_id == 1 (XChaCha20-Poly1305 secretstream)
|
|
10 1 kdf_id == 1 (Argon2id)
|
|
11 16 salt passed to the KDF as-is
|
|
27 8 kdf_memlimit_bytes <= 268435456
|
|
35 8 kdf_opslimit <= 3
|
|
43 1 kdf_parallelism == 1
|
|
44 32 ciphertext_digest == sha256(file[164:])
|
|
76 64 reserved_signature all zero
|
|
------ -----
|
|
140 24 secretstream header libsodium HEADERBYTES
|
|
164 ... chunk stream 64 KiB plaintext chunks, each +17 bytes of AEAD tag
|
|
```
|
|
|
|
`format_version` is checked for equality, not for a range: an unknown version is refused at gate 2
|
|
rather than parsed optimistically. `reserved_signature` must be all zero today, so a signed bundle
|
|
will be a new format version and not a silent change of meaning.
|
|
|
|
`laborious/utils/bundle/format.py` is the normative layout on this side, and the golden fixture is
|
|
what keeps it and the producer from drifting apart.
|
|
|
|
## How it is read
|
|
|
|
`BundleReader.open()` runs seven ordered gates, each mapped to an import step and to one message a
|
|
person can read:
|
|
|
|
| Gate | Step | Refuses |
|
|
|---|---|---|
|
|
| 1 | `download` | object too large, wrong prefix or suffix, digest ≠ the one the uploader sent |
|
|
| 2 | `download` | header that is not this format or not this version |
|
|
| 3 | `decryption` | wrong password, or altered bytes — the AEAD cannot tell them apart, and the message says so |
|
|
| 4 | `archive_inspection` | zip bomb, absolute or `..` entries, symlinks — read from the central directory, before extracting |
|
|
| 5 | `extraction` | anything that escapes the fresh work directory, or overwrites a file |
|
|
| 6 | `structure_validation` | missing `metadata.json` fields, missing `target_variable`, missing `prediction_model`/`data_model` |
|
|
| 7 | `content_policy` | a file that is not a known model file by name or suffix |
|
|
|
|
Gates 1 and 2 share the `download` step on purpose: neither may borrow the `decryption` sentence,
|
|
which implicates the password. A gate-6 rejection naming `target_variable` means the origin run never
|
|
logged it — the fix is a re-export, on the producer's side.
|
|
|
|
Error codes, sentences and the ceilings the gates read from configuration:
|
|
[`model-import.md`](model-import.md).
|
|
|
|
## Fixtures
|
|
|
|
- `tests/fixtures/bundle/` — a real bundle emitted by the producer, with its password and producer
|
|
commit recorded in that directory's `README.md`. It pays the producer's real Argon2id cost, so it
|
|
is used where the real bytes matter and not in every test.
|
|
- `tests/helpers/bundle_factory.py` — the test-only writer: same header at the same offsets, cheap
|
|
KDF parameters, and a knob for every hostile variant the gates above refuse.
|