This commit is contained in:
vitor-aignosi
100
docs/sientia-bundle-format.md
Normal file
100
docs/sientia-bundle-format.md
Normal file
@@ -0,0 +1,100 @@
|
||||
# The `.sientia` bundle format
|
||||
|
||||
A `.sientia` file is one exported MLflow model: its whole artifact tree, its run parameters and its
|
||||
origin metadata, zipped and then encrypted under a password the exporter chose.
|
||||
|
||||
**It is written by the Streamlit platform, `sientia-projects-templates`** (`app/src/operations/
|
||||
model_export/`). That application owns the export — its screen, its flow, its code — and documents
|
||||
it. This page is the short version: the parameters both sides have to agree on, and what this
|
||||
repository's reader accepts.
|
||||
|
||||
## How it is generated
|
||||
|
||||
The producer downloads the run's artifact tree whole, copies `run.data.params` **verbatim** (with
|
||||
`target_variable` among them, or the import fails at gate 6), writes `metadata.json` first into a
|
||||
`ZIP_DEFLATED` archive with the artifacts following as `artifacts/<relpath>` and no directory
|
||||
entries, encrypts that zip, and offers it as `{model_name}_v{model_version}.sientia`.
|
||||
|
||||
`metadata.json` has exactly two keys: `parameters` — the run's parameters verbatim — and `metadata`,
|
||||
six fields: `model_name`, `experiment_name`, `run_id`, `model_version`, `model_project`,
|
||||
`export_timestamp` (UTC ISO 8601).
|
||||
|
||||
## Encryption parameters
|
||||
|
||||
The reader derives the key with the values the header carries, but refuses anything above the
|
||||
producer's own cost — a header declaring more is a resource-exhaustion attempt, refused before any
|
||||
memory is allocated.
|
||||
|
||||
| | Value |
|
||||
|---|---|
|
||||
| KDF | Argon2id (`crypto_pwhash_ALG_ARGON2ID13`) |
|
||||
| memlimit | `268435456` (256 MiB) — a **ceiling** on the reader's side |
|
||||
| opslimit | `3` — likewise a ceiling |
|
||||
| parallelism | `1` (reserved, not configurable) |
|
||||
| salt | 16 bytes, fresh per bundle |
|
||||
| key | 32 bytes |
|
||||
| AEAD | XChaCha20-Poly1305 `secretstream`, 64 KiB plaintext chunks, last tagged `FINAL` |
|
||||
| digest | SHA-256 of the **pushed chunks only** — on disk, `sha256(file[164:])`, past the 140-byte header and the 24-byte stream header. Hashing from 140 rejects every valid bundle |
|
||||
| password | normalised `NFC`, encoded UTF-8 — no trimming, no case folding |
|
||||
|
||||
The export-side password rule (12+ characters, mixed case, digit, symbol) is enforced there only: by
|
||||
the time a bundle exists, the password is whatever it was, and the reader cannot re-check it.
|
||||
|
||||
## Shape of the file
|
||||
|
||||
`HEADER_STRUCT` is `>8sBBB16sQQB32s64s`. Big-endian, no alignment padding, so the offsets are exact:
|
||||
|
||||
```
|
||||
offset width field constraint the reader enforces
|
||||
------ ----- -------------------- ------------------------------------------------------
|
||||
0 8 magic == b'SIENTIA1'
|
||||
8 1 format_version == 1
|
||||
9 1 aead_id == 1 (XChaCha20-Poly1305 secretstream)
|
||||
10 1 kdf_id == 1 (Argon2id)
|
||||
11 16 salt passed to the KDF as-is
|
||||
27 8 kdf_memlimit_bytes <= 268435456
|
||||
35 8 kdf_opslimit <= 3
|
||||
43 1 kdf_parallelism == 1
|
||||
44 32 ciphertext_digest == sha256(file[164:])
|
||||
76 64 reserved_signature all zero
|
||||
------ -----
|
||||
140 24 secretstream header libsodium HEADERBYTES
|
||||
164 ... chunk stream 64 KiB plaintext chunks, each +17 bytes of AEAD tag
|
||||
```
|
||||
|
||||
`format_version` is checked for equality, not for a range: an unknown version is refused at gate 2
|
||||
rather than parsed optimistically. `reserved_signature` must be all zero today, so a signed bundle
|
||||
will be a new format version and not a silent change of meaning.
|
||||
|
||||
`laborious/utils/bundle/format.py` is the normative layout on this side, and the golden fixture is
|
||||
what keeps it and the producer from drifting apart.
|
||||
|
||||
## How it is read
|
||||
|
||||
`BundleReader.open()` runs seven ordered gates, each mapped to an import step and to one message a
|
||||
person can read:
|
||||
|
||||
| Gate | Step | Refuses |
|
||||
|---|---|---|
|
||||
| 1 | `download` | object too large, wrong prefix or suffix, digest ≠ the one the uploader sent |
|
||||
| 2 | `download` | header that is not this format or not this version |
|
||||
| 3 | `decryption` | wrong password, or altered bytes — the AEAD cannot tell them apart, and the message says so |
|
||||
| 4 | `archive_inspection` | zip bomb, absolute or `..` entries, symlinks — read from the central directory, before extracting |
|
||||
| 5 | `extraction` | anything that escapes the fresh work directory, or overwrites a file |
|
||||
| 6 | `structure_validation` | missing `metadata.json` fields, missing `target_variable`, missing `prediction_model`/`data_model` |
|
||||
| 7 | `content_policy` | a file that is not a known model file by name or suffix |
|
||||
|
||||
Gates 1 and 2 share the `download` step on purpose: neither may borrow the `decryption` sentence,
|
||||
which implicates the password. A gate-6 rejection naming `target_variable` means the origin run never
|
||||
logged it — the fix is a re-export, on the producer's side.
|
||||
|
||||
Error codes, sentences and the ceilings the gates read from configuration:
|
||||
[`model-import.md`](model-import.md).
|
||||
|
||||
## Fixtures
|
||||
|
||||
- `tests/fixtures/bundle/` — a real bundle emitted by the producer, with its password and producer
|
||||
commit recorded in that directory's `README.md`. It pays the producer's real Argon2id cost, so it
|
||||
is used where the real bytes matter and not in every test.
|
||||
- `tests/helpers/bundle_factory.py` — the test-only writer: same header at the same offsets, cheap
|
||||
KDF parameters, and a knob for every hostile variant the gates above refuse.
|
||||
Reference in New Issue
Block a user