Add docs
This commit is contained in:
vitor-aignosi
196
docs/sientia-bundle-format.md
Normal file
196
docs/sientia-bundle-format.md
Normal file
@@ -0,0 +1,196 @@
|
||||
# The `.sientia` bundle format
|
||||
|
||||
A `.sientia` file is one exported MLflow model: its whole artifact tree, its run parameters and six
|
||||
fields of origin metadata, zipped and then encrypted under a password the exporter chose. It is
|
||||
written by the Streamlit platform (`sientia-projects-templates`) and read by this repository's
|
||||
importer. Two programs in two repositories agree on these bytes, so the constants below are
|
||||
duplicated by necessity — the drift is held in place by tests on both sides, not by a shared library.
|
||||
|
||||
This document covers both directions: **§1–§3 how a bundle is generated**, so the producer can be
|
||||
audited against it, and **§4–§7 how it is read**, so a rejection can be explained.
|
||||
|
||||
Normative implementations:
|
||||
|
||||
| Side | Where |
|
||||
|---|---|
|
||||
| Writer | `sientia-projects-templates`, `app/src/operations/model_export/{crypto,bundle,exporter}.py` |
|
||||
| Reader | this repository, `laborious/utils/bundle/{format,reader}.py` |
|
||||
| Test writer | this repository, `tests/helpers/bundle_factory.py` — mirrors the producer field for field, for building hostile variants |
|
||||
| Golden fixture | `tests/fixtures/bundle/` — a real producer artefact, with its password and producer commit recorded in that directory's `README.md` |
|
||||
|
||||
---
|
||||
|
||||
## 1. Generation, step by step
|
||||
|
||||
Entry point: `exporter.export_model_bundle(*, client, run_id, model_name, experiment_name,
|
||||
model_version, model_project, password) -> bytes`.
|
||||
|
||||
1. **Password strength** — `crypto.validate_password_strength`: at least 12 characters, one
|
||||
uppercase, one lowercase, one digit, one symbol. **Export-side only.** The importer cannot
|
||||
re-check it and does not try: by the time a bundle exists, the password is whatever it was.
|
||||
2. **Exportability** — `bundle.validate_run_is_exportable`: the run's top-level artifacts must
|
||||
include **both** `data_model` and `prediction_model`, checked with `client.list_artifacts(run_id)`
|
||||
before anything is downloaded.
|
||||
3. **Whole-tree download** — `client.download_artifacts(run_id, '', staging)` into a
|
||||
`tempfile.mkdtemp(prefix='sientia_export_')` directory. The entire artifact tree, not a subset.
|
||||
4. **Run parameters** — `dict(run.data.params)`, copied **verbatim**. Whatever the run logged is
|
||||
what the bundle carries, including `target_variable` (see §3).
|
||||
5. **Metadata document** — `bundle.build_metadata_json`, exactly two top-level keys:
|
||||
- `parameters`: the run parameters, verbatim;
|
||||
- `metadata`: `model_name`, `experiment_name`, `run_id`, `model_version`, `model_project` and
|
||||
`export_timestamp = datetime.now(UTC).isoformat()`. Six fields, no others.
|
||||
6. **Zip** — `bundle.build_zip`, `zipfile.ZIP_DEFLATED`. `metadata.json` is written **first**, with
|
||||
`json.dumps(...)` and no indent. Every file under the staging directory follows as
|
||||
`artifacts/<relpath>`, with `\` replaced by `/`. **No directory entries are stored**, so a reader
|
||||
must derive directories from file paths.
|
||||
7. **Encryption** — `crypto.encrypt_archive`: a fresh 16-byte `os.urandom` salt, Argon2id key
|
||||
derivation, then a libsodium `secretstream` push in 64 KiB plaintext chunks, the last chunk tagged
|
||||
`FINAL` and the rest `MESSAGE`.
|
||||
8. **Download name** — `{model_name}_v{model_version}.sientia`. The staging directory is removed in a
|
||||
`finally`, and the password is popped from the Streamlit session state.
|
||||
|
||||
Producer pins: `pynacl==1.5.0`, `mlflow==2.10.1`.
|
||||
|
||||
## 2. Constants
|
||||
|
||||
Copied from `crypto.py` unless noted. The reader's names are the ones in
|
||||
`laborious/utils/bundle/format.py`.
|
||||
|
||||
| Producer | Value | Reader |
|
||||
|---|---|---|
|
||||
| `MAGIC` | `b'SIENTIA1'` | `MAGIC` |
|
||||
| `_HEADER_STRUCT` | `struct.Struct('>8sBBB16sQQB32s64s')` | `HEADER_STRUCT` |
|
||||
| `HEADER_STRUCT_SIZE` | `140` | `HEADER_SIZE` |
|
||||
| `FORMAT_VERSION` | `1` | `FORMAT_VERSION` |
|
||||
| `AEAD_ID_XCHACHA20POLY1305_SECRETSTREAM` | `1` | same name |
|
||||
| `KDF_ID_ARGON2ID` | `1` | same name |
|
||||
| `SALT_SIZE` | `16` | `SALT_SIZE` |
|
||||
| `KDF_MEMLIMIT_BYTES` | `268435456` (256 MiB) | `MAX_KDF_MEMLIMIT` — a **ceiling** on the reader's side |
|
||||
| `KDF_OPSLIMIT` | `3` | `MAX_KDF_OPSLIMIT` — likewise a ceiling |
|
||||
| `KDF_PARALLELISM_RESERVED` | `1` | `PARALLELISM` |
|
||||
| `DIGEST_SIZE` | `32` | `DIGEST_SIZE` |
|
||||
| `RESERVED_SIGNATURE_SIZE` | `64` | `RESERVED_SIGNATURE_SIZE` |
|
||||
| `CHUNK_SIZE` | `65536` (64 KiB) | `CHUNK_SIZE` |
|
||||
| KDF algorithm | `crypto_pwhash_ALG_ARGON2ID13` | `KDF_ALG` |
|
||||
| Key length | `crypto_secretstream_xchacha20poly1305_KEYBYTES` (32) | `KEY_SIZE` |
|
||||
|
||||
Fixed by libsodium rather than by the producer, but needed to frame the stream:
|
||||
|
||||
- `crypto_secretstream_xchacha20poly1305_HEADERBYTES == 24` → `STREAM_HEADER_SIZE`
|
||||
- `crypto_secretstream_xchacha20poly1305_ABYTES == 17` → `ABYTES`
|
||||
- pre-upload prefix `140 + 24 + (65536 + 17) = 65717` → `PREFIX_SIZE`, the smallest number of leading
|
||||
bytes from which a header and one full chunk can be checked
|
||||
|
||||
**Password normalisation.** `_derive_key` applies `unicodedata.normalize('NFC', password)` and
|
||||
encodes UTF-8 — no trimming, no case folding, no padding. The reader normalises identically, so a
|
||||
password typed in decomposed form opens a bundle written from the composed form.
|
||||
|
||||
## 3. `target_variable`, and where it does not come from
|
||||
|
||||
Gate 6 requires `parameters.target_variable`: it is the only source of `model_config.target` in the
|
||||
model document, and every workflow that serves a model indexes that key directly. On the producer
|
||||
side:
|
||||
|
||||
| Export path | Logs `target_variable`? | Evidence |
|
||||
|---|---|---|
|
||||
| Time-series training template | yes | `app/pages/template_01.py` L1212-1214 |
|
||||
| Model manager, "Save Experiment" | yes, if the session carried it | `app/pages/model_manager.py` L791-796 re-logs `st.session_state.parameters` verbatim |
|
||||
| Model ensemble | same condition | `app/pages/model_ensemble.py` L695-700 |
|
||||
| Pipeline / timeseries template | **no** | `app/pages/timeseries_template.py` L1761-1762, L1811-1812 log `run_metadata` only |
|
||||
| AutoML | **no** | `automl_logging_operations.py` L454-491; the target appears as `"target"` inside the model card JSON, never as a run parameter |
|
||||
|
||||
So a gate-6 rejection naming `target_variable` means the origin run came from the pipeline or AutoML
|
||||
path. The remedy is to re-export from a run trained by the time-series template, which is what the
|
||||
importer's `IMPORT_BUNDLE_INCOMPLETE` sentence already asks for.
|
||||
|
||||
---
|
||||
|
||||
## 4. The file on disk
|
||||
|
||||
```
|
||||
offset width field constraint the reader enforces
|
||||
------ ----- -------------------- ------------------------------------------------------
|
||||
0 8 magic == b'SIENTIA1'
|
||||
8 1 format_version == 1
|
||||
9 1 aead_id == 1 (XChaCha20-Poly1305 secretstream)
|
||||
10 1 kdf_id == 1 (Argon2id)
|
||||
11 16 salt passed to the KDF as-is
|
||||
27 8 kdf_memlimit_bytes <= 268435456
|
||||
35 8 kdf_opslimit <= 3
|
||||
43 1 kdf_parallelism == 1
|
||||
44 32 ciphertext_digest == sha256(file[164:])
|
||||
76 64 reserved_signature all zero
|
||||
------ -----
|
||||
140 24 secretstream header libsodium HEADERBYTES
|
||||
164 ... chunk stream 64 KiB plaintext chunks, each +17 bytes of AEAD tag
|
||||
```
|
||||
|
||||
All integers are big-endian; the format string starts with `>`, so there is no alignment padding and
|
||||
the offsets above are exact.
|
||||
|
||||
**The digest boundary.** The producer computes `hashlib.sha256(ciphertext)` where `ciphertext` is the
|
||||
concatenation of the *pushed chunks only*, and then writes `header + stream_header + ciphertext`. On
|
||||
disk, therefore, the digest covers everything after offset **164**, not after 140:
|
||||
|
||||
> `ciphertext_digest == sha256(file[HEADER_SIZE + STREAM_HEADER_SIZE:])`
|
||||
|
||||
A reader that hashes from 140 will reject every valid bundle. The fixture test that recomputes both
|
||||
values and asserts the first matches while the second does not is what pins this, and is the reason
|
||||
the golden fixture exists.
|
||||
|
||||
**Version policy.** `format_version` is checked for equality, not for a range: a version this reader
|
||||
does not know is refused at gate 2 rather than parsed optimistically. `reserved_signature` is
|
||||
reserved for a future signature and must be all zero today, so a signed bundle is a new format
|
||||
version and not a silent change of meaning.
|
||||
|
||||
## 5. The seven gates
|
||||
|
||||
`BundleReader.open()` runs them in this order, in one function, so the order cannot be changed by a
|
||||
caller. Each raises `BundleGateError(gate=n)`, which the activity layer maps to an import step.
|
||||
|
||||
| Gate | Step recorded | What it checks | Written to disk |
|
||||
|---|---|---|---|
|
||||
| 1 | `download` | object size ceiling, key prefix, `.sientia` suffix — from `stat_object`, before the body is fetched; then the SHA-256 of the downloaded bytes against the digest the uploader computed | the downloaded object only |
|
||||
| 2 | `download` | the whole header table of §4, including the chunk-stream digest | nothing |
|
||||
| 3 | `decryption` | Argon2id derivation with the header's parameters, then `secretstream` pull. The AEAD is the integrity check: a wrong password and an altered file fail identically, and the message says so | the decrypted zip, in the work directory |
|
||||
| 4 | `archive_inspection` | central directory only: entry count, total uncompressed size, compression ratio, absolute names, `..` components, symlink and non-regular modes | nothing |
|
||||
| 5 | `extraction` | extraction into a **fresh** directory (a pre-existing one is a failure), every member's `realpath` contained in it, no overwrite of a pre-existing file; a rejection removes its own partial tree | the extracted tree |
|
||||
| 6 | `structure_validation` | `metadata.json` + `artifacts/` layout, exactly two top-level keys, all six metadata fields present and non-blank, `experiment_name` between 3 and 50 characters, `parameters.target_variable` present and non-blank, and `prediction_model`/`data_model` each holding `MLmodel` and `model.pkl` | nothing |
|
||||
| 7 | `content_policy` | every file is either a known name (`MLmodel`, `metadata.json`, `model.pkl`, `conda.yaml`, `python_env.yaml`, `requirements.txt`, `model_card.json`, `model_card.svg`) or a known suffix (`.csv .json .yaml .yml .txt .pkl .svg .md`) | nothing |
|
||||
|
||||
Gates 1 and 2 share the `download` step deliberately: an oversized object, a wrong magic, an
|
||||
unsupported version and a digest that disagrees all say *these are not the bytes that were sent*, and
|
||||
none of them may borrow the `decryption` sentence, which implicates the password.
|
||||
|
||||
The `experiment_name` bound in gate 6 is not cosmetic: the import record's `experiment_name` column is
|
||||
`VARCHAR(50)`, and the name is written onto the record before provisioning starts. Enforcing the
|
||||
column's limit at the gate is what keeps it from surfacing three activities later as a constraint
|
||||
violation.
|
||||
|
||||
## 6. Ceilings and where they come from
|
||||
|
||||
Gate 1 and gate 4 read their limits from `build_import_config()`, so a deployment can tighten them:
|
||||
|
||||
| Variable | Default | Gate |
|
||||
|---|---|---|
|
||||
| `IMPORT_MAX_OBJECT_BYTES` | 1 GiB | 1 |
|
||||
| `IMPORT_MAX_ARCHIVE_ENTRIES` | 5000 | 4 |
|
||||
| `IMPORT_MAX_UNCOMPRESSED_BYTES` | 4 GiB | 4 |
|
||||
| `IMPORT_MAX_COMPRESSION_RATIO` | 200 | 4 |
|
||||
| `IMPORT_BUNDLE_PREFIX` | `imported_models/` | 1 |
|
||||
|
||||
The KDF ceilings are **not** configurable: they are the producer's own cost parameters, and a header
|
||||
declaring more is a resource-exhaustion attempt, refused before any memory is allocated.
|
||||
|
||||
## 7. Fixtures
|
||||
|
||||
- `tests/fixtures/bundle/golden_model_v3.sientia` — emitted by the producer's own code in its venv,
|
||||
with the password and producer commit recorded in `tests/fixtures/bundle/README.md`. Regenerate it
|
||||
with `tests/fixtures/bundle/regenerate.py`. It pays the producer's real Argon2id cost (256 MiB,
|
||||
3 passes), so it is used where the real thing matters and not in every test.
|
||||
- `tests/helpers/bundle_factory.py` — the test-only writer. Same header fields at the same offsets,
|
||||
but 8 MiB and 1 pass, and with a knob for every hostile variant: wrong magic, unknown version, KDF
|
||||
parameters above the ceiling, digest mismatch, tampered ciphertext, truncated final chunk, non-zero
|
||||
reserved signature, zip bomb, `..` entry, absolute entry, symlink, missing `data_model`, missing
|
||||
`MLmodel`, missing `model.pkl`, malformed metadata, missing `target_variable`, extra `.sh` file. A
|
||||
test asserts its header bytes are structurally identical to the golden fixture's.
|
||||
Reference in New Issue
Block a user