sientia-dataops-laborious_temporal/docs/sientia-bundle-format.md

# The `.sientia` bundle format

A `.sientia` file is one exported MLflow model: its whole artifact tree, its run parameters and six
fields of origin metadata, zipped and then encrypted under a password the exporter chose. It is
written by the Streamlit platform (`sientia-projects-templates`) and read by this repository's
importer. Two programs in two repositories agree on these bytes, so the constants below are
duplicated by necessity — the drift is held in place by tests on both sides, not by a shared library.

This document covers both directions: **§1–§3 how a bundle is generated**, so the producer can be
audited against it, and **§4–§7 how it is read**, so a rejection can be explained.

Normative implementations:

| Side | Where |
|---|---|
| Writer | `sientia-projects-templates`, `app/src/operations/model_export/{crypto,bundle,exporter}.py` |
| Reader | this repository, `laborious/utils/bundle/{format,reader}.py` |
| Test writer | this repository, `tests/helpers/bundle_factory.py` — mirrors the producer field for field, for building hostile variants |
| Golden fixture | `tests/fixtures/bundle/` — a real producer artefact, with its password and producer commit recorded in that directory's `README.md` |

---

## 1. Generation, step by step

Entry point: `exporter.export_model_bundle(*, client, run_id, model_name, experiment_name,
model_version, model_project, password) -> bytes`.

1. **Password strength** — `crypto.validate_password_strength`: at least 12 characters, one
   uppercase, one lowercase, one digit, one symbol. **Export-side only.** The importer cannot
   re-check it and does not try: by the time a bundle exists, the password is whatever it was.
2. **Exportability** — `bundle.validate_run_is_exportable`: the run's top-level artifacts must
   include **both** `data_model` and `prediction_model`, checked with `client.list_artifacts(run_id)`
   before anything is downloaded.
3. **Whole-tree download** — `client.download_artifacts(run_id, '', staging)` into a
   `tempfile.mkdtemp(prefix='sientia_export_')` directory. The entire artifact tree, not a subset.
4. **Run parameters** — `dict(run.data.params)`, copied **verbatim**. Whatever the run logged is
   what the bundle carries, including `target_variable` (see §3).
5. **Metadata document** — `bundle.build_metadata_json`, exactly two top-level keys:
   - `parameters`: the run parameters, verbatim;
   - `metadata`: `model_name`, `experiment_name`, `run_id`, `model_version`, `model_project` and
     `export_timestamp = datetime.now(UTC).isoformat()`. Six fields, no others.
6. **Zip** — `bundle.build_zip`, `zipfile.ZIP_DEFLATED`. `metadata.json` is written **first**, with
   `json.dumps(...)` and no indent. Every file under the staging directory follows as
   `artifacts/<relpath>`, with `\` replaced by `/`. **No directory entries are stored**, so a reader
   must derive directories from file paths.
7. **Encryption** — `crypto.encrypt_archive`: a fresh 16-byte `os.urandom` salt, Argon2id key
   derivation, then a libsodium `secretstream` push in 64 KiB plaintext chunks, the last chunk tagged
   `FINAL` and the rest `MESSAGE`.
8. **Download name** — `{model_name}_v{model_version}.sientia`. The staging directory is removed in a
   `finally`, and the password is popped from the Streamlit session state.

Producer pins: `pynacl==1.5.0`, `mlflow==2.10.1`.

## 2. Constants

Copied from `crypto.py` unless noted. The reader's names are the ones in
`laborious/utils/bundle/format.py`.

| Producer | Value | Reader |
|---|---|---|
| `MAGIC` | `b'SIENTIA1'` | `MAGIC` |
| `_HEADER_STRUCT` | `struct.Struct('>8sBBB16sQQB32s64s')` | `HEADER_STRUCT` |
| `HEADER_STRUCT_SIZE` | `140` | `HEADER_SIZE` |
| `FORMAT_VERSION` | `1` | `FORMAT_VERSION` |
| `AEAD_ID_XCHACHA20POLY1305_SECRETSTREAM` | `1` | same name |
| `KDF_ID_ARGON2ID` | `1` | same name |
| `SALT_SIZE` | `16` | `SALT_SIZE` |
| `KDF_MEMLIMIT_BYTES` | `268435456` (256 MiB) | `MAX_KDF_MEMLIMIT` — a **ceiling** on the reader's side |
| `KDF_OPSLIMIT` | `3` | `MAX_KDF_OPSLIMIT` — likewise a ceiling |
| `KDF_PARALLELISM_RESERVED` | `1` | `PARALLELISM` |
| `DIGEST_SIZE` | `32` | `DIGEST_SIZE` |
| `RESERVED_SIGNATURE_SIZE` | `64` | `RESERVED_SIGNATURE_SIZE` |
| `CHUNK_SIZE` | `65536` (64 KiB) | `CHUNK_SIZE` |
| KDF algorithm | `crypto_pwhash_ALG_ARGON2ID13` | `KDF_ALG` |
| Key length | `crypto_secretstream_xchacha20poly1305_KEYBYTES` (32) | `KEY_SIZE` |

Fixed by libsodium rather than by the producer, but needed to frame the stream:

- `crypto_secretstream_xchacha20poly1305_HEADERBYTES == 24` → `STREAM_HEADER_SIZE`
- `crypto_secretstream_xchacha20poly1305_ABYTES == 17` → `ABYTES`
- pre-upload prefix `140 + 24 + (65536 + 17) = 65717` → `PREFIX_SIZE`, the smallest number of leading
  bytes from which a header and one full chunk can be checked

**Password normalisation.** `_derive_key` applies `unicodedata.normalize('NFC', password)` and
encodes UTF-8 — no trimming, no case folding, no padding. The reader normalises identically, so a
password typed in decomposed form opens a bundle written from the composed form.

## 3. `target_variable`, and where it does not come from

Gate 6 requires `parameters.target_variable`: it is the only source of `model_config.target` in the
model document, and every workflow that serves a model indexes that key directly. On the producer
side:

| Export path | Logs `target_variable`? | Evidence |
|---|---|---|
| Time-series training template | yes | `app/pages/template_01.py` L1212-1214 |
| Model manager, "Save Experiment" | yes, if the session carried it | `app/pages/model_manager.py` L791-796 re-logs `st.session_state.parameters` verbatim |
| Model ensemble | same condition | `app/pages/model_ensemble.py` L695-700 |
| Pipeline / timeseries template | **no** | `app/pages/timeseries_template.py` L1761-1762, L1811-1812 log `run_metadata` only |
| AutoML | **no** | `automl_logging_operations.py` L454-491; the target appears as `"target"` inside the model card JSON, never as a run parameter |

So a gate-6 rejection naming `target_variable` means the origin run came from the pipeline or AutoML
path. The remedy is to re-export from a run trained by the time-series template, which is what the
importer's `IMPORT_BUNDLE_INCOMPLETE` sentence already asks for.

---

## 4. The file on disk

```
offset  width  field                 constraint the reader enforces
------  -----  --------------------  ------------------------------------------------------
     0      8  magic                 == b'SIENTIA1'
     8      1  format_version        == 1
     9      1  aead_id               == 1 (XChaCha20-Poly1305 secretstream)
    10      1  kdf_id                == 1 (Argon2id)
    11     16  salt                  passed to the KDF as-is
    27      8  kdf_memlimit_bytes    <= 268435456
    35      8  kdf_opslimit          <= 3
    43      1  kdf_parallelism       == 1
    44     32  ciphertext_digest     == sha256(file[164:])
    76     64  reserved_signature    all zero
------  -----
   140     24  secretstream header   libsodium HEADERBYTES
   164    ...  chunk stream          64 KiB plaintext chunks, each +17 bytes of AEAD tag
```

All integers are big-endian; the format string starts with `>`, so there is no alignment padding and
the offsets above are exact.

**The digest boundary.** The producer computes `hashlib.sha256(ciphertext)` where `ciphertext` is the
concatenation of the *pushed chunks only*, and then writes `header + stream_header + ciphertext`. On
disk, therefore, the digest covers everything after offset **164**, not after 140:

> `ciphertext_digest == sha256(file[HEADER_SIZE + STREAM_HEADER_SIZE:])`

A reader that hashes from 140 will reject every valid bundle. The fixture test that recomputes both
values and asserts the first matches while the second does not is what pins this, and is the reason
the golden fixture exists.

**Version policy.** `format_version` is checked for equality, not for a range: a version this reader
does not know is refused at gate 2 rather than parsed optimistically. `reserved_signature` is
reserved for a future signature and must be all zero today, so a signed bundle is a new format
version and not a silent change of meaning.

## 5. The seven gates

`BundleReader.open()` runs them in this order, in one function, so the order cannot be changed by a
caller. Each raises `BundleGateError(gate=n)`, which the activity layer maps to an import step.

| Gate | Step recorded | What it checks | Written to disk |
|---|---|---|---|
| 1 | `download` | object size ceiling, key prefix, `.sientia` suffix — from `stat_object`, before the body is fetched; then the SHA-256 of the downloaded bytes against the digest the uploader computed | the downloaded object only |
| 2 | `download` | the whole header table of §4, including the chunk-stream digest | nothing |
| 3 | `decryption` | Argon2id derivation with the header's parameters, then `secretstream` pull. The AEAD is the integrity check: a wrong password and an altered file fail identically, and the message says so | the decrypted zip, in the work directory |
| 4 | `archive_inspection` | central directory only: entry count, total uncompressed size, compression ratio, absolute names, `..` components, symlink and non-regular modes | nothing |
| 5 | `extraction` | extraction into a **fresh** directory (a pre-existing one is a failure), every member's `realpath` contained in it, no overwrite of a pre-existing file; a rejection removes its own partial tree | the extracted tree |
| 6 | `structure_validation` | `metadata.json` + `artifacts/` layout, exactly two top-level keys, all six metadata fields present and non-blank, `experiment_name` between 3 and 50 characters, `parameters.target_variable` present and non-blank, and `prediction_model`/`data_model` each holding `MLmodel` and `model.pkl` | nothing |
| 7 | `content_policy` | every file is either a known name (`MLmodel`, `metadata.json`, `model.pkl`, `conda.yaml`, `python_env.yaml`, `requirements.txt`, `model_card.json`, `model_card.svg`) or a known suffix (`.csv .json .yaml .yml .txt .pkl .svg .md`) | nothing |

Gates 1 and 2 share the `download` step deliberately: an oversized object, a wrong magic, an
unsupported version and a digest that disagrees all say *these are not the bytes that were sent*, and
none of them may borrow the `decryption` sentence, which implicates the password.

The `experiment_name` bound in gate 6 is not cosmetic: the import record's `experiment_name` column is
`VARCHAR(50)`, and the name is written onto the record before provisioning starts. Enforcing the
column's limit at the gate is what keeps it from surfacing three activities later as a constraint
violation.

## 6. Ceilings and where they come from

Gate 1 and gate 4 read their limits from `build_import_config()`, so a deployment can tighten them:

| Variable | Default | Gate |
|---|---|---|
| `IMPORT_MAX_OBJECT_BYTES` | 1 GiB | 1 |
| `IMPORT_MAX_ARCHIVE_ENTRIES` | 5000 | 4 |
| `IMPORT_MAX_UNCOMPRESSED_BYTES` | 4 GiB | 4 |
| `IMPORT_MAX_COMPRESSION_RATIO` | 200 | 4 |
| `IMPORT_BUNDLE_PREFIX` | `imported_models/` | 1 |

The KDF ceilings are **not** configurable: they are the producer's own cost parameters, and a header
declaring more is a resource-exhaustion attempt, refused before any memory is allocated.

## 7. Fixtures

- `tests/fixtures/bundle/golden_model_v3.sientia` — emitted by the producer's own code in its venv,
  with the password and producer commit recorded in `tests/fixtures/bundle/README.md`. Regenerate it
  with `tests/fixtures/bundle/regenerate.py`. It pays the producer's real Argon2id cost (256 MiB,
  3 passes), so it is used where the real thing matters and not in every test.
- `tests/helpers/bundle_factory.py` — the test-only writer. Same header fields at the same offsets,
  but 8 MiB and 1 pass, and with a knob for every hostile variant: wrong magic, unknown version, KDF
  parameters above the ceiling, digest mismatch, tampered ciphertext, truncated final chunk, non-zero
  reserved signature, zip bomb, `..` entry, absolute entry, symlink, missing `data_model`, missing
  `MLmodel`, missing `model.pkl`, malformed metadata, missing `target_variable`, extra `.sh` file. A
  test asserts its header bytes are structurally identical to the golden fixture's.