sientia-dataops-laborious_temporal/docs/sientia-bundle-format.md

The .sientia bundle format

A .sientia file is one exported MLflow model: its whole artifact tree, its run parameters and six fields of origin metadata, zipped and then encrypted under a password the exporter chose. It is written by the Streamlit platform (sientia-projects-templates) and read by this repository's importer. Two programs in two repositories agree on these bytes, so the constants below are duplicated by necessity — the drift is held in place by tests on both sides, not by a shared library.

This document covers both directions: §1–§3 how a bundle is generated, so the producer can be audited against it, and §4–§7 how it is read, so a rejection can be explained.

Normative implementations:

Side	Where
Writer	sientia-projects-templates, app/src/operations/model_export/{crypto,bundle,exporter}.py
Reader	this repository, laborious/utils/bundle/{format,reader}.py
Test writer	this repository, tests/helpers/bundle_factory.py — mirrors the producer field for field, for building hostile variants
Golden fixture	tests/fixtures/bundle/ — a real producer artefact, with its password and producer commit recorded in that directory's README.md

---

1. Generation, step by step

Entry point: exporter.export_model_bundle(*, client, run_id, model_name, experiment_name, model_version, model_project, password) -> bytes.

1. Password strength — crypto.validate_password_strength: at least 12 characters, one uppercase, one lowercase, one digit, one symbol. Export-side only. The importer cannot re-check it and does not try: by the time a bundle exists, the password is whatever it was.
2. Exportability — bundle.validate_run_is_exportable: the run's top-level artifacts must include both data_model and prediction_model, checked with client.list_artifacts(run_id) before anything is downloaded.
3. Whole-tree download — client.download_artifacts(run_id, '', staging) into a tempfile.mkdtemp(prefix='sientia_export_') directory. The entire artifact tree, not a subset.
4. Run parameters — dict(run.data.params), copied verbatim. Whatever the run logged is what the bundle carries, including target_variable (see §3).
5. Metadata document — bundle.build_metadata_json, exactly two top-level keys:
   • parameters: the run parameters, verbatim;
   • metadata: model_name, experiment_name, run_id, model_version, model_project and export_timestamp = datetime.now(UTC).isoformat(). Six fields, no others.
6. Zip — bundle.build_zip, zipfile.ZIP_DEFLATED. metadata.json is written first, with json.dumps(...) and no indent. Every file under the staging directory follows as artifacts/<relpath>, with \ replaced by /. No directory entries are stored, so a reader must derive directories from file paths.
7. Encryption — crypto.encrypt_archive: a fresh 16-byte os.urandom salt, Argon2id key derivation, then a libsodium secretstream push in 64 KiB plaintext chunks, the last chunk tagged FINAL and the rest MESSAGE.
8. Download name — {model_name}_v{model_version}.sientia. The staging directory is removed in a finally, and the password is popped from the Streamlit session state.

Producer pins: pynacl==1.5.0, mlflow==2.10.1.

2. Constants

Copied from crypto.py unless noted. The reader's names are the ones in laborious/utils/bundle/format.py.

Producer	Value	Reader
MAGIC	b'SIENTIA1'	MAGIC
_HEADER_STRUCT	struct.Struct('>8sBBB16sQQB32s64s')	HEADER_STRUCT
HEADER_STRUCT_SIZE	140	HEADER_SIZE
FORMAT_VERSION	1	FORMAT_VERSION
AEAD_ID_XCHACHA20POLY1305_SECRETSTREAM	1	same name
KDF_ID_ARGON2ID	1	same name
SALT_SIZE	16	SALT_SIZE
KDF_MEMLIMIT_BYTES	268435456 (256 MiB)	MAX_KDF_MEMLIMIT — a ceiling on the reader's side
KDF_OPSLIMIT	3	MAX_KDF_OPSLIMIT — likewise a ceiling
KDF_PARALLELISM_RESERVED	1	PARALLELISM
DIGEST_SIZE	32	DIGEST_SIZE
RESERVED_SIGNATURE_SIZE	64	RESERVED_SIGNATURE_SIZE
CHUNK_SIZE	65536 (64 KiB)	CHUNK_SIZE
KDF algorithm	crypto_pwhash_ALG_ARGON2ID13	KDF_ALG
Key length	crypto_secretstream_xchacha20poly1305_KEYBYTES (32)	KEY_SIZE

Fixed by libsodium rather than by the producer, but needed to frame the stream:

• crypto_secretstream_xchacha20poly1305_HEADERBYTES == 24 → STREAM_HEADER_SIZE
• crypto_secretstream_xchacha20poly1305_ABYTES == 17 → ABYTES
• pre-upload prefix 140 + 24 + (65536 + 17) = 65717 → PREFIX_SIZE, the smallest number of leading bytes from which a header and one full chunk can be checked

Password normalisation. _derive_key applies unicodedata.normalize('NFC', password) and encodes UTF-8 — no trimming, no case folding, no padding. The reader normalises identically, so a password typed in decomposed form opens a bundle written from the composed form.

3. target_variable, and where it does not come from

Gate 6 requires parameters.target_variable: it is the only source of model_config.target in the model document, and every workflow that serves a model indexes that key directly. On the producer side:

Export path	Logs target_variable?	Evidence
Time-series training template	yes	app/pages/template_01.py L1212-1214
Model manager, "Save Experiment"	yes, if the session carried it	app/pages/model_manager.py L791-796 re-logs st.session_state.parameters verbatim
Model ensemble	same condition	app/pages/model_ensemble.py L695-700
Pipeline / timeseries template	no	app/pages/timeseries_template.py L1761-1762, L1811-1812 log run_metadata only
AutoML	no	automl_logging_operations.py L454-491; the target appears as "target" inside the model card JSON, never as a run parameter

So a gate-6 rejection naming target_variable means the origin run came from the pipeline or AutoML path. The remedy is to re-export from a run trained by the time-series template, which is what the importer's IMPORT_BUNDLE_INCOMPLETE sentence already asks for.

---

4. The file on disk

offset  width  field                 constraint the reader enforces
------  -----  --------------------  ------------------------------------------------------
     0      8  magic                 == b'SIENTIA1'
     8      1  format_version        == 1
     9      1  aead_id               == 1 (XChaCha20-Poly1305 secretstream)
    10      1  kdf_id                == 1 (Argon2id)
    11     16  salt                  passed to the KDF as-is
    27      8  kdf_memlimit_bytes    <= 268435456
    35      8  kdf_opslimit          <= 3
    43      1  kdf_parallelism       == 1
    44     32  ciphertext_digest     == sha256(file[164:])
    76     64  reserved_signature    all zero
------  -----
   140     24  secretstream header   libsodium HEADERBYTES
   164    ...  chunk stream          64 KiB plaintext chunks, each +17 bytes of AEAD tag

All integers are big-endian; the format string starts with >, so there is no alignment padding and the offsets above are exact.

The digest boundary. The producer computes hashlib.sha256(ciphertext) where ciphertext is the concatenation of the pushed chunks only, and then writes header + stream_header + ciphertext. On disk, therefore, the digest covers everything after offset 164, not after 140:

ciphertext_digest == sha256(file[HEADER_SIZE + STREAM_HEADER_SIZE:])

A reader that hashes from 140 will reject every valid bundle. The fixture test that recomputes both values and asserts the first matches while the second does not is what pins this, and is the reason the golden fixture exists.

Version policy. format_version is checked for equality, not for a range: a version this reader does not know is refused at gate 2 rather than parsed optimistically. reserved_signature is reserved for a future signature and must be all zero today, so a signed bundle is a new format version and not a silent change of meaning.

5. The seven gates

BundleReader.open() runs them in this order, in one function, so the order cannot be changed by a caller. Each raises BundleGateError(gate=n), which the activity layer maps to an import step.

Gate	Step recorded	What it checks	Written to disk
1	download	object size ceiling, key prefix, .sientia suffix — from stat_object, before the body is fetched; then the SHA-256 of the downloaded bytes against the digest the uploader computed	the downloaded object only
2	download	the whole header table of §4, including the chunk-stream digest	nothing
3	decryption	Argon2id derivation with the header's parameters, then secretstream pull. The AEAD is the integrity check: a wrong password and an altered file fail identically, and the message says so	the decrypted zip, in the work directory
4	archive_inspection	central directory only: entry count, total uncompressed size, compression ratio, absolute names, .. components, symlink and non-regular modes	nothing
5	extraction	extraction into a fresh directory (a pre-existing one is a failure), every member's realpath contained in it, no overwrite of a pre-existing file; a rejection removes its own partial tree	the extracted tree
6	structure_validation	metadata.json + artifacts/ layout, exactly two top-level keys, all six metadata fields present and non-blank, experiment_name between 3 and 50 characters, parameters.target_variable present and non-blank, and prediction_model/data_model each holding MLmodel and model.pkl	nothing
7	content_policy	every file is either a known name (MLmodel, metadata.json, model.pkl, conda.yaml, python_env.yaml, requirements.txt, model_card.json, model_card.svg) or a known suffix (.csv .json .yaml .yml .txt .pkl .svg .md)	nothing

Gates 1 and 2 share the download step deliberately: an oversized object, a wrong magic, an unsupported version and a digest that disagrees all say these are not the bytes that were sent, and none of them may borrow the decryption sentence, which implicates the password.

The experiment_name bound in gate 6 is not cosmetic: the import record's experiment_name column is VARCHAR(50), and the name is written onto the record before provisioning starts. Enforcing the column's limit at the gate is what keeps it from surfacing three activities later as a constraint violation.

6. Ceilings and where they come from

Gate 1 and gate 4 read their limits from build_import_config(), so a deployment can tighten them:

Variable	Default	Gate
IMPORT_MAX_OBJECT_BYTES	1 GiB	1
IMPORT_MAX_ARCHIVE_ENTRIES	5000	4
IMPORT_MAX_UNCOMPRESSED_BYTES	4 GiB	4
IMPORT_MAX_COMPRESSION_RATIO	200	4
IMPORT_BUNDLE_PREFIX	imported_models/	1

The KDF ceilings are not configurable: they are the producer's own cost parameters, and a header declaring more is a resource-exhaustion attempt, refused before any memory is allocated.

7. Fixtures

• tests/fixtures/bundle/golden_model_v3.sientia — emitted by the producer's own code in its venv, with the password and producer commit recorded in tests/fixtures/bundle/README.md. Regenerate it with tests/fixtures/bundle/regenerate.py. It pays the producer's real Argon2id cost (256 MiB, 3 passes), so it is used where the real thing matters and not in every test.
• tests/helpers/bundle_factory.py — the test-only writer. Same header fields at the same offsets, but 8 MiB and 1 pass, and with a knob for every hostile variant: wrong magic, unknown version, KDF parameters above the ceiling, digest mismatch, tampered ciphertext, truncated final chunk, non-zero reserved signature, zip bomb, .. entry, absolute entry, symlink, missing data_model, missing MLmodel, missing model.pkl, malformed metadata, missing target_variable, extra .sh file. A test asserts its header bytes are structurally identical to the golden fixture's.