gitea_admin/sientia-dataops-laborious-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

QTZPOC-20

Browse Source 
QTZPOC-20: Unify artifact code directory handling for all model flavors and standardize joblib pickle path

Implemented a robust mechanism to ensure an artifact's code/ directory is prepended to sys.path before loading for sklearn, pyfunc, and joblib flavors. sklearn and pyfunc now resolve MLflow URIs to local directories first to facilitate this sys.path modification. Standardized the joblib flavor to explicitly read and write model.pkl within the artifact directory. Updated documentation and added comprehensive tests to reflect these changes.
This commit is contained in:
Vitor Pimentel dos Santos
2026-08-28 15:10:41 -03:00
parent f4c110cd8a
commit c1a44a9d7a
6 changed files with 475 additions and 273 deletions
Download Patch File Download Diff File Expand all files Collapse all files

224
docs/sientia-bundle-format.md
View File

@@ -1,111 +1,48 @@
# The `.sientia` bundle format
A `.sientia` file is one exported MLflow model: its whole artifact tree, its run parameters and six
fields of origin metadata, zipped and then encrypted under a password the exporter chose. It is
written by the Streamlit platform (`sientia-projects-templates`) and read by this repository's
importer. Two programs in two repositories agree on these bytes, so the constants below are
duplicated by necessity — the drift is held in place by tests on both sides, not by a shared library.
A `.sientia` file is one exported MLflow model: its whole artifact tree, its run parameters and its
origin metadata, zipped and then encrypted under a password the exporter chose.
This document covers both directions: **§1–§3 how a bundle is generated**, so the producer can be
audited against it, and **§4–§7 how it is read**, so a rejection can be explained.
**It is written by the Streamlit platform, `sientia-projects-templates`** (`app/src/operations/
model_export/`). That application owns the export — its screen, its flow, its code — and documents
it. This page is the short version: the parameters both sides have to agree on, and what this
repository's reader accepts.
Normative implementations:
## How it is generated
| Side | Where |
The producer downloads the run's artifact tree whole, copies `run.data.params` **verbatim** (with
`target_variable` among them, or the import fails at gate 6), writes `metadata.json` first into a
`ZIP_DEFLATED` archive with the artifacts following as `artifacts/<relpath>` and no directory
entries, encrypts that zip, and offers it as `{model_name}_v{model_version}.sientia`.
`metadata.json` has exactly two keys: `parameters` — the run's parameters verbatim — and `metadata`,
six fields: `model_name`, `experiment_name`, `run_id`, `model_version`, `model_project`,
`export_timestamp` (UTC ISO 8601).
## Encryption parameters
The reader derives the key with the values the header carries, but refuses anything above the
producer's own cost — a header declaring more is a resource-exhaustion attempt, refused before any
memory is allocated.
| | Value |
|---|---|
| Writer | `sientia-projects-templates`, `app/src/operations/model_export/{crypto,bundle,exporter}.py` |
| Reader | this repository, `laborious/utils/bundle/{format,reader}.py` |
| Test writer | this repository, `tests/helpers/bundle_factory.py` — mirrors the producer field for field, for building hostile variants |
| Golden fixture | `tests/fixtures/bundle/` — a real producer artefact, with its password and producer commit recorded in that directory's `README.md` |
| KDF | Argon2id (`crypto_pwhash_ALG_ARGON2ID13`) |
| memlimit | `268435456` (256 MiB) — a **ceiling** on the reader's side |
| opslimit | `3` — likewise a ceiling |
| parallelism | `1` (reserved, not configurable) |
| salt | 16 bytes, fresh per bundle |
| key | 32 bytes |
| AEAD | XChaCha20-Poly1305 `secretstream`, 64 KiB plaintext chunks, last tagged `FINAL` |
| digest | SHA-256 of the **pushed chunks only** — on disk, `sha256(file[164:])`, past the 140-byte header and the 24-byte stream header. Hashing from 140 rejects every valid bundle |
| password | normalised `NFC`, encoded UTF-8 — no trimming, no case folding |
---
The export-side password rule (12+ characters, mixed case, digit, symbol) is enforced there only: by
the time a bundle exists, the password is whatever it was, and the reader cannot re-check it.
## 1. Generation, step by step
## Shape of the file
Entry point: `exporter.export_model_bundle(*, client, run_id, model_name, experiment_name,
model_version, model_project, password) -> bytes`.
1. **Password strength** — `crypto.validate_password_strength`: at least 12 characters, one
uppercase, one lowercase, one digit, one symbol. **Export-side only.** The importer cannot
re-check it and does not try: by the time a bundle exists, the password is whatever it was.
2. **Exportability** — `bundle.validate_run_is_exportable`: the run's top-level artifacts must
include **both** `data_model` and `prediction_model`, checked with `client.list_artifacts(run_id)`
before anything is downloaded.
3. **Whole-tree download** — `client.download_artifacts(run_id, '', staging)` into a
`tempfile.mkdtemp(prefix='sientia_export_')` directory. The entire artifact tree, not a subset.
4. **Run parameters** — `dict(run.data.params)`, copied **verbatim**. Whatever the run logged is
what the bundle carries, including `target_variable` (see §3).
5. **Metadata document** — `bundle.build_metadata_json`, exactly two top-level keys:
- `parameters`: the run parameters, verbatim;
- `metadata`: `model_name`, `experiment_name`, `run_id`, `model_version`, `model_project` and
`export_timestamp = datetime.now(UTC).isoformat()`. Six fields, no others.
6. **Zip** — `bundle.build_zip`, `zipfile.ZIP_DEFLATED`. `metadata.json` is written **first**, with
`json.dumps(...)` and no indent. Every file under the staging directory follows as
`artifacts/<relpath>`, with `\` replaced by `/`. **No directory entries are stored**, so a reader
must derive directories from file paths.
7. **Encryption** — `crypto.encrypt_archive`: a fresh 16-byte `os.urandom` salt, Argon2id key
derivation, then a libsodium `secretstream` push in 64 KiB plaintext chunks, the last chunk tagged
`FINAL` and the rest `MESSAGE`.
8. **Download name** — `{model_name}_v{model_version}.sientia`. The staging directory is removed in a
`finally`, and the password is popped from the Streamlit session state.
Producer pins: `pynacl==1.5.0`, `mlflow==2.10.1`.
## 2. Constants
Copied from `crypto.py` unless noted. The reader's names are the ones in
`laborious/utils/bundle/format.py`.
| Producer | Value | Reader |
|---|---|---|
| `MAGIC` | `b'SIENTIA1'` | `MAGIC` |
| `_HEADER_STRUCT` | `struct.Struct('>8sBBB16sQQB32s64s')` | `HEADER_STRUCT` |
| `HEADER_STRUCT_SIZE` | `140` | `HEADER_SIZE` |
| `FORMAT_VERSION` | `1` | `FORMAT_VERSION` |
| `AEAD_ID_XCHACHA20POLY1305_SECRETSTREAM` | `1` | same name |
| `KDF_ID_ARGON2ID` | `1` | same name |
| `SALT_SIZE` | `16` | `SALT_SIZE` |
| `KDF_MEMLIMIT_BYTES` | `268435456` (256 MiB) | `MAX_KDF_MEMLIMIT` — a **ceiling** on the reader's side |
| `KDF_OPSLIMIT` | `3` | `MAX_KDF_OPSLIMIT` — likewise a ceiling |
| `KDF_PARALLELISM_RESERVED` | `1` | `PARALLELISM` |
| `DIGEST_SIZE` | `32` | `DIGEST_SIZE` |
| `RESERVED_SIGNATURE_SIZE` | `64` | `RESERVED_SIGNATURE_SIZE` |
| `CHUNK_SIZE` | `65536` (64 KiB) | `CHUNK_SIZE` |
| KDF algorithm | `crypto_pwhash_ALG_ARGON2ID13` | `KDF_ALG` |
| Key length | `crypto_secretstream_xchacha20poly1305_KEYBYTES` (32) | `KEY_SIZE` |
Fixed by libsodium rather than by the producer, but needed to frame the stream:
- `crypto_secretstream_xchacha20poly1305_HEADERBYTES == 24` → `STREAM_HEADER_SIZE`
- `crypto_secretstream_xchacha20poly1305_ABYTES == 17` → `ABYTES`
- pre-upload prefix `140 + 24 + (65536 + 17) = 65717` → `PREFIX_SIZE`, the smallest number of leading
bytes from which a header and one full chunk can be checked
**Password normalisation.** `_derive_key` applies `unicodedata.normalize('NFC', password)` and
encodes UTF-8 — no trimming, no case folding, no padding. The reader normalises identically, so a
password typed in decomposed form opens a bundle written from the composed form.
## 3. `target_variable`, and where it does not come from
Gate 6 requires `parameters.target_variable`: it is the only source of `model_config.target` in the
model document, and every workflow that serves a model indexes that key directly. On the producer
side:
| Export path | Logs `target_variable`? | Evidence |
|---|---|---|
| Time-series training template | yes | `app/pages/template_01.py` L1212-1214 |
| Model manager, "Save Experiment" | yes, if the session carried it | `app/pages/model_manager.py` L791-796 re-logs `st.session_state.parameters` verbatim |
| Model ensemble | same condition | `app/pages/model_ensemble.py` L695-700 |
| Pipeline / timeseries template | **no** | `app/pages/timeseries_template.py` L1761-1762, L1811-1812 log `run_metadata` only |
| AutoML | **no** | `automl_logging_operations.py` L454-491; the target appears as `"target"` inside the model card JSON, never as a run parameter |
So a gate-6 rejection naming `target_variable` means the origin run came from the pipeline or AutoML
path. The remedy is to re-export from a run trained by the time-series template, which is what the
importer's `IMPORT_BUNDLE_INCOMPLETE` sentence already asks for.
---
## 4. The file on disk
`HEADER_STRUCT` is `>8sBBB16sQQB32s64s`. Big-endian, no alignment padding, so the offsets are exact:
```
offset width field constraint the reader enforces
@@ -125,72 +62,39 @@ offset width field constraint the reader enforces
164 ... chunk stream 64 KiB plaintext chunks, each +17 bytes of AEAD tag
```
All integers are big-endian; the format string starts with `>`, so there is no alignment padding and
the offsets above are exact.
`format_version` is checked for equality, not for a range: an unknown version is refused at gate 2
rather than parsed optimistically. `reserved_signature` must be all zero today, so a signed bundle
will be a new format version and not a silent change of meaning.
**The digest boundary.** The producer computes `hashlib.sha256(ciphertext)` where `ciphertext` is the
concatenation of the *pushed chunks only*, and then writes `header + stream_header + ciphertext`. On
disk, therefore, the digest covers everything after offset **164**, not after 140:
`laborious/utils/bundle/format.py` is the normative layout on this side, and the golden fixture is
what keeps it and the producer from drifting apart.
> `ciphertext_digest == sha256(file[HEADER_SIZE + STREAM_HEADER_SIZE:])`
## How it is read
A reader that hashes from 140 will reject every valid bundle. The fixture test that recomputes both
values and asserts the first matches while the second does not is what pins this, and is the reason
the golden fixture exists.
`BundleReader.open()` runs seven ordered gates, each mapped to an import step and to one message a
person can read:
**Version policy.** `format_version` is checked for equality, not for a range: a version this reader
does not know is refused at gate 2 rather than parsed optimistically. `reserved_signature` is
reserved for a future signature and must be all zero today, so a signed bundle is a new format
version and not a silent change of meaning.
## 5. The seven gates
`BundleReader.open()` runs them in this order, in one function, so the order cannot be changed by a
caller. Each raises `BundleGateError(gate=n)`, which the activity layer maps to an import step.
| Gate | Step recorded | What it checks | Written to disk |
|---|---|---|---|
| 1 | `download` | object size ceiling, key prefix, `.sientia` suffix — from `stat_object`, before the body is fetched; then the SHA-256 of the downloaded bytes against the digest the uploader computed | the downloaded object only |
| 2 | `download` | the whole header table of §4, including the chunk-stream digest | nothing |
| 3 | `decryption` | Argon2id derivation with the header's parameters, then `secretstream` pull. The AEAD is the integrity check: a wrong password and an altered file fail identically, and the message says so | the decrypted zip, in the work directory |
| 4 | `archive_inspection` | central directory only: entry count, total uncompressed size, compression ratio, absolute names, `..` components, symlink and non-regular modes | nothing |
| 5 | `extraction` | extraction into a **fresh** directory (a pre-existing one is a failure), every member's `realpath` contained in it, no overwrite of a pre-existing file; a rejection removes its own partial tree | the extracted tree |
| 6 | `structure_validation` | `metadata.json` + `artifacts/` layout, exactly two top-level keys, all six metadata fields present and non-blank, `experiment_name` between 3 and 50 characters, `parameters.target_variable` present and non-blank, and `prediction_model`/`data_model` each holding `MLmodel` and `model.pkl` | nothing |
| 7 | `content_policy` | every file is either a known name (`MLmodel`, `metadata.json`, `model.pkl`, `conda.yaml`, `python_env.yaml`, `requirements.txt`, `model_card.json`, `model_card.svg`) or a known suffix (`.csv .json .yaml .yml .txt .pkl .svg .md`) | nothing |
Gates 1 and 2 share the `download` step deliberately: an oversized object, a wrong magic, an
unsupported version and a digest that disagrees all say *these are not the bytes that were sent*, and
none of them may borrow the `decryption` sentence, which implicates the password.
The `experiment_name` bound in gate 6 is not cosmetic: the import record's `experiment_name` column is
`VARCHAR(50)`, and the name is written onto the record before provisioning starts. Enforcing the
column's limit at the gate is what keeps it from surfacing three activities later as a constraint
violation.
## 6. Ceilings and where they come from
Gate 1 and gate 4 read their limits from `build_import_config()`, so a deployment can tighten them:
| Variable | Default | Gate |
| Gate | Step | Refuses |
|---|---|---|
| `IMPORT_MAX_OBJECT_BYTES` | 1 GiB | 1 |
| `IMPORT_MAX_ARCHIVE_ENTRIES` | 5000 | 4 |
| `IMPORT_MAX_UNCOMPRESSED_BYTES` | 4 GiB | 4 |
| `IMPORT_MAX_COMPRESSION_RATIO` | 200 | 4 |
| `IMPORT_BUNDLE_PREFIX` | `imported_models/` | 1 |
| 1 | `download` | object too large, wrong prefix or suffix, digest ≠ the one the uploader sent |
| 2 | `download` | header that is not this format or not this version |
| 3 | `decryption` | wrong password, or altered bytes — the AEAD cannot tell them apart, and the message says so |
| 4 | `archive_inspection` | zip bomb, absolute or `..` entries, symlinks — read from the central directory, before extracting |
| 5 | `extraction` | anything that escapes the fresh work directory, or overwrites a file |
| 6 | `structure_validation` | missing `metadata.json` fields, missing `target_variable`, missing `prediction_model`/`data_model` |
| 7 | `content_policy` | a file that is not a known model file by name or suffix |
The KDF ceilings are **not** configurable: they are the producer's own cost parameters, and a header
declaring more is a resource-exhaustion attempt, refused before any memory is allocated.
Gates 1 and 2 share the `download` step on purpose: neither may borrow the `decryption` sentence,
which implicates the password. A gate-6 rejection naming `target_variable` means the origin run never
logged it — the fix is a re-export, on the producer's side.
## 7. Fixtures
Error codes, sentences and the ceilings the gates read from configuration:
[`model-import.md`](model-import.md).
- `tests/fixtures/bundle/golden_model_v3.sientia` — emitted by the producer's own code in its venv,
with the password and producer commit recorded in `tests/fixtures/bundle/README.md`. Regenerate it
with `tests/fixtures/bundle/regenerate.py`. It pays the producer's real Argon2id cost (256 MiB,
3 passes), so it is used where the real thing matters and not in every test.
- `tests/helpers/bundle_factory.py` — the test-only writer. Same header fields at the same offsets,
but 8 MiB and 1 pass, and with a knob for every hostile variant: wrong magic, unknown version, KDF
parameters above the ceiling, digest mismatch, tampered ciphertext, truncated final chunk, non-zero
reserved signature, zip bomb, `..` entry, absolute entry, symlink, missing `data_model`, missing
`MLmodel`, missing `model.pkl`, malformed metadata, missing `target_variable`, extra `.sh` file. A
test asserts its header bytes are structurally identical to the golden fixture's.
## Fixtures
- `tests/fixtures/bundle/` — a real bundle emitted by the producer, with its password and producer
commit recorded in that directory's `README.md`. It pays the producer's real Argon2id cost, so it
is used where the real bytes matter and not in every test.
- `tests/helpers/bundle_factory.py` — the test-only writer: same header at the same offsets, cheap
KDF parameters, and a knob for every hostile variant the gates above refuse.