vitor-aignosi
0e445de386
QTZPOC-20: Implement .sientia model bundle import workflow Introduces a new Temporal workflow and activities to import .sientia model bundles. This enables: - Secure decryption and validation of bundles through a seven-gate process. - Provisioning imported models in MLflow and updating the MongoDB model listing. - Comprehensive status tracking in the public.experiment_run table with new IMPORT_SUCCESS/IMPORT_ERROR states. - Dedicated metrics, robust error handling, and configurable settings. - Includes updated e2e database fixtures and extensive unit/integration tests for bundle format, reader, and overall workflow.
4.1 KiB
Golden .sientia fixtures
Binary conformance vectors for laborious/utils/bundle/. They are bytes the producer's own code
emitted, not bytes this repository knows how to write: there is no production writer here (change
import-model-workflow, design D1). If the producer's layout moves, the reader's tests fail against
these files instead of failing in production.
| File | What it is |
|---|---|
golden_model_v3.sientia |
format_version 1 bundle, 110767 bytes, plaintext zip 110569 bytes — two chunks, so MESSAGE/FINAL framing is exercised by the golden fixture itself |
golden_model_v3.metadata.json |
the metadata.json that went into that bundle, in plaintext, so a test can assert the recovered value without decrypting first |
regenerate.py |
the script that produced them, importing the producer's bundle.py/crypto.py by path |
Password
Import1Model!Fixture
Recorded deliberately: the fixture protects nothing. It satisfies the producer's export-side
strength rule (≥ 12 characters, upper, lower, digit, symbol), which is checked by
crypto.validate_password_strength at generation time.
Provenance
- Producer repository:
sientia-projects-templates - Producer commit:
8d3c1fd117c697e8b352c34f11c732809a4a129b(8d3c1fd QTZPOC-7: adapt to lint rules picked up from merging main) - Producer modules used:
app/src/operations/model_export/bundle.py,app/src/operations/model_export/crypto.py— loaded by path, so every constant, thestructlayout, the zip layout, the KDF parameters and the chunk framing come from the producer and none of them is re-implemented here - Producer pin honoured:
pynacl==1.5.0(this repository pins the same version)
One deviation from task 0.3, recorded rather than hidden: the bytes were generated with the
producer's code under this repository's venv, not inside the producer's own venv, because that
venv has neither pynacl nor mlflow installed. The dependency that matters for the bytes is
pynacl, pinned identically on both sides, and mlflow is not reached — regenerate.py builds the
artifact tree itself instead of calling client.download_artifacts.
Contents of the bundle
metadata.json (written first, no indent, by bundle.build_zip)
artifacts/prediction_model/MLmodel
artifacts/prediction_model/model.pkl (90 KiB of incompressible filler)
artifacts/prediction_model/conda.yaml
artifacts/prediction_model/python_env.yaml
artifacts/prediction_model/requirements.txt
artifacts/data_model/MLmodel
artifacts/data_model/model.pkl (8 KiB)
artifacts/data_model/conda.yaml
artifacts/data_model/python_env.yaml
artifacts/data_model/requirements.txt
artifacts/model_card/model_card.json
artifacts/train_data.csv
artifacts/test_data.csv
artifacts/raw_data.csv
No directory entries are stored — the producer only adds paths for which is_file() is true.
parameters carries target_variable: "SE", which gate 6 requires and which becomes
model_config.target in the MongoDB model document.
Digest boundary — the reason this fixture exists
The header's ciphertext_digest is sha256(file[164:]): the producer computes
hashlib.sha256(ciphertext) over the joined chunks and only then writes
header + stream_header + ciphertext, so the 24-byte secretstream header is not covered.
Verified against this fixture and against a freshly produced bundle. Earlier prose (this change's own
first draft, and aig-docs/03-sientia-bundle-format.md) said "everything after the plaintext
header", i.e. from offset 140 — that is wrong, and tests/laborious/utils/bundle/test_format.py
asserts both directions so it cannot come back.
Regenerating
source ./venv/bin/activate
PRODUCER_REPO=../sientia-projects-templates python tests/fixtures/bundle/regenerate.py
Two fields change on every run and are expected to: the 16-byte salt (os.urandom) and
metadata.metadata.export_timestamp (datetime.now(UTC)). Everything else is deterministic —
the filler bytes come from a seeded PRNG. After regenerating, update the producer commit above and
the byte counts in the table.