vitor-aignosi
0e445de386
QTZPOC-20: Implement .sientia model bundle import workflow Introduces a new Temporal workflow and activities to import .sientia model bundles. This enables: - Secure decryption and validation of bundles through a seven-gate process. - Provisioning imported models in MLflow and updating the MongoDB model listing. - Comprehensive status tracking in the public.experiment_run table with new IMPORT_SUCCESS/IMPORT_ERROR states. - Dedicated metrics, robust error handling, and configurable settings. - Includes updated e2e database fixtures and extensive unit/integration tests for bundle format, reader, and overall workflow.
87 lines
4.1 KiB
Markdown
87 lines
4.1 KiB
Markdown
# Golden `.sientia` fixtures
|
|
|
|
Binary conformance vectors for `laborious/utils/bundle/`. They are **bytes the producer's own code
|
|
emitted**, not bytes this repository knows how to write: there is no production writer here (change
|
|
`import-model-workflow`, design D1). If the producer's layout moves, the reader's tests fail against
|
|
these files instead of failing in production.
|
|
|
|
| File | What it is |
|
|
|---|---|
|
|
| `golden_model_v3.sientia` | `format_version` 1 bundle, 110767 bytes, plaintext zip 110569 bytes — **two chunks**, so `MESSAGE`/`FINAL` framing is exercised by the golden fixture itself |
|
|
| `golden_model_v3.metadata.json` | the `metadata.json` that went into that bundle, in plaintext, so a test can assert the recovered value without decrypting first |
|
|
| `regenerate.py` | the script that produced them, importing the producer's `bundle.py`/`crypto.py` by path |
|
|
|
|
## Password
|
|
|
|
```
|
|
Import1Model!Fixture
|
|
```
|
|
|
|
Recorded deliberately: the fixture protects nothing. It satisfies the producer's export-side
|
|
strength rule (≥ 12 characters, upper, lower, digit, symbol), which is checked by
|
|
`crypto.validate_password_strength` at generation time.
|
|
|
|
## Provenance
|
|
|
|
- Producer repository: `sientia-projects-templates`
|
|
- Producer commit: `8d3c1fd117c697e8b352c34f11c732809a4a129b`
|
|
(`8d3c1fd QTZPOC-7: adapt to lint rules picked up from merging main`)
|
|
- Producer modules used: `app/src/operations/model_export/bundle.py`,
|
|
`app/src/operations/model_export/crypto.py` — loaded by path, so every constant, the `struct`
|
|
layout, the zip layout, the KDF parameters and the chunk framing come from the producer and none of
|
|
them is re-implemented here
|
|
- Producer pin honoured: `pynacl==1.5.0` (this repository pins the same version)
|
|
|
|
One deviation from task 0.3, recorded rather than hidden: the bytes were generated with the
|
|
producer's **code** under *this* repository's venv, not inside the producer's own venv, because that
|
|
venv has neither `pynacl` nor `mlflow` installed. The dependency that matters for the bytes is
|
|
`pynacl`, pinned identically on both sides, and `mlflow` is not reached — `regenerate.py` builds the
|
|
artifact tree itself instead of calling `client.download_artifacts`.
|
|
|
|
## Contents of the bundle
|
|
|
|
```
|
|
metadata.json (written first, no indent, by bundle.build_zip)
|
|
artifacts/prediction_model/MLmodel
|
|
artifacts/prediction_model/model.pkl (90 KiB of incompressible filler)
|
|
artifacts/prediction_model/conda.yaml
|
|
artifacts/prediction_model/python_env.yaml
|
|
artifacts/prediction_model/requirements.txt
|
|
artifacts/data_model/MLmodel
|
|
artifacts/data_model/model.pkl (8 KiB)
|
|
artifacts/data_model/conda.yaml
|
|
artifacts/data_model/python_env.yaml
|
|
artifacts/data_model/requirements.txt
|
|
artifacts/model_card/model_card.json
|
|
artifacts/train_data.csv
|
|
artifacts/test_data.csv
|
|
artifacts/raw_data.csv
|
|
```
|
|
|
|
No directory entries are stored — the producer only adds paths for which `is_file()` is true.
|
|
|
|
`parameters` carries `target_variable: "SE"`, which gate 6 requires and which becomes
|
|
`model_config.target` in the MongoDB model document.
|
|
|
|
## Digest boundary — the reason this fixture exists
|
|
|
|
The header's `ciphertext_digest` is `sha256(file[164:])`: the producer computes
|
|
`hashlib.sha256(ciphertext)` over the joined chunks and only then writes
|
|
`header + stream_header + ciphertext`, so the 24-byte `secretstream` header is **not** covered.
|
|
Verified against this fixture and against a freshly produced bundle. Earlier prose (this change's own
|
|
first draft, and `aig-docs/03-sientia-bundle-format.md`) said "everything after the plaintext
|
|
header", i.e. from offset 140 — that is wrong, and `tests/laborious/utils/bundle/test_format.py`
|
|
asserts both directions so it cannot come back.
|
|
|
|
## Regenerating
|
|
|
|
```bash
|
|
source ./venv/bin/activate
|
|
PRODUCER_REPO=../sientia-projects-templates python tests/fixtures/bundle/regenerate.py
|
|
```
|
|
|
|
Two fields change on every run and are expected to: the 16-byte salt (`os.urandom`) and
|
|
`metadata.metadata.export_timestamp` (`datetime.now(UTC)`). Everything else is deterministic —
|
|
the filler bytes come from a seeded PRNG. After regenerating, update the producer commit above and
|
|
the byte counts in the table.
|