sientia-dataops-laborious_temporal/docs/model-import.md

Importing a model (import_model)

The import_model workflow turns an encrypted .sientia object in MinIO into a registered MLflow model plus one document in the MongoDB models collection — or into a row that says, in a sentence a person can read, why it did not.

This document is the contract for the two sides that talk to it: the frontend, which uploads the object and builds the request, and the BFF, which owns the import log row. Everything here is enforced by code in this repository; where a statement is about a column or a constraint, the authority is the migration named in §7.

Format of the file itself: sientia-bundle-format.md.

---

1. What the caller does, in order

1. Upload the .sientia object to the import bucket, under the configured prefix (default imported_models/). The key must end in .sientia.
2. Compute expected_digest — the SHA-256, lowercase hex, of the exact bytes that were uploaded. Not of a re-read, not of a re-serialisation: of what went over the wire.
3. Encrypt the bundle password into a password_envelope (§3).
4. Insert the import log row (§5) and take its id.
5. Start the workflow on import_model-{RUNTIME}-queue with the input of §2.

The workflow claims the row, opens the bundle, provisions MLflow, writes the model document, cleans up and records one terminal verdict. It never inserts a row, never promotes a version and never activates a model.

2. The workflow input

{
  "import_run_id": 41,
  "object_key": "imported_models/sales_forecast_v3.sientia",
  "expected_digest": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
  "password_envelope": "Base64OfNonceCiphertextTag==",
  "bucket": "sientia"
}

Field	Required	Meaning
import_run_id	yes	primary key of the public.experiment_run row the BFF inserted. Must be a positive integer
object_key	yes	the uploaded object's key. Must match the row's file_name when the row carries one
expected_digest	yes	SHA-256 of the uploaded bytes, 64 lowercase hex characters. There is no path that skips the comparison: a missing value fails at received, a mismatching one at download, before the header is parsed
password_envelope	yes	the bundle password, encrypted (§3). A plaintext password field is refused
bucket	no	defaults to IMPORT_BUNDLE_BUCKET

Fields that are refused rather than ignored: project_id, project_name, provisioning_target, schema, table_name, status_table, password. An input carrying any of them fails at received, so no caller can believe it recorded something the import does not record (see §10).

3. The password envelope

AES-256-GCM, and the ciphertext layout is what the WebCrypto API returns natively:

envelope = base64( nonce(12 bytes) || ciphertext || tag(16 bytes) )

crypto.subtle.encrypt({name: 'AES-GCM', iv: nonce}, key, plaintext) returns ciphertext || tag already concatenated, so the browser side is: generate 12 random bytes, encrypt, concatenate the nonce in front, base64 the result.

The key is a 32-byte AES-256 key shared with the worker through IMPORT_PASSWORD_KEY (base64). The worker decrypts the envelope inside the activity that uses the password, so no plaintext password is ever an activity input, an activity result or a workflow variable — and therefore never appears in the Temporal event history. A test asserts that for every failure path.

A failure to open the envelope is recorded as a decryption failure, distinct in the logs from a wrong bundle password but sharing its step, because the step says where the import stopped, not who is at fault.

4. Environment

Variable	Default	What it is
IMPORT_MODEL_ENABLED	unset	registers the fifth (import) worker. Truthy values: 1, true, yes, on
IMPORT_PASSWORD_KEY	—	base64 of the 32-byte AES-256-GCM key that wraps bundle passwords
IMPORT_BUNDLE_BUCKET	sientia	bucket holding the uploaded bundles
IMPORT_BUNDLE_PREFIX	imported_models/	key prefix the uploads must sit under
IMPORT_BUNDLE_RETENTION_DAYS	7	lifecycle expiry the importer applies to that prefix
IMPORT_WORK_DIR	<tempdir>/sientia-import	worker scratch directory
IMPORT_MAX_OBJECT_BYTES	1 GiB	gate 1
IMPORT_MAX_ARCHIVE_ENTRIES	5000	gate 4
IMPORT_MAX_UNCOMPRESSED_BYTES	4 GiB	gate 4
IMPORT_MAX_COMPRESSION_RATIO	200	gate 4
IMPORT_MODELS_COLLECTION	models	the MongoDB collection that is the model listing
IMPORT_STATUS_DB_NAME	sientia-core-mlops-bff	the BFF database holding the import log
IMPORT_STATUS_DB_HOST	POSTGRES_HOST	override: the log is on another server
IMPORT_STATUS_DB_PORT	POSTGRES_PORT	override
IMPORT_STATUS_DB_USER	POSTGRES_USER	override
IMPORT_STATUS_DB_PASSWORD	POSTGRES_PASSWORD	override

The import log is a second connection but the same server: host, port, user and password come from POSTGRES_*, and only the database name is import configuration, because the log lives in the BFF's database rather than in the one Laborious holds for predictions. The import writes to that database and no other, and to one table in it. The four IMPORT_STATUS_DB_* connection variables are overrides for a deployment whose log genuinely sits elsewhere — the e2e suite, which runs the two databases as two containers, is exactly that case.

A missing or malformed IMPORT_PASSWORD_KEY is an import failure at the decryption step, not a boot failure: the worker starts either way.

5. The import log row — the BFF's half

The row is public.experiment_run in sientia-core-mlops-bff. The workflow never inserts it. The inserting side writes:

Column	Value
run_type	'IMPORT'
status	'PENDING'
username	whoever asked for the import
bucket_name	the bucket the object was uploaded to
file_name	the object key
request_data	the request payload. If it carries a digest under expected_digest, digest, sha256 or file_digest, the workflow checks it against the input's

Then it passes the returned id as import_run_id.

The workflow's first act is to claim that row:

UPDATE public.experiment_run
   SET status = 'ORCHESTRATOR_WAITING_PROC', updated_at = :now,
       orchestrator_response_data = :detail
 WHERE id = :import_run_id
   AND run_type = 'IMPORT'
   AND (status = 'PENDING'
        OR (status = 'ORCHESTRATOR_WAITING_PROC'
            AND orchestrator_response_data -> 'import' ->> 'workflow_id' = :workflow_id))

The second disjunct makes a re-execution of the same workflow idempotent instead of a conflict. Every later write keys on id alone, because ownership was settled here.

Columns the workflow never writes: username, request_data, bucket_name, file_name, run_type, created_at. Columns it does write: status, error_message, updated_at, orchestrator_response_data, and — as soon as the bundle is open, before anything exists in MLflow — experiment_name (the bundle's own, verbatim) and run_name (import-{model_name}-v{model_version}, truncated inside the name segment to fit 50 characters). Those two are overwritten by the workflow; whatever the inserting side put there is replaced.

When the row cannot be claimed

If the claim matches nothing — no such id, a run_type='TRAIN' row, or a row already running under another workflow — the import writes nothing anywhere, because the row belongs to someone else. Instead it:

• fails the Temporal run with error type ImportLogRowNotClaimableError and code IMPORT_LOG_ROW_NOT_CLAIMABLE, non-retryable;
• increments the import error metric and fires a notification;
• logs the id and the observed state (which of the three causes it was).

So a row left PENDING with no workflow_id in its detail JSON means the workflow never owned it. Closing that row out belongs to whichever side inserted it — by timeout, by a sweeper, by a person — however that side chooses. The importer deliberately does not guess.

6. Status vocabulary

Status	Written when	New in V12?
ORCHESTRATOR_WAITING_PROC	claimed, and while running	no — reused from the training flow
ORCHESTRATOR_VALIDATION_ERROR	failure at received … content_policy — the user can act on it	no — reused
IMPORT_SUCCESS	every step succeeded	yes
IMPORT_ERROR	failure at experiment_creation … model_document, or unclassified — only support can act on it	yes

The two reused values describe a stage, not a flow: what says the row is an import is run_type, not the status. There is deliberately no IMPORT_RUNNING; it would carry no information run_type does not already carry, and it would widen both the pairing CHECK and the frontend's switch for nothing.

error_message is the user-facing sentence, taken from the catalog in §8. On success it is written as NULL, never '' — the column carries a LENGTH(error_message) >= 3 check.

7. The migration this depends on

sientia-dataops-database-migrations, postgres/sientia-core-mlops-bff/migrations/V12__add_import_statuses_to_experiment_run.sql:

• adds IMPORT_SUCCESS and IMPORT_ERROR to chk_experiment_run_status;
• adds chk_experiment_run_terminal_status_run_type, pairing only the terminal values with their run_type (IMPORT_* on 'IMPORT', TRAINING_* on 'TRAIN') and leaving the shared ORCHESTRATOR_* values out of the rule;
• adds no columns at all.

V12 must be merged and applied before IMPORT_MODEL_ENABLED is set anywhere. Without it, an import provisions the model and then cannot record its verdict: the claim still succeeds (no new columns are needed for it), the old CHECK refuses IMPORT_SUCCESS, and the failure surfaces as IMPORT_STATUS_NOT_RECORDED on the first attempt — no retry wait — leaving the row in ORCHESTRATOR_WAITING_PROC with a fully provisioned model behind it.

8. orchestrator_response_data, and the error catalog

There is no column for the error code and none for the workflow identifier (decision D21). Both live at fixed keys in the JSON column the workflow already owns, and the whole import object is rewritten on every write, so the identifier the claim wrote is never dropped:

{
  "import": {
    "workflow_id": "import_model-41-01HW…",
    "code": "IMPORT_BUNDLE_INCOMPLETE",
    "step": "structure_validation",
    "gate": 6,
    "source": { "model_name": "sales_forecast", "experiment_name": "…", "…": "…" },
    "cleanup_failed": false
  }
}

• workflow_id — what a user quotes to support, and what the claim's idempotency disjunct reads.
• code — the frontend's i18n key. Absent on success.
• step — where the import stopped (the values of §9).
• gate — which of the seven gates rejected it, when one did.
• source — the bundle's own metadata block, once it is known.
• cleanup_failed — temporary files were left on the worker. Present on a successful import too: it never changes the status, because the user's model was imported.

Code → reason

error_message is exactly the sentence in this table; nothing is interpolated into it, and the exception's text — paths, table names, exception classes, tracebacks — reaches only the logs.

Code	Step	Sentence written to error_message
IMPORT_REQUEST_INVALID	received	The import could not be started because the request was incomplete or contradictory. Nothing was created. Submit the import again.
IMPORT_FILE_UNREADABLE	download	The uploaded file could not be read, or it is not the file that was sent. Upload it again.
IMPORT_BUNDLE_CANNOT_BE_OPENED	decryption	This file could not be opened: the password is wrong, or the file is damaged or was altered. Confirm the password with whoever exported the model, then try again.
IMPORT_BUNDLE_REJECTED	archive_inspection	The file's contents did not pass the platform's safety checks, so it was not opened. Export the model again from the origin platform.
IMPORT_BUNDLE_UNPACK_FAILED	extraction	The file could not be unpacked safely and nothing from it was kept. Export the model again from the origin platform.
IMPORT_BUNDLE_INCOMPLETE	structure_validation	This file is not a complete model export — part of what the platform needs is missing from it. Export the model again from the origin platform.
IMPORT_BUNDLE_UNEXPECTED_CONTENT	content_policy	The file contains something a model export should not contain, so it was rejected. Export the model again from the origin platform.
IMPORT_STORAGE_PREPARATION_FAILED	experiment_creation	The platform could not prepare a place to keep this model. The import stopped and no model was created. Contact support with this import's identifier.
IMPORT_MODEL_FILES_NOT_STORED	artifact_upload	The model's files could not be stored on this platform. The import stopped and the model is not available. Contact support with this import's identifier.
IMPORT_MODEL_NOT_REGISTERED	registration	The model's files were stored, but the model itself could not be registered, so it cannot be used. Contact support with this import's identifier.
IMPORT_MODEL_SETTINGS_NOT_SAVED	model_document	The model was registered, but the settings that tell the platform how to run it could not be saved. Contact support with this import's identifier.
IMPORT_UNEXPECTED_ERROR	—	The import stopped for an unexpected reason and no model was created. Contact support with this import's identifier.

Three codes never reach a row's error_message, because there is no row to write or no failure to report to a user. They exist in the logs, the metric and the notification only:

Code	When
IMPORT_LOG_ROW_NOT_CLAIMABLE	the row is not this workflow's (§5)
IMPORT_STATUS_NOT_RECORDED	the verdict itself could not be written; the log line carries the verdict it could not record
IMPORT_CLEANUP_INCOMPLETE	temporary files were left behind; also flagged as cleanup_failed in the detail

9. Steps

received, download, decryption, archive_inspection, extraction, structure_validation, content_policy, experiment_creation, artifact_upload, registration, model_document, cleanup. The first seven are the gates (see the format document); a failure in any of them is a ORCHESTRATOR_VALIDATION_ERROR. The next four are provisioning; a failure there is an IMPORT_ERROR. cleanup is never a reported failure.

10. The import knows nothing about projects

The model listing is the MongoDB models document. No migrated relational database in this platform holds a project table or a model table, nothing in this repository reads a project-to-model link, and the workflow input carries no project target — which is why naming one is refused rather than accepted and ignored. Whatever the import screen does with projects is the frontend's business.

The bundle's own metadata.model_project is logged as an MLflow parameter (source.model_project), as a record of where the model came from. It is not a link.

11. What MLflow ends up holding

Thing	Value
Experiment	metadata.experiment_name from the bundle, created if absent, reused if present
Run	always new, created with the run_name already written on the record
Artifacts	the bundle's whole artifacts/ tree, with prediction_model and data_model at the run's artifact root
Parameters	the bundle's parameters, verbatim, plus the origin metadata as source.* (including source.run_id, which is never dereferenced)
Registered version	created on demand; source is the run's resolved artifact URI with /prediction_model appended, e.g. mlflow-artifacts:/17/<run_id>/artifacts/prediction_model
Stage	untouched

The source shape matters and is easy to get wrong: the legacy loader extracts the run id as source.split('/')[2], which is the run id in the resolved artifact URI but the artifact path in a literal runs:/<run_id>/prediction_model. Registering with the unresolved URI breaks loading silently — nothing raises, the loader just returns the wrong string. This is the same value mlflow.register_model stores for the training flow, because it resolves the runs:/ URI before writing it; MlflowClient.create_model_version does not, so the import resolves it itself.

12. The model document

Written last, into the collection named by IMPORT_MODELS_COLLECTION (default models):

{
  "id": "31",
  "name": "sales_forecast",
  "active": false,
  "model_config": {
    "transform_flavor": "sklearn",
    "predict_flavor": "joblib",
    "retention_minutes": 60,
    "target": "SE"
  }
}

• id is str(max(int(id) for id in collection) + 1) — computed in Python, from the ids that are numbers. Never a count, never a lexicographic sort (which would answer "9" > "30"). Ids that are not decimal numbers are ignored for the maximum and counted in the log.
• name is metadata.model_name from the bundle, verbatim.
• active is false, always. No code path in the import writes or updates a document with active: true.
• target comes from parameters.target_variable; nothing else supplies it.
• The models collection carries a unique sparse index on id, created by the MongoDB baseline migration. A duplicate-key insert is reported as a model_document failure — no retry, no id recomputation.

A name already in the collection is left exactly as it is. The import writes nothing, modifies nothing, logs the skip and succeeds: active, target and the flavors of a model already on the platform are operator-owned.

13. After a successful import — the manual steps

A COMPLETED import is not a servable model. Stated plainly:

• the registered version is unstaged, and load_predict_model only sees a version once it is in stage Production;
• the model document is inactive (active: false).

So an operator must promote the version to Production and set active: true. This workflow never promotes and never activates — that runbook is QTZPOC-21's.

One more thing to know before serving an imported model: predict_flavor: 'joblib' is what the artifact actually is, and it loads, predicts and retrains — log_model writes that flavor too, as model.pkl + code/. The artifact it writes carries no MLmodel manifest, so an imported model stays readable only by this runtime, before and after a retrain. Full contract: model-flavors.md.

14. Retry, and what does not retry

Only the status writes retry, and only because UPDATE ... SET <fixed values> WHERE id leaves the row the same however many times it runs:

Call	Attempts	Why
claim, terminal write	10, 1 s → 30 s backoff (≈151 s envelope)	the verdict must land; the database being briefly unavailable is exactly what a retry fixes
progress writes, names	3, capped at 2 s	a hint nobody reads — the terminal write rewrites it — must not stall a working import
everything else	1	a repeated registry call or Mongo insert can write twice

ImportLogRowNotClaimableError and ImportStatusRejectedError are non-retryable: the first would read the same rows and reach the same conclusion, and the second means the statement itself was refused (a CHECK violation, an undefined column, a value too long), which a second attempt cannot change.

There is no rollback, no compensation, no resume and no reconciliation. A failed import stays failed; trying again is a new upload and a new row.

15. How this contract is verified

Two suites, and they prove different things.

Unit — pytest tests/, part of the standard gate (validate --project-name=laborious). Covers the bundle reader against committed golden bytes, every gate, the failure catalog, the status SQL and the workflow's ordering, all against fakes.

End to end — pytest e2e -m import_e2e, opt-in and outside testpaths. Runs the real workflow on a real worker against real MinIO, a real Postgres holding this document's public.experiment_run (with and without V12), a real MongoDB with the unique index on models.id, and a real mlflow server. Nothing on the import path is mocked. This is what proves the claims a fake cannot: that the CHECK constraints of § 6 accept every status written, that the registry source of § 11 resolves through the existing loader, that the V12 failure shape of § 7 is what it says, and that no plaintext password reaches the Temporal event history.

Requires Docker plus one one-time setup per machine:

echo "ryuk.disabled=true" >> ~/.testcontainers.properties

Without it every e2e test fails in fixture setup with a Docker mount error — see README.md § End-to-End Tests for why and what it costs.

Scenario catalogue: e2e/scenarios.md § 4. Requirement-to-test mapping: openspec/changes/import-model-e2e-suite/traceability.md.

Doing one import by hand, standing in for the frontend that does not exist yet: scripts/import_model_manual_run.py — see README.md § Running One Model Import by Hand.