gitea_admin/sientia-dataops-laborious-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

QTZPOC-20

Browse Source 
QTZPOC-20: Implement model import workflow for .sientia bundles

Introduces the import_model workflow and its associated activities to enable importing .sientia bundles into MLflow and MongoDB. This includes:

*   New core workflow and activity logic for handling bundle download, decryption, validation, MLflow provisioning (experiments, runs, registered versions), and MongoDB model document creation.
*   Comprehensive E2E and unit test suites, including a dedicated test harness and a manual run script for development.
*   Updated .env.example and README.md with new configurations and usage instructions.
*   Ensures correct MLflow model source resolution for legacy loader compatibility and robust MongoDB document management.
*   Refactors existing utility scripts into a scripts/ directory.
This commit is contained in:
Vitor Pimentel dos Santos
2026-08-25 17:07:42 -03:00
parent 0e445de386
commit dac4f1718f
41 changed files with 8168 additions and 1045 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
.env.example
View File

@@ -32,4 +32,44 @@ MINIO_ENDPOINT_URL="http://localhost:9000"
MINIO_ACCESS_KEY="sientia"
MINIO_SECRET_KEY="sientia"
MINIO_REGION_NAME="sa-east-1"
MINIO_DEFAULT_BUCKET="sientia"
MINIO_DEFAULT_BUCKET="sientia"
# Suffix for every Temporal task queue name; required, non-empty.
# The import worker listens on "import_model-${RUNTIME}-queue".
RUNTIME="local"
# --- Model import (.sientia bundles) -----------------------------------------
# The import worker is registered only when IMPORT_MODEL_ENABLED is truthy
# (1/true/yes/on). Do not turn it on before the V12 migration is applied to
# public.experiment_run in the BFF database.
IMPORT_MODEL_ENABLED="false"
# base64 of the 32-byte AES-256-GCM key that wraps bundle passwords.
# Generate a throwaway one with: openssl rand -base64 32
IMPORT_PASSWORD_KEY="replace-with-base64-of-32-random-bytes"
# Where the frontend uploads the bundle. The object key must sit under the prefix.
IMPORT_BUNDLE_BUCKET="sientia"
IMPORT_BUNDLE_PREFIX="imported_models/"
# MongoDB collection holding the model listing the import writes its document into.
IMPORT_MODELS_COLLECTION="models"
# The import log lives in the Spring Boot BFF database, a second Postgres
# connection independent of POSTGRES_* above. The schema (public) and the table
# (experiment_run) are constants of the code, not configuration.
IMPORT_STATUS_DB_HOST="localhost"
IMPORT_STATUS_DB_PORT="5432"
IMPORT_STATUS_DB_NAME="sientia-core-mlops-bff"
IMPORT_STATUS_DB_USER="mlops_bff_user"
IMPORT_STATUS_DB_PASSWORD="bff_password"
# --- scripts/import_model_manual_run.py only ---------------------------------
# Inputs for the manual run that stands in for the frontend. The script writes
# one object and one row to whatever the variables above name.
IMPORT_SCRIPT_BUNDLE_PATH="tests/fixtures/bundle/golden_model_v3.sientia"
IMPORT_SCRIPT_BUNDLE_PASSWORD="the-password-the-bundle-was-written-with"
IMPORT_SCRIPT_USERNAME="manual.run@example.com"
# Cell 2's guard: restate IMPORT_STATUS_DB_HOST here (or in the cell) after reading what cell 1
# printed. Left empty on purpose — the script refuses to write until it matches.
IMPORT_SCRIPT_CONFIRM_TARGET=""

150
README.md
View File

@@ -21,6 +21,7 @@ A comprehensive, Temporal-based ML orchestration system for industrial data proc
- [Minimal Retrain Workflow](#4-minimal-retrain-workflow-minimal_retrainpy)
- [Drift Workflow](#5-drift-workflow-driftpy)
- [Simple Metrics Workflow](#6-simple-metrics-workflow-simple_metricspy)
- [Import Model Workflow](#7-import-model-workflow-import_modelpy)
- [Installation & Setup](#installation--setup)
- [Prerequisites](#prerequisites)
- [Environment Setup](#environment-setup)
@@ -31,6 +32,7 @@ A comprehensive, Temporal-based ML orchestration system for industrial data proc
- [Running Tests and Coverage](#running-tests-and-coverage)
- [Manual Test Execution](#manual-test-execution)
- [Manual Application Execution](#manual-application-execution)
- [Running One Model Import by Hand](#running-one-model-import-by-hand)
- [Code Quality & Validation](#code-quality--validation)
- [Overview](#overview)
- [Validation Tools](#validation-tools)
@@ -126,7 +128,8 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
### Key Components
#### **Worker (`laborious/worker/worker.py`)**
- Temporal client setup, four workers via `sientia_do.temporal.worker.prepare_worker`
- Temporal client setup, four workers via `sientia_do.temporal.worker.prepare_worker`, plus a
fifth registered only when `IMPORT_MODEL_ENABLED` is truthy (`1`, `true`, `yes`, `on`)
- Runtime-scoped task queues: `{workflow}-{RUNTIME}-queue` for all workflows
- Metrics server initialization, notification handler setup
- Graceful shutdown and autoscaling-friendly behavior
@@ -135,6 +138,16 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
`simple_metrics-queue`. Use `drift-{RUNTIME}-queue` and `simple_metrics-{RUNTIME}-queue`
matching the worker pod `RUNTIME` env (same as `predictions_batch` / `minimal_retrain`).
Task queues, one per worker:
| Queue | Worker | Registered |
|---|---|---|
| `predictions_batch-{RUNTIME}-queue` | PredictionsBatch (+ its two sub-workflows) | always |
| `minimal_retrain-{RUNTIME}-queue` | MinimalRetrain | always |
| `drift-{RUNTIME}-queue` | Drift | always |
| `simple_metrics-{RUNTIME}-queue` | SimpleMetrics | always |
| `import_model-{RUNTIME}-queue` | ImportModel | only when `IMPORT_MODEL_ENABLED` is truthy |
#### **Workflows (`laborious/workflows/`)**
- `predictions_batch.py`: Batch prediction entry point
- `sub_workflows/prediction_process.py`: Core prediction pipeline
@@ -142,6 +155,8 @@ matching the worker pod `RUNTIME` env (same as `predictions_batch` / `minimal_re
- `minimal_retrain.py`: Automated model retraining and production update
- `drift.py`: Data drift detection and monitoring
- `simple_metrics.py`: Regression metrics calculation (RMSE, MSE, MAE, R²)
- `import_model.py`: `.sientia` bundle import — registered model + model document, behind
`IMPORT_MODEL_ENABLED`
#### **Activities (`laborious/activities/`)**
- `gates.py`: Data quality validation, filtering, and data formatting operations
@@ -509,6 +524,55 @@ flowchart LR
}
```
### 7. Import Model Workflow (`import_model.py`)
The **ImportModel** workflow turns an encrypted `.sientia` bundle already uploaded to MinIO into a
registered MLflow model plus one document in the MongoDB `models` collection — or into a durable
`ERROR` row carrying a sentence a person can read. It is registered **only** when
`IMPORT_MODEL_ENABLED` is set, so the importer can live in its own runtime with its own credentials.
Full contract for the frontend and the BFF: [`docs/model-import.md`](docs/model-import.md).
The file format: [`docs/sientia-bundle-format.md`](docs/sientia-bundle-format.md).
#### Execution Flow
1. **Claim**: takes ownership of the `public.experiment_run` row the BFF inserted (it never inserts one)
2. **Download**: object limits, then the mandatory `expected_digest` comparison
3. **Open**: seven ordered gates — header, decryption, archive inspection, extraction, structure, content policy
4. **Provision**: experiment, a new run with the recorded name, the artifact tree, a registered version (stage untouched)
5. **List**: the MongoDB model document, `active: false`
6. **Finish**: cleanup, then one terminal write classifying the outcome once
#### Architecture Diagram
```mermaid
flowchart LR
A[1. claim_import_status] --> B[2. download_import_bundle] --> C[3. open_import_bundle]
C --> D[4. create_import_experiment] --> E[5. upload_import_artifacts]
E --> F[6. register_import_model_version] --> G[7. write_import_model_document]
G --> H[8. cleanup_import_files] --> I[9. record_import_terminal_status]
A -.-> BFF[(sientia-core-mlops-bff)]
B -.-> MinIO[(MinIO)]
F -.-> MLflow[(MLflow)]
G -.-> Mongo[(MongoDB)]
I -.-> BFF
```
#### Input Parameters
```json
{
"import_run_id": 41,
"object_key": "imported_models/sales_forecast_v3.sientia",
"expected_digest": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
"password_envelope": "Base64OfNonceCiphertextTag==",
"bucket": "sientia"
}
```
A successful import leaves the registered version **unstaged** and the model document **inactive**:
promoting and activating are manual operator steps (QTZPOC-21). The `V12` migration of
`sientia-core-mlops-bff` must be applied before enabling the worker — see
[`docs/model-import.md`](docs/model-import.md) § 7.
## 📋 Prerequisites
- Python 3.11+
@@ -589,10 +653,10 @@ Use the provided script to run the application locally:
```bash
# Make script executable (first time only)
chmod +x run_local.sh
chmod +x scripts/run_local.sh
# Run the application
./run_local.sh
# Run the application (from the repository root)
./scripts/run_local.sh
```
The script will:
@@ -606,10 +670,10 @@ Use the provided script to run tests with coverage:
```bash
# Make script executable (first time only)
chmod +x run_coverage.sh
chmod +x scripts/run_coverage.sh
# Run tests with coverage
./run_coverage.sh
# Run tests with coverage (from the repository root)
./scripts/run_coverage.sh
```
The script will:
@@ -654,6 +718,62 @@ fi
python -m laborious.worker.worker
```
### Running One Model Import by Hand
[`scripts/import_model_manual_run.py`](scripts/import_model_manual_run.py) stands in for the frontend
that does not exist yet. The Angular screen plus the Spring Boot BFF will, between them, hash a
`.sientia` file, wrap the bundle password, upload the object and insert one `PENDING` row before
starting `import_model`; this script does exactly those four things against whatever `.env` points
at, then reads the outcome back and can undo itself.
It is a **development tool**. It writes to whatever `.env` names — a real object in a real bucket and
a real row in a real `public.experiment_run` — and a `.env` copied from a cluster deployment can
perfectly well name production. It is not for production use.
**Prerequisites**
- A reachable environment: the BFF Postgres, MinIO, MongoDB, MLflow and Temporal that `.env` names.
- The `V12` migration applied to `public.experiment_run` in the BFF database. Without it the import
provisions the model and then cannot record the outcome — see
[`docs/model-import.md`](docs/model-import.md) § 7.
- A worker running with `IMPORT_MODEL_ENABLED` truthy and the same `RUNTIME`, if you want cell 8 (or
the Temporal UI) to actually execute the run.
- The import variables from [`.env.example`](.env.example) filled in, including `IMPORT_PASSWORD_KEY`
and the script's own `IMPORT_SCRIPT_BUNDLE_PATH`, `IMPORT_SCRIPT_BUNDLE_PASSWORD` and
`IMPORT_SCRIPT_USERNAME`.
**Running it.** The file is split into `# %%` cells: run them one at a time in the VS Code
interactive window, or run the module top to bottom.
```bash
source ./venv/bin/activate
# top to bottom (cells 5 and 6 write; cell 2 must pass first)
python scripts/import_model_manual_run.py
```
| Cell | Does | Writes? |
|---|---|---|
| 1 | Resolves and prints every target — Postgres, MinIO, Mongo, MLflow, Temporal, the task queue | no |
| 2 | **Confirmation guard** | no |
| 3 | Picks the bundle, computes `expected_digest`, prints the parsed header | no |
| 4 | Builds the AES-256-GCM password envelope | no |
| 5 | Uploads the object, re-reads it and checks the digest still matches | **yes** |
| 6 | Inserts the `PENDING` / `run_type='IMPORT'` row, prints `import_run_id` | **yes** |
| 7 | Prints the workflow input as JSON, ready for the Temporal UI | no |
| 8 | Optionally starts `import_model` and waits (`START_THE_WORKFLOW`) | starts a run |
| 9 | Reads the outcome back: the row, the MLflow experiment/run/version, the model document | no |
| 10 | Undo: deletes the object and the row always; the provisioning behind `DELETE_PROVISIONING` | deletes |
**The guard.** Cells 5 and 6 are the only cells that write, and neither runs until cell 2 passes.
Cell 2 compares a confirmation value against the Postgres host cell 1 resolved: set
`IMPORT_SCRIPT_CONFIRM_TARGET` in `.env`, or paste the host into the cell. It is a read-then-restate
rather than a boolean on purpose — a flag can be flipped without ever looking at what it points at.
No credential is written in the script and none is printed: it prints hosts, ports, database, bucket
and object names, the row id and the workflow input, and never the bundle password, the envelope key
or the envelope's plaintext.
## Code Quality & Validation
### Overview
@@ -819,6 +939,22 @@ See [OPC UA Communication](#opc-ua-communication) for semantics, concurrency, an
| `MLFLOW_PORT` | MLFlow server port | `5080` | Yes |
| `MLFLOW_USERNAME` | MLFlow username | `aignosi` | Yes |
| `MLFLOW_PASSWORD` | MLFlow password | `aignosi` | Yes |
| `IMPORT_MODEL_ENABLED` | Register the import worker on `import_model-{RUNTIME}-queue` | _(unset)_ | No |
| `IMPORT_PASSWORD_KEY` | base64 of the 32-byte AES-256-GCM key wrapping bundle passwords | _(none)_ | If importing |
| `IMPORT_BUNDLE_BUCKET` | Bucket holding uploaded `.sientia` objects | `sientia` | No |
| `IMPORT_BUNDLE_PREFIX` | Key prefix the uploads must sit under | `imported_models/` | No |
| `IMPORT_BUNDLE_RETENTION_DAYS` | Lifecycle expiry applied to that prefix | `7` | No |
| `IMPORT_WORK_DIR` | Worker scratch directory for bundles | `<tempdir>/sientia-import` | No |
| `IMPORT_MAX_OBJECT_BYTES` | Gate 1 object size ceiling | `1073741824` | No |
| `IMPORT_MAX_ARCHIVE_ENTRIES` | Gate 4 entry count ceiling | `5000` | No |
| `IMPORT_MAX_UNCOMPRESSED_BYTES` | Gate 4 uncompressed size ceiling | `4294967296` | No |
| `IMPORT_MAX_COMPRESSION_RATIO` | Gate 4 compression ratio ceiling | `200` | No |
| `IMPORT_MODELS_COLLECTION` | MongoDB collection holding the model listing | `models` | No |
| `IMPORT_STATUS_DB_HOST` | Import log database host (the BFF's) | `localhost` | If importing |
| `IMPORT_STATUS_DB_PORT` | Import log database port | `5432` | No |
| `IMPORT_STATUS_DB_NAME` | Import log database name | `sientia-core-mlops-bff` | No |
| `IMPORT_STATUS_DB_USER` | Import log database user | `mlops_bff_user` | If importing |
| `IMPORT_STATUS_DB_PASSWORD` | Import log database password | _(empty)_ | If importing |
| `OPC_CONFIG` | OPC server configuration (JSON) | `{}` | No |
| `OPC_ID` | OPC server identifier | `1` | No |
| `OPC_URL` | OPC server URL | `opc.tcp://localhost:4840` | No |

349
docs/model-import.md Normal file
View File

@@ -0,0 +1,349 @@
# Importing a model (`import_model`)
The `import_model` workflow turns an encrypted `.sientia` object in MinIO into a registered MLflow
model plus one document in the MongoDB `models` collection — or into a row that says, in a sentence a
person can read, why it did not.
This document is the contract for the two sides that talk to it: the **frontend**, which uploads the
object and builds the request, and the **BFF**, which owns the import log row. Everything here is
enforced by code in this repository; where a statement is about a column or a constraint, the
authority is the migration named in §7.
Format of the file itself: [`sientia-bundle-format.md`](sientia-bundle-format.md).
---
## 1. What the caller does, in order
1. **Upload** the `.sientia` object to the import bucket, under the configured prefix (default
`imported_models/`). The key must end in `.sientia`.
2. **Compute `expected_digest`** — the SHA-256, lowercase hex, of the **exact bytes that were
uploaded**. Not of a re-read, not of a re-serialisation: of what went over the wire.
3. **Encrypt the bundle password** into a `password_envelope` (§3).
4. **Insert the import log row** (§5) and take its `id`.
5. **Start the workflow** on `import_model-{RUNTIME}-queue` with the input of §2.
The workflow claims the row, opens the bundle, provisions MLflow, writes the model document, cleans
up and records one terminal verdict. It never inserts a row, never promotes a version and never
activates a model.
## 2. The workflow input
```json
{
"import_run_id": 41,
"object_key": "imported_models/sales_forecast_v3.sientia",
"expected_digest": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
"password_envelope": "Base64OfNonceCiphertextTag==",
"bucket": "sientia"
}
```
| Field | Required | Meaning |
|---|---|---|
| `import_run_id` | yes | primary key of the `public.experiment_run` row the BFF inserted. Must be a positive integer |
| `object_key` | yes | the uploaded object's key. Must match the row's `file_name` when the row carries one |
| `expected_digest` | **yes** | SHA-256 of the uploaded bytes, 64 lowercase hex characters. There is no path that skips the comparison: a missing value fails at `received`, a mismatching one at `download`, before the header is parsed |
| `password_envelope` | yes | the bundle password, encrypted (§3). A plaintext `password` field is **refused** |
| `bucket` | no | defaults to `IMPORT_BUNDLE_BUCKET` |
**Fields that are refused rather than ignored**: `project_id`, `project_name`,
`provisioning_target`, `schema`, `table_name`, `status_table`, `password`. An input carrying any of
them fails at `received`, so no caller can believe it recorded something the import does not record
(see §10).
## 3. The password envelope
AES-256-GCM, and the ciphertext layout is what the WebCrypto API returns natively:
```
envelope = base64( nonce(12 bytes) || ciphertext || tag(16 bytes) )
```
`crypto.subtle.encrypt({name: 'AES-GCM', iv: nonce}, key, plaintext)` returns `ciphertext || tag`
already concatenated, so the browser side is: generate 12 random bytes, encrypt, concatenate the
nonce in front, base64 the result.
The key is a 32-byte AES-256 key shared with the worker through `IMPORT_PASSWORD_KEY` (base64). The
worker decrypts the envelope **inside the activity that uses the password**, so no plaintext password
is ever an activity input, an activity result or a workflow variable — and therefore never appears in
the Temporal event history. A test asserts that for every failure path.
A failure to open the envelope is recorded as a `decryption` failure, distinct in the logs from a
wrong *bundle* password but sharing its step, because the step says where the import stopped, not who
is at fault.
## 4. Environment
| Variable | Default | What it is |
|---|---|---|
| `IMPORT_MODEL_ENABLED` | unset | registers the fifth (import) worker. Truthy values: `1`, `true`, `yes`, `on` |
| `IMPORT_PASSWORD_KEY` | — | base64 of the 32-byte AES-256-GCM key that wraps bundle passwords |
| `IMPORT_BUNDLE_BUCKET` | `sientia` | bucket holding the uploaded bundles |
| `IMPORT_BUNDLE_PREFIX` | `imported_models/` | key prefix the uploads must sit under |
| `IMPORT_BUNDLE_RETENTION_DAYS` | `7` | lifecycle expiry the importer applies to that prefix |
| `IMPORT_WORK_DIR` | `<tempdir>/sientia-import` | worker scratch directory |
| `IMPORT_MAX_OBJECT_BYTES` | 1 GiB | gate 1 |
| `IMPORT_MAX_ARCHIVE_ENTRIES` | `5000` | gate 4 |
| `IMPORT_MAX_UNCOMPRESSED_BYTES` | 4 GiB | gate 4 |
| `IMPORT_MAX_COMPRESSION_RATIO` | `200` | gate 4 |
| `IMPORT_MODELS_COLLECTION` | `models` | the MongoDB collection that *is* the model listing |
| `IMPORT_STATUS_DB_HOST` | `localhost` | the BFF database holding the import log |
| `IMPORT_STATUS_DB_PORT` | `5432` | |
| `IMPORT_STATUS_DB_NAME` | `sientia-core-mlops-bff` | |
| `IMPORT_STATUS_DB_USER` | `mlops_bff_user` | |
| `IMPORT_STATUS_DB_PASSWORD` | empty | |
`IMPORT_STATUS_DB_*` is a **second** Postgres connection, independent of the one Laborious already
holds for predictions. The import writes to that database and no other, and to one table in it.
A missing or malformed `IMPORT_PASSWORD_KEY` is an import failure at the decryption step, not a boot
failure: the worker starts either way.
## 5. The import log row — the BFF's half
The row is `public.experiment_run` in `sientia-core-mlops-bff`. **The workflow never inserts it.**
The inserting side writes:
| Column | Value |
|---|---|
| `run_type` | `'IMPORT'` |
| `status` | `'PENDING'` |
| `username` | whoever asked for the import |
| `bucket_name` | the bucket the object was uploaded to |
| `file_name` | the object key |
| `request_data` | the request payload. If it carries a digest under `expected_digest`, `digest`, `sha256` or `file_digest`, the workflow checks it against the input's |
Then it passes the returned `id` as `import_run_id`.
The workflow's first act is to **claim** that row:
```sql
UPDATE public.experiment_run
SET status = 'ORCHESTRATOR_WAITING_PROC', updated_at = :now,
orchestrator_response_data = :detail
WHERE id = :import_run_id
AND run_type = 'IMPORT'
AND (status = 'PENDING'
OR (status = 'ORCHESTRATOR_WAITING_PROC'
AND orchestrator_response_data -> 'import' ->> 'workflow_id' = :workflow_id))
```
The second disjunct makes a re-execution of the same workflow idempotent instead of a conflict. Every
later write keys on `id` alone, because ownership was settled here.
**Columns the workflow never writes**: `username`, `request_data`, `bucket_name`, `file_name`,
`run_type`, `created_at`. **Columns it does write**: `status`, `error_message`, `updated_at`,
`orchestrator_response_data`, and — as soon as the bundle is open, before anything exists in MLflow —
`experiment_name` (the bundle's own, verbatim) and `run_name` (`import-{model_name}-v{model_version}`,
truncated inside the name segment to fit 50 characters). Those two are **overwritten** by the
workflow; whatever the inserting side put there is replaced.
### When the row cannot be claimed
If the claim matches nothing — no such id, a `run_type='TRAIN'` row, or a row already running under
another workflow — the import **writes nothing anywhere**, because the row belongs to someone else.
Instead it:
- fails the Temporal run with error type `ImportLogRowNotClaimableError` and code
`IMPORT_LOG_ROW_NOT_CLAIMABLE`, non-retryable;
- increments the import error metric and fires a notification;
- logs the id and the observed state (which of the three causes it was).
So a row left `PENDING` with no `workflow_id` in its detail JSON means the workflow never owned it.
**Closing that row out belongs to whichever side inserted it** — by timeout, by a sweeper, by a person
— however that side chooses. The importer deliberately does not guess.
## 6. Status vocabulary
| Status | Written when | New in `V12`? |
|---|---|---|
| `ORCHESTRATOR_WAITING_PROC` | claimed, and while running | no — **reused** from the training flow |
| `ORCHESTRATOR_VALIDATION_ERROR` | failure at `received` … `content_policy` — the user can act on it | no — **reused** |
| `IMPORT_SUCCESS` | every step succeeded | yes |
| `IMPORT_ERROR` | failure at `experiment_creation` … `model_document`, or unclassified — only support can act on it | yes |
The two reused values describe a *stage*, not a flow: what says the row is an import is `run_type`,
not the status. There is deliberately no `IMPORT_RUNNING`; it would carry no information `run_type`
does not already carry, and it would widen both the pairing `CHECK` and the frontend's switch for
nothing.
`error_message` is the user-facing sentence, taken from the catalog in §8. On success it is written as
`NULL`, never `''` — the column carries a `LENGTH(error_message) >= 3` check.
## 7. The migration this depends on
`sientia-dataops-database-migrations`,
`postgres/sientia-core-mlops-bff/migrations/V12__add_import_statuses_to_experiment_run.sql`:
- adds `IMPORT_SUCCESS` and `IMPORT_ERROR` to `chk_experiment_run_status`;
- adds `chk_experiment_run_terminal_status_run_type`, pairing only the **terminal** values with their
`run_type` (`IMPORT_*` on `'IMPORT'`, `TRAINING_*` on `'TRAIN'`) and leaving the shared
`ORCHESTRATOR_*` values out of the rule;
- **adds no columns at all.**
`V12` must be merged and applied before `IMPORT_MODEL_ENABLED` is set anywhere. Without it, an import
provisions the model and then cannot record its verdict: the claim still succeeds (no new columns are
needed for it), the old `CHECK` refuses `IMPORT_SUCCESS`, and the failure surfaces as
`IMPORT_STATUS_NOT_RECORDED` on the first attempt — no retry wait — leaving the row in
`ORCHESTRATOR_WAITING_PROC` with a fully provisioned model behind it.
## 8. `orchestrator_response_data`, and the error catalog
There is no column for the error code and none for the workflow identifier (decision D21). Both live
at fixed keys in the JSON column the workflow already owns, and the whole `import` object is rewritten
on every write, so the identifier the claim wrote is never dropped:
```json
{
"import": {
"workflow_id": "import_model-41-01HW…",
"code": "IMPORT_BUNDLE_INCOMPLETE",
"step": "structure_validation",
"gate": 6,
"source": { "model_name": "sales_forecast", "experiment_name": "…", "…": "…" },
"cleanup_failed": false
}
}
```
- `workflow_id` — what a user quotes to support, and what the claim's idempotency disjunct reads.
- `code` — the frontend's i18n key. **Absent on success.**
- `step` — where the import stopped (the values of §9).
- `gate` — which of the seven gates rejected it, when one did.
- `source` — the bundle's own metadata block, once it is known.
- `cleanup_failed` — temporary files were left on the worker. Present on a **successful** import too:
it never changes the status, because the user's model was imported.
### Code → reason
`error_message` is exactly the sentence in this table; nothing is interpolated into it, and the
exception's text — paths, table names, exception classes, tracebacks — reaches only the logs.
| Code | Step | Sentence written to `error_message` |
|---|---|---|
| `IMPORT_REQUEST_INVALID` | `received` | The import could not be started because the request was incomplete or contradictory. Nothing was created. Submit the import again. |
| `IMPORT_FILE_UNREADABLE` | `download` | The uploaded file could not be read, or it is not the file that was sent. Upload it again. |
| `IMPORT_BUNDLE_CANNOT_BE_OPENED` | `decryption` | This file could not be opened: the password is wrong, or the file is damaged or was altered. Confirm the password with whoever exported the model, then try again. |
| `IMPORT_BUNDLE_REJECTED` | `archive_inspection` | The file's contents did not pass the platform's safety checks, so it was not opened. Export the model again from the origin platform. |
| `IMPORT_BUNDLE_UNPACK_FAILED` | `extraction` | The file could not be unpacked safely and nothing from it was kept. Export the model again from the origin platform. |
| `IMPORT_BUNDLE_INCOMPLETE` | `structure_validation` | This file is not a complete model export — part of what the platform needs is missing from it. Export the model again from the origin platform. |
| `IMPORT_BUNDLE_UNEXPECTED_CONTENT` | `content_policy` | The file contains something a model export should not contain, so it was rejected. Export the model again from the origin platform. |
| `IMPORT_STORAGE_PREPARATION_FAILED` | `experiment_creation` | The platform could not prepare a place to keep this model. The import stopped and no model was created. Contact support with this import's identifier. |
| `IMPORT_MODEL_FILES_NOT_STORED` | `artifact_upload` | The model's files could not be stored on this platform. The import stopped and the model is not available. Contact support with this import's identifier. |
| `IMPORT_MODEL_NOT_REGISTERED` | `registration` | The model's files were stored, but the model itself could not be registered, so it cannot be used. Contact support with this import's identifier. |
| `IMPORT_MODEL_SETTINGS_NOT_SAVED` | `model_document` | The model was registered, but the settings that tell the platform how to run it could not be saved. Contact support with this import's identifier. |
| `IMPORT_UNEXPECTED_ERROR` | — | The import stopped for an unexpected reason and no model was created. Contact support with this import's identifier. |
Three codes never reach a row's `error_message`, because there is no row to write or no failure to
report to a user. They exist in the logs, the metric and the notification only:
| Code | When |
|---|---|
| `IMPORT_LOG_ROW_NOT_CLAIMABLE` | the row is not this workflow's (§5) |
| `IMPORT_STATUS_NOT_RECORDED` | the verdict itself could not be written; the log line carries the verdict it could not record |
| `IMPORT_CLEANUP_INCOMPLETE` | temporary files were left behind; also flagged as `cleanup_failed` in the detail |
## 9. Steps
`received`, `download`, `decryption`, `archive_inspection`, `extraction`, `structure_validation`,
`content_policy`, `experiment_creation`, `artifact_upload`, `registration`, `model_document`,
`cleanup`. The first seven are the gates (see the format document); a failure in any of them is a
`ORCHESTRATOR_VALIDATION_ERROR`. The next four are provisioning; a failure there is an
`IMPORT_ERROR`. `cleanup` is never a reported failure.
## 10. The import knows nothing about projects
The model listing **is** the MongoDB `models` document. No migrated relational database in this
platform holds a project table or a model table, nothing in this repository reads a project-to-model
link, and the workflow input carries no project target — which is why naming one is refused rather
than accepted and ignored. Whatever the import screen does with projects is the frontend's business.
The bundle's own `metadata.model_project` is logged as an MLflow parameter (`source.model_project`),
as a record of where the model came from. It is not a link.
## 11. What MLflow ends up holding
| Thing | Value |
|---|---|
| Experiment | `metadata.experiment_name` from the bundle, created if absent, reused if present |
| Run | always **new**, created with the `run_name` already written on the record |
| Artifacts | the bundle's whole `artifacts/` tree, with `prediction_model` and `data_model` at the run's artifact root |
| Parameters | the bundle's `parameters`, verbatim, plus the origin metadata as `source.*` (including `source.run_id`, which is never dereferenced) |
| Registered version | created on demand; source is the run's **resolved artifact URI** with `/prediction_model` appended, e.g. `mlflow-artifacts:/17/<run_id>/artifacts/prediction_model` |
| Stage | **untouched** |
The source shape matters and is easy to get wrong: the legacy loader extracts the run id as
`source.split('/')[2]`, which is the run id in the resolved artifact URI but the *artifact path* in a
literal `runs:/<run_id>/prediction_model`. Registering with the unresolved URI breaks loading
silently — nothing raises, the loader just returns the wrong string. This is the same value
`mlflow.register_model` stores for the training flow, because it resolves the `runs:/` URI before
writing it; `MlflowClient.create_model_version` does not, so the import resolves it itself.
## 12. The model document
Written **last**, into the collection named by `IMPORT_MODELS_COLLECTION` (default `models`):
```json
{
"id": "31",
"name": "sales_forecast",
"active": false,
"model_config": {
"transform_flavor": "sklearn",
"predict_flavor": "joblib",
"retention_minutes": 60,
"target": "SE"
}
}
```
- `id` is `str(max(int(id) for id in collection) + 1)` — computed in Python, from the ids that are
numbers. Never a `count`, never a lexicographic sort (which would answer `"9" > "30"`). Ids that
are not decimal numbers are ignored for the maximum and counted in the log.
- `name` is `metadata.model_name` from the bundle, verbatim.
- `active` is `false`, always. No code path in the import writes or updates a document with
`active: true`.
- `target` comes from `parameters.target_variable`; nothing else supplies it.
- The `models` collection carries a **unique sparse index on `id`**, created by the MongoDB baseline
migration. A duplicate-key insert is reported as a `model_document` failure — no retry, no id
recomputation.
**A name already in the collection is left exactly as it is.** The import writes nothing, modifies
nothing, logs the skip and succeeds: `active`, `target` and the flavors of a model already on the
platform are operator-owned.
## 13. After a successful import — the manual steps
A `COMPLETED` import is **not** a servable model. Stated plainly:
- the registered version is **unstaged**, and `load_predict_model` only sees a version once it is in
stage `Production`;
- the model document is **inactive** (`active: false`).
So an operator must promote the version to `Production` and set `active: true`. This workflow never
promotes and never activates — that runbook is QTZPOC-21's.
One more thing to know before serving an imported model: `predict_flavor: 'joblib'` is a **read-only**
flavor. It is what the artifact actually is, and it is enough to load and predict, but nothing in this
platform writes that flavor — so an imported model cannot be retrained until a write flavor is
configured for it.
## 14. Retry, and what does not retry
Only the status writes retry, and only because `UPDATE ... SET <fixed values> WHERE id` leaves the row
the same however many times it runs:
| Call | Attempts | Why |
|---|---|---|
| claim, terminal write | 10, 1 s → 30 s backoff (≈151 s envelope) | the verdict must land; the database being briefly unavailable is exactly what a retry fixes |
| progress writes, names | 3, capped at 2 s | a hint nobody reads — the terminal write rewrites it — must not stall a working import |
| everything else | 1 | a repeated registry call or Mongo insert can write twice |
`ImportLogRowNotClaimableError` and `ImportStatusRejectedError` are non-retryable: the first would
read the same rows and reach the same conclusion, and the second means the statement itself was
refused (a `CHECK` violation, an undefined column, a value too long), which a second attempt cannot
change.
There is no rollback, no compensation, no resume and no reconciliation. A failed import stays failed;
trying again is a new upload and a new row.

196
docs/sientia-bundle-format.md Normal file
View File

@@ -0,0 +1,196 @@
# The `.sientia` bundle format
A `.sientia` file is one exported MLflow model: its whole artifact tree, its run parameters and six
fields of origin metadata, zipped and then encrypted under a password the exporter chose. It is
written by the Streamlit platform (`sientia-projects-templates`) and read by this repository's
importer. Two programs in two repositories agree on these bytes, so the constants below are
duplicated by necessity — the drift is held in place by tests on both sides, not by a shared library.
This document covers both directions: **§1–§3 how a bundle is generated**, so the producer can be
audited against it, and **§4–§7 how it is read**, so a rejection can be explained.
Normative implementations:
| Side | Where |
|---|---|
| Writer | `sientia-projects-templates`, `app/src/operations/model_export/{crypto,bundle,exporter}.py` |
| Reader | this repository, `laborious/utils/bundle/{format,reader}.py` |
| Test writer | this repository, `tests/helpers/bundle_factory.py` — mirrors the producer field for field, for building hostile variants |
| Golden fixture | `tests/fixtures/bundle/` — a real producer artefact, with its password and producer commit recorded in that directory's `README.md` |
---
## 1. Generation, step by step
Entry point: `exporter.export_model_bundle(*, client, run_id, model_name, experiment_name,
model_version, model_project, password) -> bytes`.
1. **Password strength** — `crypto.validate_password_strength`: at least 12 characters, one
uppercase, one lowercase, one digit, one symbol. **Export-side only.** The importer cannot
re-check it and does not try: by the time a bundle exists, the password is whatever it was.
2. **Exportability** — `bundle.validate_run_is_exportable`: the run's top-level artifacts must
include **both** `data_model` and `prediction_model`, checked with `client.list_artifacts(run_id)`
before anything is downloaded.
3. **Whole-tree download** — `client.download_artifacts(run_id, '', staging)` into a
`tempfile.mkdtemp(prefix='sientia_export_')` directory. The entire artifact tree, not a subset.
4. **Run parameters** — `dict(run.data.params)`, copied **verbatim**. Whatever the run logged is
what the bundle carries, including `target_variable` (see §3).
5. **Metadata document** — `bundle.build_metadata_json`, exactly two top-level keys:
- `parameters`: the run parameters, verbatim;
- `metadata`: `model_name`, `experiment_name`, `run_id`, `model_version`, `model_project` and
`export_timestamp = datetime.now(UTC).isoformat()`. Six fields, no others.
6. **Zip** — `bundle.build_zip`, `zipfile.ZIP_DEFLATED`. `metadata.json` is written **first**, with
`json.dumps(...)` and no indent. Every file under the staging directory follows as
`artifacts/<relpath>`, with `\` replaced by `/`. **No directory entries are stored**, so a reader
must derive directories from file paths.
7. **Encryption** — `crypto.encrypt_archive`: a fresh 16-byte `os.urandom` salt, Argon2id key
derivation, then a libsodium `secretstream` push in 64 KiB plaintext chunks, the last chunk tagged
`FINAL` and the rest `MESSAGE`.
8. **Download name** — `{model_name}_v{model_version}.sientia`. The staging directory is removed in a
`finally`, and the password is popped from the Streamlit session state.
Producer pins: `pynacl==1.5.0`, `mlflow==2.10.1`.
## 2. Constants
Copied from `crypto.py` unless noted. The reader's names are the ones in
`laborious/utils/bundle/format.py`.
| Producer | Value | Reader |
|---|---|---|
| `MAGIC` | `b'SIENTIA1'` | `MAGIC` |
| `_HEADER_STRUCT` | `struct.Struct('>8sBBB16sQQB32s64s')` | `HEADER_STRUCT` |
| `HEADER_STRUCT_SIZE` | `140` | `HEADER_SIZE` |
| `FORMAT_VERSION` | `1` | `FORMAT_VERSION` |
| `AEAD_ID_XCHACHA20POLY1305_SECRETSTREAM` | `1` | same name |
| `KDF_ID_ARGON2ID` | `1` | same name |
| `SALT_SIZE` | `16` | `SALT_SIZE` |
| `KDF_MEMLIMIT_BYTES` | `268435456` (256 MiB) | `MAX_KDF_MEMLIMIT` — a **ceiling** on the reader's side |
| `KDF_OPSLIMIT` | `3` | `MAX_KDF_OPSLIMIT` — likewise a ceiling |
| `KDF_PARALLELISM_RESERVED` | `1` | `PARALLELISM` |
| `DIGEST_SIZE` | `32` | `DIGEST_SIZE` |
| `RESERVED_SIGNATURE_SIZE` | `64` | `RESERVED_SIGNATURE_SIZE` |
| `CHUNK_SIZE` | `65536` (64 KiB) | `CHUNK_SIZE` |
| KDF algorithm | `crypto_pwhash_ALG_ARGON2ID13` | `KDF_ALG` |
| Key length | `crypto_secretstream_xchacha20poly1305_KEYBYTES` (32) | `KEY_SIZE` |
Fixed by libsodium rather than by the producer, but needed to frame the stream:
- `crypto_secretstream_xchacha20poly1305_HEADERBYTES == 24` → `STREAM_HEADER_SIZE`
- `crypto_secretstream_xchacha20poly1305_ABYTES == 17` → `ABYTES`
- pre-upload prefix `140 + 24 + (65536 + 17) = 65717` → `PREFIX_SIZE`, the smallest number of leading
bytes from which a header and one full chunk can be checked
**Password normalisation.** `_derive_key` applies `unicodedata.normalize('NFC', password)` and
encodes UTF-8 — no trimming, no case folding, no padding. The reader normalises identically, so a
password typed in decomposed form opens a bundle written from the composed form.
## 3. `target_variable`, and where it does not come from
Gate 6 requires `parameters.target_variable`: it is the only source of `model_config.target` in the
model document, and every workflow that serves a model indexes that key directly. On the producer
side:
| Export path | Logs `target_variable`? | Evidence |
|---|---|---|
| Time-series training template | yes | `app/pages/template_01.py` L1212-1214 |
| Model manager, "Save Experiment" | yes, if the session carried it | `app/pages/model_manager.py` L791-796 re-logs `st.session_state.parameters` verbatim |
| Model ensemble | same condition | `app/pages/model_ensemble.py` L695-700 |
| Pipeline / timeseries template | **no** | `app/pages/timeseries_template.py` L1761-1762, L1811-1812 log `run_metadata` only |
| AutoML | **no** | `automl_logging_operations.py` L454-491; the target appears as `"target"` inside the model card JSON, never as a run parameter |
So a gate-6 rejection naming `target_variable` means the origin run came from the pipeline or AutoML
path. The remedy is to re-export from a run trained by the time-series template, which is what the
importer's `IMPORT_BUNDLE_INCOMPLETE` sentence already asks for.
---
## 4. The file on disk
```
offset width field constraint the reader enforces
------ ----- -------------------- ------------------------------------------------------
0 8 magic == b'SIENTIA1'
8 1 format_version == 1
9 1 aead_id == 1 (XChaCha20-Poly1305 secretstream)
10 1 kdf_id == 1 (Argon2id)
11 16 salt passed to the KDF as-is
27 8 kdf_memlimit_bytes <= 268435456
35 8 kdf_opslimit <= 3
43 1 kdf_parallelism == 1
44 32 ciphertext_digest == sha256(file[164:])
76 64 reserved_signature all zero
------ -----
140 24 secretstream header libsodium HEADERBYTES
164 ... chunk stream 64 KiB plaintext chunks, each +17 bytes of AEAD tag
```
All integers are big-endian; the format string starts with `>`, so there is no alignment padding and
the offsets above are exact.
**The digest boundary.** The producer computes `hashlib.sha256(ciphertext)` where `ciphertext` is the
concatenation of the *pushed chunks only*, and then writes `header + stream_header + ciphertext`. On
disk, therefore, the digest covers everything after offset **164**, not after 140:
> `ciphertext_digest == sha256(file[HEADER_SIZE + STREAM_HEADER_SIZE:])`
A reader that hashes from 140 will reject every valid bundle. The fixture test that recomputes both
values and asserts the first matches while the second does not is what pins this, and is the reason
the golden fixture exists.
**Version policy.** `format_version` is checked for equality, not for a range: a version this reader
does not know is refused at gate 2 rather than parsed optimistically. `reserved_signature` is
reserved for a future signature and must be all zero today, so a signed bundle is a new format
version and not a silent change of meaning.
## 5. The seven gates
`BundleReader.open()` runs them in this order, in one function, so the order cannot be changed by a
caller. Each raises `BundleGateError(gate=n)`, which the activity layer maps to an import step.
| Gate | Step recorded | What it checks | Written to disk |
|---|---|---|---|
| 1 | `download` | object size ceiling, key prefix, `.sientia` suffix — from `stat_object`, before the body is fetched; then the SHA-256 of the downloaded bytes against the digest the uploader computed | the downloaded object only |
| 2 | `download` | the whole header table of §4, including the chunk-stream digest | nothing |
| 3 | `decryption` | Argon2id derivation with the header's parameters, then `secretstream` pull. The AEAD is the integrity check: a wrong password and an altered file fail identically, and the message says so | the decrypted zip, in the work directory |
| 4 | `archive_inspection` | central directory only: entry count, total uncompressed size, compression ratio, absolute names, `..` components, symlink and non-regular modes | nothing |
| 5 | `extraction` | extraction into a **fresh** directory (a pre-existing one is a failure), every member's `realpath` contained in it, no overwrite of a pre-existing file; a rejection removes its own partial tree | the extracted tree |
| 6 | `structure_validation` | `metadata.json` + `artifacts/` layout, exactly two top-level keys, all six metadata fields present and non-blank, `experiment_name` between 3 and 50 characters, `parameters.target_variable` present and non-blank, and `prediction_model`/`data_model` each holding `MLmodel` and `model.pkl` | nothing |
| 7 | `content_policy` | every file is either a known name (`MLmodel`, `metadata.json`, `model.pkl`, `conda.yaml`, `python_env.yaml`, `requirements.txt`, `model_card.json`, `model_card.svg`) or a known suffix (`.csv .json .yaml .yml .txt .pkl .svg .md`) | nothing |
Gates 1 and 2 share the `download` step deliberately: an oversized object, a wrong magic, an
unsupported version and a digest that disagrees all say *these are not the bytes that were sent*, and
none of them may borrow the `decryption` sentence, which implicates the password.
The `experiment_name` bound in gate 6 is not cosmetic: the import record's `experiment_name` column is
`VARCHAR(50)`, and the name is written onto the record before provisioning starts. Enforcing the
column's limit at the gate is what keeps it from surfacing three activities later as a constraint
violation.
## 6. Ceilings and where they come from
Gate 1 and gate 4 read their limits from `build_import_config()`, so a deployment can tighten them:
| Variable | Default | Gate |
|---|---|---|
| `IMPORT_MAX_OBJECT_BYTES` | 1 GiB | 1 |
| `IMPORT_MAX_ARCHIVE_ENTRIES` | 5000 | 4 |
| `IMPORT_MAX_UNCOMPRESSED_BYTES` | 4 GiB | 4 |
| `IMPORT_MAX_COMPRESSION_RATIO` | 200 | 4 |
| `IMPORT_BUNDLE_PREFIX` | `imported_models/` | 1 |
The KDF ceilings are **not** configurable: they are the producer's own cost parameters, and a header
declaring more is a resource-exhaustion attempt, refused before any memory is allocated.
## 7. Fixtures
- `tests/fixtures/bundle/golden_model_v3.sientia` — emitted by the producer's own code in its venv,
with the password and producer commit recorded in `tests/fixtures/bundle/README.md`. Regenerate it
with `tests/fixtures/bundle/regenerate.py`. It pays the producer's real Argon2id cost (256 MiB,
3 passes), so it is used where the real thing matters and not in every test.
- `tests/helpers/bundle_factory.py` — the test-only writer. Same header fields at the same offsets,
but 8 MiB and 1 pass, and with a knob for every hostile variant: wrong magic, unknown version, KDF
parameters above the ceiling, digest mismatch, tampered ciphertext, truncated final chunk, non-zero
reserved signature, zip bomb, `..` entry, absolute entry, symlink, missing `data_model`, missing
`MLmodel`, missing `model.pkl`, malformed metadata, missing `target_variable`, extra `.sh` file. A
test asserts its header bytes are structurally identical to the golden fixture's.

11
e2e/conftest.py
View File

@@ -33,6 +33,11 @@ from laborious.workflows.predictions_batch import PredictionsBatch
from laborious.workflows.sub_workflows.format_and_export_prediction import FormatAndExportPrediction
from laborious.workflows.sub_workflows.prediction_process import PredictionProcess
# The `.sientia` import fixtures live in their own module rather than here: four containers' worth
# of setup, an MLflow server subprocess and an ImportModel worker would double this file. Every
# fixture it adds is `import_`-prefixed, so nothing below changes meaning.
pytest_plugins = ['e2e.import_harness']
# Test constants
TEST_MONGODB_CONNECTION_STRING = 'mongodb://localhost:27017'
TEST_DATABASE_NAME = 'test_db'
@@ -60,8 +65,12 @@ def postgres_container():
This fixture creates a real PostgreSQL database in a Docker container
that will be used for all tests in the session.
`log_statement=all` is the one thing the import suite needs from this fixture: proving an import
issues no statement at all against the predictions database requires a log that would have
shown one. It changes what the server writes to its own log and nothing a test observes.
"""
postgres = PostgresContainer('postgres:15')
postgres = PostgresContainer('postgres:15').with_command('postgres -c log_statement=all')
postgres.start()
yield postgres
postgres.stop()

22
e2e/fixtures/experiment_run.sql
View File

@@ -1,4 +1,4 @@
-- public.experiment_run as migrations V1 … V12 leave it, for the e2e Postgres container.
-- public.experiment_run as migrations V1 … V11 leave it, for the e2e Postgres container.
--
-- Source of truth: sientia-dataops-database-migrations,
-- postgres/sientia-core-mlops-bff/migrations/ — this file mirrors the *final state* of:
@@ -14,7 +14,11 @@
-- V9 remove_file_upload_statuses (status vocabulary)
-- V10 remove_unused_statuses (status vocabulary)
-- V11 add_run_type_to_experiment_run (run_type + reindexed)
-- V12 add_import_statuses_to_experiment_run (IMPORT_SUCCESS / IMPORT_ERROR + pairing)
--
-- V12 (add_import_statuses_to_experiment_run) lives in `experiment_run_v12.sql`, applied on top of
-- this file. It is separate because one e2e test — the evidence for why the migration gates
-- enabling the import worker — has to run against a table that does *not* have it, and a second
-- whole-schema copy with V12 removed would rot the moment either copy were edited.
--
-- The same drift trade already accepted for the bundle fixtures: this is a copy, and the tests
-- that read it are what make the copy auditable. If a migration lands there, update this file and
@@ -48,19 +52,13 @@ CREATE TABLE IF NOT EXISTS public.experiment_run (
CONSTRAINT chk_experiment_run_file_name_pattern
CHECK (file_name ~* '^[a-zA-Z0-9 /._-]+$'),
-- V12: the two shared stage values plus the training and import terminals.
-- The V1…V11 status vocabulary: the two shared stage values plus the training terminals.
-- V12 adds IMPORT_SUCCESS and IMPORT_ERROR to this list.
CONSTRAINT chk_experiment_run_status CHECK (status IN ('PENDING', 'ORCHESTRATOR_REQUEST_ERROR',
'ORCHESTRATOR_WAITING_PROC',
'ORCHESTRATOR_VALIDATION_ERROR',
'TRAINING_SUCCESS', 'TRAINING_ERROR',
'IMPORT_SUCCESS', 'IMPORT_ERROR')),
CONSTRAINT chk_experiment_run_run_type CHECK (run_type IN ('TRAIN', 'IMPORT')),
-- V12: each pair of terminal values belongs to its own run_type. The shared values stay out.
CONSTRAINT chk_experiment_run_terminal_status_run_type CHECK (
(run_type = 'IMPORT' OR status NOT IN ('IMPORT_SUCCESS', 'IMPORT_ERROR'))
AND (run_type = 'TRAIN' OR status NOT IN ('TRAINING_SUCCESS', 'TRAINING_ERROR'))
)
'TRAINING_SUCCESS', 'TRAINING_ERROR')),
CONSTRAINT chk_experiment_run_run_type CHECK (run_type IN ('TRAIN', 'IMPORT'))
);
CREATE INDEX IF NOT EXISTS idx_experiment_run_user_exp_status_date_type

36
e2e/fixtures/experiment_run_v12.sql Normal file
View File

@@ -0,0 +1,36 @@
-- V12 add_import_statuses_to_experiment_run, as a delta applied on top of `experiment_run.sql`.
--
-- Source of truth: sientia-dataops-database-migrations,
-- postgres/sientia-core-mlops-bff/migrations/V12__add_import_statuses_to_experiment_run.sql
--
-- Two things, and nothing else:
--
-- 1. the status vocabulary gains IMPORT_SUCCESS and IMPORT_ERROR;
-- 2. chk_experiment_run_terminal_status_run_type pairs each terminal value with its own
-- run_type, so a TRAIN row can never end IMPORT_SUCCESS and an IMPORT row can never end
-- TRAINING_ERROR. The two shared stage values (PENDING, ORCHESTRATOR_WAITING_PROC) stay out
-- of the pairing — both run types legitimately pass through them.
--
-- This file is applied separately because one e2e test runs an import against a table *without*
-- it: that failure shape — provisioned, then unrecordable, refused on the first attempt — is the
-- evidence for why this migration has to land before IMPORT_MODEL_ENABLED is turned on anywhere.
ALTER TABLE public.experiment_run
DROP CONSTRAINT IF EXISTS chk_experiment_run_status;
ALTER TABLE public.experiment_run
ADD CONSTRAINT chk_experiment_run_status CHECK (status IN ('PENDING',
'ORCHESTRATOR_REQUEST_ERROR',
'ORCHESTRATOR_WAITING_PROC',
'ORCHESTRATOR_VALIDATION_ERROR',
'TRAINING_SUCCESS', 'TRAINING_ERROR',
'IMPORT_SUCCESS', 'IMPORT_ERROR'));
ALTER TABLE public.experiment_run
DROP CONSTRAINT IF EXISTS chk_experiment_run_terminal_status_run_type;
ALTER TABLE public.experiment_run
ADD CONSTRAINT chk_experiment_run_terminal_status_run_type CHECK (
(run_type = 'IMPORT' OR status NOT IN ('IMPORT_SUCCESS', 'IMPORT_ERROR'))
AND (run_type = 'TRAIN' OR status NOT IN ('TRAINING_SUCCESS', 'TRAINING_ERROR'))
);

950
e2e/import_harness.py Normal file
View File

@@ -0,0 +1,950 @@
"""Fixtures for the `.sientia` model-import e2e suite. Registered from `e2e/conftest.py`.
**Why this is a module of its own.** `e2e/conftest.py` is shared by every existing e2e suite and
long enough already; four containers' worth of setup, an MLflow server subprocess, the import
environment variables and an `ImportModel` worker do not belong in it. Every fixture here is
prefixed `import_`, so nothing collides with the predictions harness, and no existing fixture is
edited — the one exception being `log_statement=all` on the shared `postgres_container`, which the
absence assertions need and which changes no test's behaviour.
**The fidelity contract, which is the whole point of this file.** On the import path *nothing is
mocked*:
| Dependency | Here | Why real |
|---|---|---|
| MinIO | the shared `minio_container`, plus an import bucket | gate 1 reads a real object |
| BFF Postgres | a container of its own, `sientia-core-mlops-bff` | the real `CHECK` constraints are half of what is under test, and two tests stop the server |
| MongoDB | a container, `models` seeded with gapped ids and a unique index on `id` | `max(int(id)) + 1` against a real unique index |
| MLflow | `mlflow server` from *this* venv, sqlite-backed, artifacts proxied | experiment/run/artifact/registry semantics are what the provisioning activities are tested for |
| the crypto | the production reader, real Argon2id, real `secretstream` | a bundle the producer wrote must open |
What stays stubbed, deliberately: `CoreNotificationHandler`'s Mongo client keeps the existing
`mock_mongo_client`. Notifications are a side channel this suite counts, not a dependency it tests,
and a real notification collection would add a second write path to every absence assertion for
nothing.
The import `Activities` is **never** composed with `patch_mlflow`, `patch_minio_repository` or
`patch_create_engine`. `mlflow.set_tracking_uri` is process-global, so a leak would silently turn a
real assertion into an assertion about a `MagicMock`; `import_activities` asserts the live tracking
URI names the test server before it hands the object over.
**Bundles come from the unit suite.** `tests/helpers/bundle_factory.py` writes them and
`tests/fixtures/bundle/golden_model_v3.sientia` is the producer-emitted one. No third mirrored copy
of the producer's crypto is introduced here.
**Isolation** (the containers are session-scoped): per test the status table is rebuilt, the `models`
collection is dropped and reseeded, and the import prefix is emptied. MLflow state is *not* reset —
experiment and model names are made unique per test instead, because deleted registered models
leave tombstones `get_registered_model` still finds, which would make "second import of the same
model" unreadable.
"""
from __future__ import annotations
import contextlib
import hashlib
import json
import os
import signal
import socket
import subprocess
import sys
import time
import uuid
from collections.abc import Callable, Iterator
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
import mlflow
import pytest
import pytest_asyncio
import requests
from pymongo import MongoClient
from sqlalchemy import Engine, create_engine, text
from temporalio.client import WorkflowHandle
from temporalio.worker import Worker
from testcontainers.mongodb import MongoDbContainer
from testcontainers.postgres import PostgresContainer
from laborious.activities.activities import Activities
from laborious.workflows.import_model import ImportModel
from tests.helpers.bundle_factory import (
DEFAULT_ENVELOPE_KEY,
DEFAULT_PASSWORD,
ZipSpec,
build_envelope,
build_metadata_document,
default_tree,
envelope_key_b64,
make_bundle,
)
# --- constants -----------------------------------------------------------------------------------
REPO_ROOT = Path(__file__).resolve().parent.parent
FIXTURES_DIR = Path(__file__).resolve().parent / 'fixtures'
STATUS_TABLE_DDL = FIXTURES_DIR / 'experiment_run.sql'
STATUS_TABLE_DDL_V12 = FIXTURES_DIR / 'experiment_run_v12.sql'
GOLDEN_BUNDLE = REPO_ROOT / 'tests' / 'fixtures' / 'bundle' / 'golden_model_v3.sientia'
GOLDEN_BUNDLE_PASSWORD = 'Import1Model!Fixture'
BFF_DATABASE = 'sientia-core-mlops-bff'
BFF_USER = 'bff'
BFF_PASSWORD = 'bff'
# Lowercase and hyphens only: `chk_experiment_run_bucket_name_pattern` on the real table.
IMPORT_BUCKET = 'sientia-import'
IMPORT_PREFIX = 'imported_models/'
MODELS_COLLECTION = 'models'
MONGO_DATABASE = 'sientia'
IMPORT_TASK_QUEUE = 'import-model-e2e-queue'
# Deliberately gapped, and deliberately strings: the baseline migration stores `id` as a string and
# `max(int(id)) + 1` must not be confused with a count or with a lexicographic maximum ("9" > "30").
SEEDED_MODEL_IDS = ('1', '2', '7')
MLFLOW_HEALTH_TIMEOUT_SECONDS = 90.0
MLFLOW_HEALTH_POLL_SECONDS = 0.5
IMPORT_ACTIVITY_NAMES = (
'claim_import_status',
'record_import_status',
'record_import_names',
'record_import_terminal_status',
'report_import_status_write_failure',
'download_import_bundle',
'open_import_bundle',
'create_import_experiment',
'upload_import_artifacts',
'register_import_model_version',
'write_import_model_document',
'cleanup_import_files',
'ensure_import_bundle_retention',
)
# --- the BFF Postgres container ------------------------------------------------------------------
@pytest.fixture(scope='session')
def import_bff_container() -> Iterator[PostgresContainer]:
"""The import log database, on a server of its own.
Separate from the shared `postgres_container` for two independent reasons, either sufficient:
the retry-recovery tests **stop and start this server**, which must not disturb the predictions
database every other e2e suite shares session scope with; and "the import touches no Postgres
database other than `sientia-core-mlops-bff`" is a stronger statement when the other database
lives on a different server.
`log_statement=all` is what makes the absence assertions possible: the server log is the only
vantage point that sees every client, including the psycopg pool `Storage` holds.
"""
container = PostgresContainer(
'postgres:15',
username=BFF_USER,
password=BFF_PASSWORD,
dbname=BFF_DATABASE,
).with_command('postgres -c log_statement=all')
container.start()
try:
yield container
finally:
container.stop()
@pytest.fixture(scope='session')
def import_bff_engine(import_bff_container: PostgresContainer) -> Iterator[Engine]:
"""A SQLAlchemy engine on the BFF container.
`pool_pre_ping` matters here rather than being habit: two tests stop the container underneath
this engine, and a stale pooled connection would fail the *fixture* instead of the assertion.
"""
engine = create_engine(import_bff_container.get_connection_url(), pool_pre_ping=True)
try:
yield engine
finally:
engine.dispose()
def _apply_ddl(engine: Engine, files: tuple[Path, ...]) -> None:
"""Drop `public.experiment_run` and rebuild it from `files`, in order."""
with engine.begin() as connection:
connection.execute(text('DROP TABLE IF EXISTS public.experiment_run CASCADE'))
for path in files:
# One transaction per file, because each file is one migration's worth of statements.
with engine.begin() as connection:
connection.execute(text(path.read_text(encoding='utf-8')))
@pytest.fixture
def import_status_table(import_bff_engine: Engine) -> Engine:
"""`public.experiment_run` as `V1`…`V12` leave it, rebuilt for this test.
Rebuilt rather than truncated: the pre-`V12` variant below has to be able to alternate with
this one inside a single session, and `TRUNCATE` cannot take a constraint away. The effect a
test sees is the one the spec asks for — the table holds only the row that test inserted.
"""
_apply_ddl(import_bff_engine, (STATUS_TABLE_DDL, STATUS_TABLE_DDL_V12))
return import_bff_engine
@pytest.fixture
def import_status_table_pre_v12(import_bff_engine: Engine) -> Engine:
"""The same table without `V12`: `run_type` present, the import statuses and pairing absent.
There is no second copy of the base schema — this applies the same file, and simply does not
apply the delta.
"""
_apply_ddl(import_bff_engine, (STATUS_TABLE_DDL,))
return import_bff_engine
InsertStatusRow = Callable[..., int]
@pytest.fixture
def import_insert_status_row(import_bff_engine: Engine) -> InsertStatusRow:
"""Insert one import log row the way the BFF will, and return its `id`.
Defaults mirror what the frontend sends: `PENDING`, `run_type='IMPORT'`, the object key in
`file_name`, and the uploader's digest inside `request_data` — which is where
`claim_import_status` looks for it.
"""
def insert(
*,
object_key: str,
expected_digest: str | None = None,
username: str = 'import.e2e@example.com',
experiment_name: str = 'import-e2e-placeholder',
status: str = 'PENDING',
run_type: str = 'IMPORT',
bucket_name: str = IMPORT_BUCKET,
request_data: dict[str, Any] | None = None,
orchestrator_response_data: dict[str, Any] | None = None,
) -> int:
payload = request_data if request_data is not None else {}
if expected_digest is not None and request_data is None:
payload = {'expected_digest': expected_digest}
now = datetime.now(UTC).replace(tzinfo=None)
statement = text(
'INSERT INTO public.experiment_run '
'(experiment_name, username, status, created_at, updated_at, bucket_name, file_name, '
' request_data, orchestrator_response_data, run_type) '
'VALUES (:experiment_name, :username, :status, :created_at, :updated_at, '
' :bucket_name, :file_name, CAST(:request_data AS JSONB), '
' CAST(:detail AS JSONB), :run_type) '
'RETURNING id'
)
with import_bff_engine.begin() as connection:
return int(
connection.execute(
statement,
{
'experiment_name': experiment_name,
'username': username,
'status': status,
'created_at': now,
'updated_at': now,
'bucket_name': bucket_name,
'file_name': object_key,
'request_data': json.dumps(payload),
'detail': (
json.dumps(orchestrator_response_data)
if orchestrator_response_data is not None
else None
),
'run_type': run_type,
},
).scalar_one()
)
return insert
ReadStatusRow = Callable[[int], dict[str, Any] | None]
@pytest.fixture
def import_read_status_row(import_bff_engine: Engine) -> ReadStatusRow:
"""Read one import log row as a plain mapping, or `None` when it is gone."""
def read(import_run_id: int) -> dict[str, Any] | None:
with import_bff_engine.begin() as connection:
row = (
connection.execute(
text('SELECT * FROM public.experiment_run WHERE id = :id'),
{'id': import_run_id},
)
.mappings()
.one_or_none()
)
return dict(row) if row is not None else None
return read
# --- MongoDB ------------------------------------------------------------------------------------
@pytest.fixture(scope='session')
def import_mongo_container() -> Iterator[MongoDbContainer]:
"""A real MongoDB for the `models` collection the import writes its document into."""
container = MongoDbContainer('mongo:7')
container.start()
try:
yield container
finally:
container.stop()
@pytest.fixture(scope='session')
def import_mongo_client(import_mongo_container: MongoDbContainer) -> Iterator[MongoClient]:
"""A client on the Mongo container, for seeding and for reading assertions back."""
client: MongoClient = MongoClient(import_mongo_container.get_connection_url())
try:
yield client
finally:
client.close()
@pytest.fixture
def import_models_collection(import_mongo_client: MongoClient) -> Any:
"""The `models` collection, dropped and reseeded with gapped ids and a unique index on `id`.
The index is the assertion's teeth: without it "the id continues the collection" and "a name
already present is left alone" would both pass against a collection that quietly took a
duplicate.
"""
collection = import_mongo_client[MONGO_DATABASE][MODELS_COLLECTION]
collection.drop()
collection.insert_many(
[
{
'id': model_id,
'name': f'seeded_model_{model_id}',
'active': True,
'model_config': {
'transform_flavor': 'sklearn',
'predict_flavor': 'joblib',
'retention_minutes': 60,
'target': 'SE',
},
}
for model_id in SEEDED_MODEL_IDS
]
)
collection.create_index('id', unique=True)
return collection
# --- MinIO --------------------------------------------------------------------------------------
@pytest.fixture
def import_bucket(minio_container) -> Iterator[Any]:
"""The import bucket in the shared MinIO container, with its prefix emptied for this test.
The bucket is kept rather than recreated — a bucket carries the retention lifecycle rule the
import sets, and dropping it between tests would make that rule untestable.
"""
client = minio_container.get_client()
if not client.bucket_exists(IMPORT_BUCKET):
client.make_bucket(IMPORT_BUCKET)
_empty_prefix(client, IMPORT_BUCKET, IMPORT_PREFIX)
yield client
_empty_prefix(client, IMPORT_BUCKET, IMPORT_PREFIX)
def _empty_prefix(client: Any, bucket: str, prefix: str) -> None:
"""Remove every object under `prefix`, so a leftover cannot satisfy the next test."""
for obj in client.list_objects(bucket, prefix=prefix, recursive=True):
client.remove_object(bucket, obj.object_name)
UploadBundle = Callable[..., tuple[str, str]]
@pytest.fixture
def import_upload_bundle(import_bucket: Any) -> UploadBundle:
"""Upload bundle bytes under the import prefix; return `(object_key, hex_digest)`.
The digest is computed over the bytes handed in, which is what the frontend promises and what
gate 1 holds it to. A test that wants the digest to *disagree* passes its own.
"""
import io
def upload(payload: bytes, name: str | None = None) -> tuple[str, str]:
object_name = name or f'e2e_{uuid.uuid4().hex}.sientia'
object_key = f'{IMPORT_PREFIX}{object_name}'
import_bucket.put_object(
IMPORT_BUCKET,
object_key,
io.BytesIO(payload),
length=len(payload),
content_type='application/octet-stream',
)
return object_key, hashlib.sha256(payload).hexdigest()
return upload
# --- building bundles ----------------------------------------------------------------------------
@pytest.fixture
def import_envelope() -> Callable[[str], str]:
"""Wrap a bundle password the way the frontend will, under the key the worker is configured with.
The key is `bundle_factory.DEFAULT_ENVELOPE_KEY`, which `import_activities` puts in
`IMPORT_PASSWORD_KEY` — so an envelope built here is one the worker can actually open, and an
envelope built under any other key is a decryption-step failure, which is a test of its own.
"""
def build(password: str = DEFAULT_PASSWORD) -> str:
return build_envelope(password, DEFAULT_ENVELOPE_KEY)
return build
@pytest.fixture
def import_bundle_factory(import_names: dict[str, str]) -> Callable[..., bytes]:
"""Build a `.sientia` bundle carrying *this test's* unique model and experiment names.
Every knob the gate tests need is passed straight through to the shared factory — there is no
second implementation of the producer's header, KDF or framing here, which is the whole point of
reusing `tests/helpers/bundle_factory.py`.
The names default to `import_names` so two tests provisioning models in the same session never
observe each other's registry state (MLflow is deliberately not reset — design E13). A test that
*wants* the collision, like the second-import case, passes the same names twice.
"""
def build(
*,
password: str = DEFAULT_PASSWORD,
metadata: dict[str, Any] | None = None,
parameters: dict[str, Any] | None = None,
drop_metadata_field: str | None = None,
drop_top_level_key: str | None = None,
drop_parameter: str | None = None,
files: dict[str, bytes] | None = None,
include_metadata: bool = True,
symlinks: dict[str, str] | None = None,
absolute_entry: str | None = None,
traversal_entry: str | None = None,
bomb_bytes: int = 0,
extra_files: dict[str, bytes] | None = None,
**header_kwargs: Any,
) -> bytes:
document = build_metadata_document(
metadata={
'model_name': import_names['model_name'],
'experiment_name': import_names['experiment_name'],
**(metadata or {}),
},
parameters={'model_name': import_names['model_name'], **(parameters or {})},
drop_metadata_field=drop_metadata_field,
drop_top_level_key=drop_top_level_key,
drop_parameter=drop_parameter,
)
spec = ZipSpec(
files=default_tree() if files is None else files,
document=document,
include_metadata=include_metadata,
symlinks=symlinks or {},
absolute_entry=absolute_entry,
traversal_entry=traversal_entry,
bomb_bytes=bomb_bytes,
extra_files=extra_files or {},
)
return make_bundle(spec, password, **header_kwargs)
return build
# --- the MLflow tracking server -----------------------------------------------------------------
def _free_port() -> int:
"""A loopback port nothing is listening on, taken from an ephemeral bind."""
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.bind(('127.0.0.1', 0))
return int(sock.getsockname()[1])
@pytest.fixture(scope='session')
def import_mlflow_server(tmp_path_factory: pytest.TempPathFactory) -> Iterator[dict[str, Any]]:
"""`mlflow server` from *this* venv, on a free loopback port.
A subprocess rather than a container because the venv pin **is** the production pin (2.10.1,
the same version the producer pins), so this tests the exact client/server pair that will run
in production — no image tag to drift out of step with `requirements.txt`.
sqlite for the backend store because the model registry is not available on a file-backed
tracking store, and `create_model_version` is half of what the provisioning activities do.
`--serve-artifacts` sends `log_artifacts` over HTTP through the server, as it goes in
production, while leaving the uploaded tree readable straight off the local filesystem.
Yields the host, the port and the artifacts directory. A server that never answers `/health`
fails *here*, with its own stderr attached, rather than making the first test fail obscurely.
"""
root = tmp_path_factory.mktemp('import_mlflow')
artifacts = root / 'mlartifacts'
artifacts.mkdir()
log_path = root / 'mlflow-server.log'
port = _free_port()
host = '127.0.0.1'
tracking_uri = f'http://{host}:{port}'
server: dict[str, Any] = {
'host': f'http://{host}',
'port': str(port),
'tracking_uri': tracking_uri,
'artifacts_dir': artifacts,
'root': root,
'log_path': log_path,
'process': _launch_mlflow(root, host, port, artifacts, log_path),
}
try:
_wait_for_mlflow(server['process'], tracking_uri, log_path)
yield server
finally:
_terminate_group(server['process'])
def _launch_mlflow(
root: Path, host: str, port: int, artifacts: Path, log_path: Path
) -> subprocess.Popen:
"""Start the server process. Separate from the fixture so one test can restart it.
The sqlite file and the artifacts directory are reused across a restart on purpose: a test that
stops the server to fail one activity must not also erase what earlier tests provisioned.
"""
with log_path.open('ab') as log_handle:
return subprocess.Popen( # noqa: S603 - fixed argv, no shell, all values built here
[
sys.executable,
'-m',
'mlflow',
'server',
'--host',
host,
'--port',
str(port),
'--backend-store-uri',
f'sqlite:///{root / "mlflow.db"}',
'--artifacts-destination',
str(artifacts),
'--serve-artifacts',
],
stdout=log_handle,
stderr=subprocess.STDOUT,
cwd=str(root),
# Its own process group: `mlflow server` is gunicorn, so it has children, and killing
# only the parent would leave workers holding the port for the rest of the session.
start_new_session=True,
)
@pytest.fixture
def import_mlflow_stopped(import_mlflow_server: dict[str, Any]) -> Any:
"""A context manager that stops the tracking server and **always** brings it back.
This is design E7's second preference in its purest form: no patch, no mock — the dependency is
genuinely gone, which is the thing that actually happens in production when a pod is replaced.
The restart is in a `finally` because the server is session-scoped: a test that left it down
would fail every test after it, and the failure would point at the wrong place.
"""
@contextlib.contextmanager
def stopped() -> Iterator[None]:
_terminate_group(import_mlflow_server['process'])
try:
yield
finally:
import_mlflow_server['process'] = _launch_mlflow(
import_mlflow_server['root'],
'127.0.0.1',
int(import_mlflow_server['port']),
import_mlflow_server['artifacts_dir'],
import_mlflow_server['log_path'],
)
_wait_for_mlflow(
import_mlflow_server['process'],
import_mlflow_server['tracking_uri'],
import_mlflow_server['log_path'],
)
return stopped
def _wait_for_mlflow(process: subprocess.Popen, tracking_uri: str, log_path: Path) -> None:
"""Poll `/health` until it answers, or fail with what the server actually said."""
deadline = time.monotonic() + MLFLOW_HEALTH_TIMEOUT_SECONDS
while time.monotonic() < deadline:
if process.poll() is not None:
_terminate_group(process)
raise RuntimeError(
f'mlflow server exited with {process.returncode} before answering /health:\n'
f'{log_path.read_text(encoding="utf-8", errors="replace")}'
)
try:
if requests.get(f'{tracking_uri}/health', timeout=2).status_code == 200:
return
except requests.RequestException:
pass
time.sleep(MLFLOW_HEALTH_POLL_SECONDS)
_terminate_group(process)
raise RuntimeError(
f'mlflow server did not answer /health within {MLFLOW_HEALTH_TIMEOUT_SECONDS}s:\n'
f'{log_path.read_text(encoding="utf-8", errors="replace")}'
)
def _terminate_group(process: subprocess.Popen) -> None:
"""Stop the server and every gunicorn worker it forked."""
if process.poll() is not None:
return
with contextlib.suppress(ProcessLookupError, PermissionError):
os.killpg(os.getpgid(process.pid), signal.SIGTERM)
try:
process.wait(timeout=15)
except subprocess.TimeoutExpired:
with contextlib.suppress(ProcessLookupError, PermissionError):
os.killpg(os.getpgid(process.pid), signal.SIGKILL)
process.wait(timeout=15)
# --- per-test naming ----------------------------------------------------------------------------
@pytest.fixture
def import_names(request: pytest.FixtureRequest) -> dict[str, str]:
"""Experiment and model names unique to this test.
MLflow state is deliberately not reset between tests (design E13): deleting a registered model
leaves a tombstone `get_registered_model` still finds, which would make "second import of the
same model" read as a leak rather than as the thing it is testing. Unique names per test buy
the same isolation without the tombstones.
"""
token = uuid.uuid4().hex[:8]
stem = request.node.name.replace('[', '_').replace(']', '')[:24]
return {
'token': token,
'model_name': f'm_{stem}_{token}'[:40],
'experiment_name': f'e_{stem}_{token}'[:40],
}
# --- the import Activities ----------------------------------------------------------------------
@pytest.fixture
def import_limits_override() -> dict[str, str]:
"""Extra import environment variables for this test, applied last so they win.
Empty by default. A test that needs a ceiling it can actually cross — gate 1's object size is
1 GiB, and uploading a gigabyte to prove a limit is not a test anybody runs twice — overrides
this fixture inside its own class, so the narrowed ceiling is scoped to the cases that need it
and every other test keeps the production defaults.
"""
return {}
@pytest_asyncio.fixture
async def import_activities(
monkeypatch: pytest.MonkeyPatch,
import_limits_override: dict[str, str],
tmp_path: Path,
postgres_container,
import_bff_container: PostgresContainer,
import_mongo_container: MongoDbContainer,
import_mlflow_server: dict[str, Any],
import_bucket: Any,
minio_container,
mock_logger,
notification_handler,
) -> Any:
"""An `Activities` whose import path talks to nothing fake.
Note what is *not* in the fixture's parameter list: `patch_mlflow`, `patch_minio_repository`
and `patch_create_engine`. Composing any of them would replace exactly the thing under test,
and because `mlflow.set_tracking_uri` is process-global the substitution would be silent — so
the live tracking URI is asserted below rather than assumed.
The import configuration is built by the production `build_import_config` /
`build_import_status_config` from environment variables set here, so the variable *names* are
covered too and not just the dictionaries they produce.
"""
from laborious.utils.connectors_config import build_import_config, build_import_status_config
minio_port = minio_container.get_exposed_port(9000)
monkeypatch.setenv('IMPORT_PASSWORD_KEY', envelope_key_b64(DEFAULT_ENVELOPE_KEY))
monkeypatch.setenv('IMPORT_BUNDLE_BUCKET', IMPORT_BUCKET)
monkeypatch.setenv('IMPORT_BUNDLE_PREFIX', IMPORT_PREFIX)
monkeypatch.setenv('IMPORT_MODELS_COLLECTION', MODELS_COLLECTION)
monkeypatch.setenv('IMPORT_WORK_DIR', str(tmp_path / 'import-work'))
monkeypatch.setenv('IMPORT_STATUS_DB_HOST', import_bff_container.get_container_host_ip())
monkeypatch.setenv('IMPORT_STATUS_DB_PORT', str(import_bff_container.get_exposed_port(5432)))
monkeypatch.setenv('IMPORT_STATUS_DB_NAME', BFF_DATABASE)
monkeypatch.setenv('IMPORT_STATUS_DB_USER', BFF_USER)
monkeypatch.setenv('IMPORT_STATUS_DB_PASSWORD', BFF_PASSWORD)
for name, value in import_limits_override.items():
monkeypatch.setenv(name, value)
activities = Activities(
# The predictions database, which the import must never touch. It is pointed at a real
# server on purpose: "no statement reached it" is only worth asserting when there is a log
# that would have shown one.
postgres_config={
'host': 'localhost',
'port': postgres_container.get_exposed_port(5432),
'user': 'test',
'password': 'test',
'dbname': 'test',
'min_connections': 1,
'max_connections': 5,
},
mlflow_config={
'host': import_mlflow_server['host'],
'port': import_mlflow_server['port'],
'username': 'e2e',
'password': 'e2e',
},
minio_config={
'endpoint_url': f'localhost:{minio_port}',
'access_key': 'minioadmin',
'secret_key': 'minioadmin',
'default_bucket': IMPORT_BUCKET,
'retention_hours': 24,
'secure': False,
},
opc_config={},
pi_web_api_config={
'base_url': 'http://localhost:8080',
'auth_type': 'bearer',
'auth_token': 'e2e',
},
logger=mock_logger,
notification_handler=notification_handler,
mongo_config={
'connection_string': import_mongo_container.get_connection_url(),
'database_name': MONGO_DATABASE,
},
import_config=build_import_config(),
import_status_config=build_import_status_config(),
)
# If a mocked-MLflow fixture ever leaks into an import test, this is where it stops.
assert mlflow.get_tracking_uri() == import_mlflow_server['tracking_uri'], (
f'MLflow tracking URI is {mlflow.get_tracking_uri()!r}, not the test server '
f'{import_mlflow_server["tracking_uri"]!r} — a mocked MLflow fixture has leaked in'
)
try:
yield activities
finally:
await activities.shutdown()
@pytest_asyncio.fixture
async def import_worker(temporal_test_env, import_activities: Any) -> Any:
"""An `ImportModel` worker with the thirteen import activities, on the import queue."""
async with Worker(
temporal_test_env.client,
task_queue=IMPORT_TASK_QUEUE,
workflows=[ImportModel],
activities=[getattr(import_activities, name) for name in IMPORT_ACTIVITY_NAMES],
) as worker:
yield worker
# --- running an import --------------------------------------------------------------------------
@pytest.fixture
def import_build_input() -> Callable[..., dict[str, Any]]:
"""The workflow input, as the frontend assembles it."""
def build(
*,
import_run_id: int,
object_key: str,
expected_digest: str | None,
password_envelope: str,
bucket: str | None = IMPORT_BUCKET,
**extra: Any,
) -> dict[str, Any]:
payload: dict[str, Any] = {
'import_run_id': import_run_id,
'object_key': object_key,
'password_envelope': password_envelope,
}
if expected_digest is not None:
payload['expected_digest'] = expected_digest
if bucket is not None:
payload['bucket'] = bucket
payload.update(extra)
return payload
return build
@pytest.fixture
def import_execute(temporal_test_env, import_worker: Any) -> Callable[..., Any]:
"""Run one import to completion and hand back the handle, whatever the outcome.
The handle rather than the result, because half this suite asserts on *how* a run failed —
the error type Temporal recorded, the event history — and `execute_workflow` would have thrown
that away.
"""
async def execute(payload: dict[str, Any], workflow_id: str | None = None) -> WorkflowHandle:
handle = await temporal_test_env.client.start_workflow(
ImportModel.run,
payload,
id=workflow_id or f'import-e2e-{uuid.uuid4().hex}',
task_queue=IMPORT_TASK_QUEUE,
)
with contextlib.suppress(Exception):
# The exception is the assertion's subject in the negative cases; the caller reads it
# back off the handle so a failure here is never mistaken for a harness error.
await handle.result()
return handle
return execute
@pytest.fixture
def import_history_events() -> Callable[[WorkflowHandle], Any]:
"""Every event of a run's history, flattened to the text the secrecy assertions scan.
Returns `(events, blob)`: the event list for structural assertions, and one string holding
every payload in the history for "this value appears nowhere" assertions.
"""
async def fetch(handle: WorkflowHandle) -> tuple[list[Any], str]:
events = [event async for event in handle.fetch_history_events()]
return events, '\n'.join(str(event) for event in events)
return fetch
# --- reading a failure ---------------------------------------------------------------------------
#
# Temporal wraps what actually went wrong: `WorkflowFailureError` -> `ActivityError` ->
# `ApplicationError`, and only the innermost one carries the sentence and the declared error type.
# `str(exception)` on the outermost is "Workflow execution failed", which is true and useless, so
# every negative assertion in this suite goes through these two functions instead.
def failure_chain(error: BaseException) -> list[BaseException]:
"""The exception and everything it was caused by, outermost first."""
chain: list[BaseException] = []
current: BaseException | None = error
while current is not None and not any(current is seen for seen in chain):
chain.append(current)
current = current.__cause__
return chain
def failure_text(error: BaseException) -> str:
"""Every link of the chain as `Type: message`, joined — what a substring assertion reads."""
return '\n'.join(f'{type(part).__name__}: {part}' for part in failure_chain(error))
def failure_types(error: BaseException) -> set[str]:
"""Every Python class name in the chain, plus every `ApplicationError.type` it declares.
The declared `type` is the one that survives serialisation into the event history, so it is
what "the run failed with the not-claimable error type" actually means.
"""
names: set[str] = set()
for part in failure_chain(error):
names.add(type(part).__name__)
declared = getattr(part, 'type', None)
if isinstance(declared, str):
names.add(declared)
return names
# --- the absence assertion ----------------------------------------------------------------------
@pytest.fixture
def import_assert_nothing_provisioned(
import_mlflow_server: dict[str, Any],
import_models_collection: Any,
import_bucket: Any,
tmp_path: Path,
) -> Callable[..., None]:
"""Assert a rejected import left nothing behind, in one call over all four places.
One helper rather than four assertions per test, because "nothing was provisioned" is a single
claim and a test that checks three of its four halves reads as if it checked all four.
"""
def assert_nothing(
*,
experiment_name: str,
model_name: str,
object_key: str | None = None,
seeded_ids: tuple[str, ...] = SEEDED_MODEL_IDS,
) -> None:
client = mlflow.tracking.MlflowClient(tracking_uri=import_mlflow_server['tracking_uri'])
experiment = client.get_experiment_by_name(experiment_name)
assert experiment is None, f'experiment {experiment_name!r} was created'
versions = client.search_model_versions(f"name='{model_name}'")
assert not list(versions), f'registry holds versions of {model_name!r}'
documents = list(import_models_collection.find({}, {'_id': False}))
assert {str(document['id']) for document in documents} == set(seeded_ids), (
f'the models collection changed: {documents}'
)
assert not [d for d in documents if d.get('name') == model_name], (
f'a model document for {model_name!r} was written'
)
# The uploaded object itself stays — the frontend put it there and retention removes it.
# What must not be there is anything the *import* wrote under the prefix.
stored = {
obj.object_name
for obj in import_bucket.list_objects(
IMPORT_BUCKET, prefix=IMPORT_PREFIX, recursive=True
)
}
assert stored <= ({object_key} if object_key else set()), (
f'the import wrote objects under {IMPORT_PREFIX}: {stored}'
)
work_dir = tmp_path / 'import-work'
leftovers = list(work_dir.rglob('*')) if work_dir.exists() else []
assert not leftovers, f'the worker filesystem still holds {leftovers}'
return assert_nothing
# --- reading the golden fixture -----------------------------------------------------------------
@pytest.fixture(scope='session')
def import_golden_bundle() -> dict[str, Any]:
"""The producer-emitted bundle: its bytes, its digest and the password it was written with.
This is the only place in the suite that pays the producer's real Argon2id cost (256 MiB,
3 passes). Everything else runs the factory's cheap cost, which is the right split — the
reader honours whatever the header declares, up to its ceiling.
"""
payload = GOLDEN_BUNDLE.read_bytes()
return {
'bytes': payload,
'digest': hashlib.sha256(payload).hexdigest(),
'password': GOLDEN_BUNDLE_PASSWORD,
'path': GOLDEN_BUNDLE,
}

196
e2e/test_import_harness_smoke.py Normal file
View File

@@ -0,0 +1,196 @@
"""Does the harness itself come up? Diagnosable separately from the suite that runs on it.
Four containers, an `mlflow server` subprocess and a Temporal worker is a lot of moving parts, and
when one of them fails to start the first *real* test fails somewhere deep inside an activity with
a message about something else. These tests exist so that a broken harness says so in its own
words: each one checks one piece is alive and does nothing with the import path.
The last test is the exception — it runs a real workflow, but with an input designed to be rejected
at the very first step, so what it proves is only that the wiring reaches the first activity.
"""
from __future__ import annotations
import json
from typing import Any
import mlflow
import pytest
from sqlalchemy import text
from temporalio.client import WorkflowFailureError
from e2e.import_harness import (
BFF_DATABASE,
IMPORT_ACTIVITY_NAMES,
IMPORT_BUCKET,
IMPORT_PREFIX,
IMPORT_TASK_QUEUE,
SEEDED_MODEL_IDS,
failure_text,
failure_types,
)
pytestmark = [pytest.mark.import_e2e, pytest.mark.asyncio]
class TestContainers:
"""Each dependency is real, reachable, and the one the harness says it is."""
async def test_bff_container_holds_the_status_table(self, import_status_table: Any) -> None:
with import_status_table.begin() as connection:
database = connection.execute(text('SELECT current_database()')).scalar_one()
columns = {
row[0]
for row in connection.execute(
text(
'SELECT column_name FROM information_schema.columns '
"WHERE table_schema = 'public' AND table_name = 'experiment_run'"
)
)
}
assert database == BFF_DATABASE
assert {'id', 'status', 'run_type', 'request_data', 'orchestrator_response_data'} <= columns
async def test_bff_container_logs_every_statement(self, import_bff_container: Any) -> None:
"""`log_statement=all` — without it the absence assertions have no vantage point."""
result = import_bff_container.exec(
['psql', '-U', 'bff', '-d', BFF_DATABASE, '-tAc', 'SHOW log_statement']
)
assert b'all' in result.output
async def test_the_v12_delta_is_what_separates_the_two_tables(
self, import_status_table_pre_v12: Any
) -> None:
"""The pre-`V12` table has `run_type` and refuses the import terminal statuses."""
with import_status_table_pre_v12.begin() as connection:
constraints = {
row[0]
for row in connection.execute(
text(
'SELECT conname FROM pg_constraint '
"WHERE conrelid = 'public.experiment_run'::regclass"
)
)
}
columns = {
row[0]
for row in connection.execute(
text(
'SELECT column_name FROM information_schema.columns '
"WHERE table_schema = 'public' AND table_name = 'experiment_run'"
)
)
}
assert 'run_type' in columns
assert 'chk_experiment_run_terminal_status_run_type' not in constraints
async def test_the_seeded_collection_has_gapped_ids_and_a_unique_index(
self, import_models_collection: Any
) -> None:
ids = {
str(document['id']) for document in import_models_collection.find({}, {'_id': False})
}
assert ids == set(SEEDED_MODEL_IDS)
unique_keys = {
tuple(index['key'].keys())
for index in import_models_collection.list_indexes()
if index.get('unique')
}
assert ('id',) in unique_keys
async def test_the_import_bucket_exists_and_its_prefix_is_empty(
self, import_bucket: Any
) -> None:
assert import_bucket.bucket_exists(IMPORT_BUCKET)
assert not list(
import_bucket.list_objects(IMPORT_BUCKET, prefix=IMPORT_PREFIX, recursive=True)
)
async def test_uploading_a_bundle_returns_a_key_and_its_digest(
self, import_upload_bundle: Any, import_bucket: Any
) -> None:
object_key, digest = import_upload_bundle(b'not a real bundle, just bytes')
assert object_key.startswith(IMPORT_PREFIX)
assert len(digest) == 64
assert import_bucket.stat_object(IMPORT_BUCKET, object_key).size > 0
async def test_the_mlflow_server_answers_and_serves_a_registry(
self, import_mlflow_server: dict[str, Any]
) -> None:
client = mlflow.tracking.MlflowClient(tracking_uri=import_mlflow_server['tracking_uri'])
experiment_id = client.create_experiment('harness_smoke_experiment')
run = client.create_run(experiment_id, run_name='harness-smoke-run')
client.create_registered_model('harness_smoke_model')
version = client.create_model_version(
'harness_smoke_model', f'runs:/{run.info.run_id}/prediction_model', run.info.run_id
)
assert version.version == '1'
assert version.current_stage in (None, 'None')
class TestActivitiesAndWorker:
"""The object under test is built from the real dependencies, and the worker registers it."""
async def test_activities_point_at_the_real_dependencies(
self, import_activities: Any, import_mlflow_server: dict[str, Any]
) -> None:
assert mlflow.get_tracking_uri() == import_mlflow_server['tracking_uri']
assert import_activities.import_config['bucket'] == IMPORT_BUCKET
assert import_activities.import_config['prefix'] == IMPORT_PREFIX
assert import_activities.status_config['dbname'] == BFF_DATABASE
assert import_activities.mongo_repository is not None
# Not a mock: a MagicMock would satisfy any attribute access, so the type is the assertion.
assert type(import_activities.import_minio_repository).__name__ == 'MinioRepository'
async def test_every_import_activity_is_registered(self, import_activities: Any) -> None:
for name in IMPORT_ACTIVITY_NAMES:
assert callable(getattr(import_activities, name)), f'{name} is missing'
async def test_the_worker_polls_the_import_queue(self, import_worker: Any) -> None:
assert import_worker.task_queue == IMPORT_TASK_QUEUE
async def test_a_workflow_input_reaches_the_first_activity(
self,
import_status_table: Any,
import_execute: Any,
import_build_input: Any,
import_read_status_row: Any,
) -> None:
"""The claim runs, and the row it could not claim proves the wiring is live.
An id no row carries is the cheapest input that gets past workflow validation and into
`claim_import_status`, so a run that ends failed *there* means the client, the worker, the
workflow, the activity and the BFF connection are all in place. Nothing else is exercised.
"""
handle = await import_execute(
import_build_input(
import_run_id=987654321,
object_key=f'{IMPORT_PREFIX}nothing.sientia',
expected_digest='0' * 64,
password_envelope='not-a-real-envelope',
)
)
with pytest.raises(WorkflowFailureError) as failure:
await handle.result()
text_of_failure = failure_text(failure.value)
assert 'claim' in text_of_failure.lower()
assert '987654321' in text_of_failure
assert 'ImportLogRowNotClaimableError' in failure_types(failure.value)
assert import_read_status_row(987654321) is None
async def test_the_insert_helper_writes_a_claimable_row(
self,
import_status_table: Any,
import_insert_status_row: Any,
import_read_status_row: Any,
) -> None:
object_key = f'{IMPORT_PREFIX}claimable.sientia'
row_id = import_insert_status_row(object_key=object_key, expected_digest='a' * 64)
row = import_read_status_row(row_id)
assert row is not None
assert row['status'] == 'PENDING'
assert row['run_type'] == 'IMPORT'
assert row['file_name'] == object_key
request_data = row['request_data']
payload = json.loads(request_data) if isinstance(request_data, str) else request_data
assert payload['expected_digest'] == 'a' * 64

214
e2e/test_import_model_digest.py Normal file
View File

@@ -0,0 +1,214 @@
"""The digest is a promise about the bytes in the bucket, and it is kept against the bucket.
Three ways the promise can be broken, and where each is caught:
- the uploader computed the digest over *other* bytes — caught at `download`, after the object is
fetched and before its header is parsed, so a wrong file never costs an Argon2id pass;
- the object was replaced between the upload and the run — the same failure, which is the point:
the check is against what is stored now, not against what was stored when the row was written;
- no digest at all — caught at `received`, before anything is downloaded, because a request with no
digest is not a request this workflow can honour.
The first two matter because MinIO is real here. A mocked object store would return whatever the
test told it to, and "the digest is enforced against the bytes that were uploaded" would be a
statement about the mock.
"""
from __future__ import annotations
import hashlib
import io
import json
from typing import Any
import pytest
from temporalio.client import WorkflowFailureError
from e2e.import_harness import IMPORT_BUCKET, failure_text
from laborious.activities.model_import_errors import IMPORT_FAILURE_CATALOG
from laborious.utils.bundle.steps import ImportStep
pytestmark = [pytest.mark.import_e2e, pytest.mark.asyncio]
VALIDATION_ERROR_STATUS = 'ORCHESTRATOR_VALIDATION_ERROR'
FACTORY_PASSWORD = 'Factory1Model!Test'
def _as_dict(value: Any) -> dict[str, Any]:
return json.loads(value) if isinstance(value, str) else (value or {})
def _assert_step(row: dict[str, Any], step: ImportStep, workflow_id: str) -> None:
code, reason = IMPORT_FAILURE_CATALOG[step]
detail = _as_dict(row['orchestrator_response_data'])['import']
assert row['status'] == VALIDATION_ERROR_STATUS, row['status']
assert detail['step'] == step.value, detail
assert detail['code'] == code.value, detail
assert detail['workflow_id'] == workflow_id
assert row['error_message'] == reason
class TestWrongDigest:
"""A digest computed over bytes that are not the ones in the bucket."""
async def test_a_digest_from_another_file_fails_at_download(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
) -> None:
"""The bundle is perfectly valid; only the digest is another file's.
The row carries the same wrong digest as the input, so the request and the record agree and
the `received` guard passes — otherwise this would be a `received` failure about a
contradictory request instead of the download-step digest check it is meant to exercise.
"""
payload = import_bundle_factory()
other_digest = hashlib.sha256(
import_bundle_factory(metadata={'run_id': 'other'})
).hexdigest()
assert other_digest != hashlib.sha256(payload).hexdigest()
object_key, _ = import_upload_bundle(payload)
import_run_id = import_insert_status_row(
object_key=object_key, expected_digest=other_digest
)
handle = await import_execute(
import_build_input(
import_run_id=import_run_id,
object_key=object_key,
expected_digest=other_digest,
password_envelope=import_envelope(FACTORY_PASSWORD),
)
)
with pytest.raises(WorkflowFailureError) as failure:
await handle.result()
row = import_read_status_row(import_run_id)
assert row is not None
_assert_step(row, ImportStep.DOWNLOAD, handle.id)
# No header was parsed: the failure names the digest, not the format. If the digest check
# ran *after* the header the message here would be about magic bytes or a version.
text_of_failure = failure_text(failure.value).lower()
assert 'digest' in text_of_failure, text_of_failure
assert 'marker' not in text_of_failure and 'version' not in text_of_failure
# The names are still unwritten, which is the other half of "no header was parsed": the
# bundle was never opened, so nothing knew what to call it.
assert row['run_name'] is None
assert row['experiment_name'] == 'import-e2e-placeholder'
import_assert_nothing_provisioned(
experiment_name=import_names['experiment_name'],
model_name=import_names['model_name'],
object_key=object_key,
)
class TestOverwrittenObject:
"""The bytes changed after the row was written. Only a real object store can show this."""
async def test_replacing_the_object_after_upload_fails_at_download(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_bucket: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
) -> None:
original = import_bundle_factory()
object_key, digest = import_upload_bundle(original)
import_run_id = import_insert_status_row(object_key=object_key, expected_digest=digest)
# Same key, different bytes — also a valid bundle, so nothing but the digest can tell.
replacement = import_bundle_factory(metadata={'run_id': 'ff00ff00ff00ff00ff00ff00ff00ff00'})
assert hashlib.sha256(replacement).hexdigest() != digest
import_bucket.put_object(
IMPORT_BUCKET,
object_key,
io.BytesIO(replacement),
length=len(replacement),
content_type='application/octet-stream',
)
handle = await import_execute(
import_build_input(
import_run_id=import_run_id,
object_key=object_key,
expected_digest=digest,
password_envelope=import_envelope(FACTORY_PASSWORD),
)
)
with pytest.raises(WorkflowFailureError):
await handle.result()
row = import_read_status_row(import_run_id)
assert row is not None
_assert_step(row, ImportStep.DOWNLOAD, handle.id)
import_assert_nothing_provisioned(
experiment_name=import_names['experiment_name'],
model_name=import_names['model_name'],
object_key=object_key,
)
class TestMissingDigest:
"""No digest is refused at the door, before a single byte is fetched."""
async def test_an_absent_digest_fails_at_received_with_nothing_downloaded(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
tmp_path: Any,
) -> None:
object_key, digest = import_upload_bundle(import_bundle_factory())
import_run_id = import_insert_status_row(object_key=object_key, expected_digest=digest)
handle = await import_execute(
import_build_input(
import_run_id=import_run_id,
object_key=object_key,
expected_digest=None,
password_envelope=import_envelope(FACTORY_PASSWORD),
)
)
with pytest.raises(WorkflowFailureError) as failure:
await handle.result()
row = import_read_status_row(import_run_id)
assert row is not None
_assert_step(row, ImportStep.RECEIVED, handle.id)
assert 'expected_digest' in failure_text(failure.value)
# Nothing downloaded: the work directory was never even created.
work_dir = tmp_path / 'import-work'
assert not work_dir.exists() or not list(work_dir.rglob('*'))
import_assert_nothing_provisioned(
experiment_name=import_names['experiment_name'],
model_name=import_names['model_name'],
object_key=object_key,
)

675
e2e/test_import_model_gates.py Normal file
View File

@@ -0,0 +1,675 @@
"""One rejection per gate, against a real object, a real archive and a real filesystem.
The gates are unit-tested already. What only a real run can show is that a rejection is *recorded*
correctly and that it provisions nothing: the row ends `ORCHESTRATOR_VALIDATION_ERROR` naming that
gate's step, its catalog code and its catalog sentence, and no experiment, run, artifact, registry
version, model document or extracted file survives.
Every case here is hostile **input** — design E7's first preference. Nothing is patched, no
dependency is broken; the bundles are built by the shared factory and the gates do the rest. The
one exception to "input only" is the object-size ceiling, where the ceiling itself is narrowed
rather than a gigabyte being uploaded; that is a configuration change, not a patched code path, and
it is scoped to the one class that needs it.
`_assert_rejected` is where the shared claim of task 5.8 lives — terminal status, recorded step,
recorded code, catalog sentence, and `assert_nothing_provisioned` — so no case can quietly assert
four fifths of it.
"""
from __future__ import annotations
import json
import tempfile
from typing import Any
import pytest
from temporalio.client import WorkflowFailureError
from e2e.import_harness import (
IMPORT_BUCKET,
IMPORT_PREFIX,
failure_text,
)
from laborious.activities.model_import_errors import IMPORT_FAILURE_CATALOG
from laborious.utils.bundle.steps import ImportStep
pytestmark = [pytest.mark.import_e2e, pytest.mark.asyncio]
VALIDATION_ERROR_STATUS = 'ORCHESTRATOR_VALIDATION_ERROR'
FACTORY_PASSWORD = 'Factory1Model!Test'
def _as_dict(value: Any) -> dict[str, Any]:
return json.loads(value) if isinstance(value, str) else (value or {})
async def _run_rejected_import(
*,
payload: bytes,
password: str,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
envelope_password: str | None = None,
) -> tuple[Any, int, str]:
"""Upload a hostile bundle, insert the row, run the import and expect it to fail."""
object_key, digest = import_upload_bundle(payload)
import_run_id = import_insert_status_row(object_key=object_key, expected_digest=digest)
handle = await import_execute(
import_build_input(
import_run_id=import_run_id,
object_key=object_key,
expected_digest=digest,
password_envelope=import_envelope(
envelope_password if envelope_password is not None else password
),
)
)
return handle, import_run_id, object_key
async def _assert_rejected(
*,
handle: Any,
import_run_id: int,
object_key: str,
step: ImportStep,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
import_names: dict[str, str],
experiment_name: str | None = None,
model_name: str | None = None,
) -> dict[str, Any]:
"""The whole of task 5.8, in one call. Returns the row so a case can add to it."""
with pytest.raises(WorkflowFailureError):
await handle.result()
row = import_read_status_row(import_run_id)
assert row is not None, 'the rejection lost the row it was recorded on'
assert row['status'] == VALIDATION_ERROR_STATUS, (
f'terminal status is {row["status"]!r}, expected {VALIDATION_ERROR_STATUS!r}'
)
expected_code, expected_reason = IMPORT_FAILURE_CATALOG[step]
detail = _as_dict(row['orchestrator_response_data'])['import']
assert detail['step'] == step.value, (
f'recorded step is {detail["step"]!r}, expected {step.value!r}'
)
assert detail['code'] == expected_code.value, (
f'recorded code is {detail["code"]!r}, expected {expected_code.value!r}'
)
assert detail['workflow_id'] == handle.id
assert row['error_message'] == expected_reason, (
'error_message is not the catalog sentence for this step'
)
import_assert_nothing_provisioned(
experiment_name=experiment_name or import_names['experiment_name'],
model_name=model_name or import_names['model_name'],
object_key=object_key,
)
return row
class TestGate1ObjectLimits:
"""Gate 1 rejects before the object's body is fetched at all."""
@pytest.fixture
def import_limits_override(self) -> dict[str, str]:
"""A 1 KiB ceiling, so a factory bundle is over it.
The production ceiling is 1 GiB. Narrowing it is the only way to cross gate 1 without
uploading a gigabyte, and it changes a configured limit rather than a code path — the same
comparison runs, against a different number.
"""
return {'IMPORT_MAX_OBJECT_BYTES': '1024'}
async def test_an_oversized_object_is_rejected_at_download(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
) -> None:
payload = import_bundle_factory()
assert len(payload) > 1024, 'the fixture bundle must exceed the narrowed ceiling'
handle, import_run_id, object_key = await _run_rejected_import(
payload=payload,
password=FACTORY_PASSWORD,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
await _assert_rejected(
handle=handle,
import_run_id=import_run_id,
object_key=object_key,
step=ImportStep.DOWNLOAD,
import_read_status_row=import_read_status_row,
import_assert_nothing_provisioned=import_assert_nothing_provisioned,
import_names=import_names,
)
class TestGate2Header:
"""Four header rejections, all landing on `download` — the step that says "not these bytes"."""
@pytest.mark.parametrize(
('case', 'header_kwargs'),
[
('wrong magic', {'magic': b'NOTSIENT'}),
('unsupported version', {'format_version': 9}),
('kdf memory above the ceiling', {'declared_memlimit': 512 * 1024 * 1024}),
('kdf passes above the ceiling', {'declared_opslimit': 9}),
('digest not of the chunk stream', {'digest_override': bytes(32)}),
],
)
async def test_a_malformed_header_is_rejected_at_download(
self,
case: str,
header_kwargs: dict[str, Any],
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
) -> None:
handle, import_run_id, object_key = await _run_rejected_import(
payload=import_bundle_factory(**header_kwargs),
password=FACTORY_PASSWORD,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
row = await _assert_rejected(
handle=handle,
import_run_id=import_run_id,
object_key=object_key,
step=ImportStep.DOWNLOAD,
import_read_status_row=import_read_status_row,
import_assert_nothing_provisioned=import_assert_nothing_provisioned,
import_names=import_names,
)
# None of these may borrow the decryption sentence: the password is not implicated.
assert 'password' not in (row['error_message'] or '').lower(), case
class TestGate3Decryption:
"""The password, the tag and the framing — the three ways a stream refuses to open."""
async def test_a_wrong_password_is_rejected_at_decryption(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
) -> None:
"""The envelope opens — it is the *bundle* password inside it that is wrong.
Worth separating: an envelope the worker cannot open is also a decryption-step failure, but
for a different reason, and only this one proves the KDF actually ran and produced a key
the stream rejected.
"""
handle, import_run_id, object_key = await _run_rejected_import(
payload=import_bundle_factory(password=FACTORY_PASSWORD),
password=FACTORY_PASSWORD,
envelope_password='Wrong1Password!Entirely',
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
await _assert_rejected(
handle=handle,
import_run_id=import_run_id,
object_key=object_key,
step=ImportStep.DECRYPTION,
import_read_status_row=import_read_status_row,
import_assert_nothing_provisioned=import_assert_nothing_provisioned,
import_names=import_names,
)
async def test_a_tampered_chunk_is_rejected_at_decryption(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
) -> None:
"""One flipped byte inside the stream, with the header digest repaired to match.
Repairing the digest is what makes this gate 3 rather than gate 2: the file is
self-consistent, so only the AEAD can tell it was altered.
"""
from tests.helpers.bundle_factory import tamper_ciphertext
handle, import_run_id, object_key = await _run_rejected_import(
payload=tamper_ciphertext(import_bundle_factory()),
password=FACTORY_PASSWORD,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
await _assert_rejected(
handle=handle,
import_run_id=import_run_id,
object_key=object_key,
step=ImportStep.DECRYPTION,
import_read_status_row=import_read_status_row,
import_assert_nothing_provisioned=import_assert_nothing_provisioned,
import_names=import_names,
)
async def test_a_stream_without_its_final_tag_is_rejected_at_decryption(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
) -> None:
"""Every chunk framed as MESSAGE, so the stream claims more is coming and never ends.
The digest cannot see this — the bytes on disk are exactly the ones it covers. Only the
`secretstream` framing knows the file is incomplete.
"""
handle, import_run_id, object_key = await _run_rejected_import(
payload=import_bundle_factory(drop_final_tag=True),
password=FACTORY_PASSWORD,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
await _assert_rejected(
handle=handle,
import_run_id=import_run_id,
object_key=object_key,
step=ImportStep.DECRYPTION,
import_read_status_row=import_read_status_row,
import_assert_nothing_provisioned=import_assert_nothing_provisioned,
import_names=import_names,
)
class TestGate4ArchiveInspection:
"""The central directory alone decides — nothing has been written to disk yet."""
@pytest.fixture
def import_limits_override(self) -> dict[str, str]:
"""A 1 MiB declared-expansion ceiling, for the oversized-expansion case in this class."""
return {'IMPORT_MAX_UNCOMPRESSED_BYTES': str(1024 * 1024)}
@pytest.mark.parametrize(
('case', 'kwargs'),
[
('entry escaping the tree', {'traversal_entry': 'artifacts/../../escaped.csv'}),
('absolute entry path', {'absolute_entry': '/opt/absolute.csv'}),
('symbolic link', {'symlinks': {'artifacts/link.csv': 'artifacts/train_data.csv'}}),
('declared expansion over the ceiling', {'bomb_bytes': 4 * 1024 * 1024}),
],
)
async def test_a_hostile_archive_is_rejected_before_anything_is_extracted(
self,
case: str,
kwargs: dict[str, Any],
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
tmp_path: Any,
) -> None:
handle, import_run_id, object_key = await _run_rejected_import(
payload=import_bundle_factory(**kwargs),
password=FACTORY_PASSWORD,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
await _assert_rejected(
handle=handle,
import_run_id=import_run_id,
object_key=object_key,
step=ImportStep.ARCHIVE_INSPECTION,
import_read_status_row=import_read_status_row,
import_assert_nothing_provisioned=import_assert_nothing_provisioned,
import_names=import_names,
)
# "Nothing extracted" is stronger than "nothing provisioned": not one member reached disk.
work_dir = tmp_path / 'import-work'
assert not list(work_dir.rglob('*')) if work_dir.exists() else True, case
class TestGate5Extraction:
"""A member that gate 4 passed and extraction still cannot place.
Gate 5 is about *placement*, not size: containment on the resolved path, and never overwriting
a file already written. A zip carrying the same name twice passes gate 4 — which checks names
and modes, not duplicates — and is refused here on the second member.
"""
async def test_a_duplicate_member_is_rejected_at_extraction(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
) -> None:
from tests.helpers.bundle_factory import default_tree
# `bomb_bytes` writes `artifacts/bomb.csv`, and `files` already carries that name, so the
# archive declares it twice. Small enough that no size ceiling is involved.
files = dict(default_tree())
files['artifacts/bomb.csv'] = b'first\n'
handle, import_run_id, object_key = await _run_rejected_import(
payload=import_bundle_factory(files=files, bomb_bytes=64),
password=FACTORY_PASSWORD,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
await _assert_rejected(
handle=handle,
import_run_id=import_run_id,
object_key=object_key,
step=ImportStep.EXTRACTION,
import_read_status_row=import_read_status_row,
import_assert_nothing_provisioned=import_assert_nothing_provisioned,
import_names=import_names,
)
class TestGate6Structure:
"""The layout and the metadata schema, checked on the extracted tree."""
@pytest.mark.parametrize(
('case', 'kwargs'),
[
('no metadata.json', {'include_metadata': False}),
('metadata missing run_id', {'drop_metadata_field': 'run_id'}),
('parameters missing target_variable', {'drop_parameter': 'target_variable'}),
],
)
async def test_an_incomplete_bundle_is_rejected_at_structure_validation(
self,
case: str,
kwargs: dict[str, Any],
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
) -> None:
handle, import_run_id, object_key = await _run_rejected_import(
payload=import_bundle_factory(**kwargs),
password=FACTORY_PASSWORD,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
await _assert_rejected(
handle=handle,
import_run_id=import_run_id,
object_key=object_key,
step=ImportStep.STRUCTURE_VALIDATION,
import_read_status_row=import_read_status_row,
import_assert_nothing_provisioned=import_assert_nothing_provisioned,
import_names=import_names,
)
async def test_a_bundle_without_prediction_model_is_rejected(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
) -> None:
"""Without `prediction_model` there is nothing to register a version against."""
from tests.helpers.bundle_factory import default_tree
files = {
name: content
for name, content in default_tree().items()
if not name.startswith('artifacts/prediction_model/')
}
handle, import_run_id, object_key = await _run_rejected_import(
payload=import_bundle_factory(files=files),
password=FACTORY_PASSWORD,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
await _assert_rejected(
handle=handle,
import_run_id=import_run_id,
object_key=object_key,
step=ImportStep.STRUCTURE_VALIDATION,
import_read_status_row=import_read_status_row,
import_assert_nothing_provisioned=import_assert_nothing_provisioned,
import_names=import_names,
)
async def test_an_over_long_experiment_name_never_reaches_the_varchar_column(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
import_bff_engine: Any,
) -> None:
"""A 60-character experiment name is refused at gate 6, not by Postgres three steps later.
`experiment_name` is `VARCHAR(50)` and the import writes the bundle's own name onto the row
*before* provisioning. If the gate did not enforce the column's limit, the names write would
be a `DataError` — non-retryable, recorded as a status failure, on a row whose import was
otherwise fine. Asserting the row's `experiment_name` is untouched is what proves the write
never happened.
"""
from sqlalchemy import text
long_name = 'x' * 60
handle, import_run_id, object_key = await _run_rejected_import(
payload=import_bundle_factory(metadata={'experiment_name': long_name}),
password=FACTORY_PASSWORD,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
row = await _assert_rejected(
handle=handle,
import_run_id=import_run_id,
object_key=object_key,
step=ImportStep.STRUCTURE_VALIDATION,
import_read_status_row=import_read_status_row,
import_assert_nothing_provisioned=import_assert_nothing_provisioned,
import_names=import_names,
experiment_name=long_name,
)
assert row['experiment_name'] == 'import-e2e-placeholder', (
'the over-long name reached the row, so the gate did not hold the column'
)
with import_bff_engine.begin() as connection:
width = connection.execute(
text(
'SELECT character_maximum_length FROM information_schema.columns '
"WHERE table_name = 'experiment_run' AND column_name = 'experiment_name'"
)
).scalar_one()
assert width == 50, 'the column this gate exists to protect is no longer VARCHAR(50)'
class TestGate7ContentPolicy:
"""A file kind an MLflow artifact tree has no reason to carry."""
async def test_a_forbidden_file_type_is_rejected_at_content_policy(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_assert_nothing_provisioned: Any,
) -> None:
handle, import_run_id, object_key = await _run_rejected_import(
payload=import_bundle_factory(extra_files={'artifacts/payload.exe': b'MZ\x00\x00'}),
password=FACTORY_PASSWORD,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
await _assert_rejected(
handle=handle,
import_run_id=import_run_id,
object_key=object_key,
step=ImportStep.CONTENT_POLICY,
import_read_status_row=import_read_status_row,
import_assert_nothing_provisioned=import_assert_nothing_provisioned,
import_names=import_names,
)
class TestTheRejectionSaysNothingTechnical:
"""`error_message` is a sentence for a person: no path, no object key, no traceback.
One case is enough to make the point structurally — the catalog is keyed by step and
`classify_import_failure` discards the exception's text entirely — and the per-gate cases above
each assert the sentence is *exactly* the catalog's, which is the same guarantee stated
positively.
"""
async def test_no_technical_detail_reaches_the_recorded_reason(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
) -> None:
handle, import_run_id, object_key = await _run_rejected_import(
payload=import_bundle_factory(include_metadata=False),
password=FACTORY_PASSWORD,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
with pytest.raises(WorkflowFailureError) as failure:
await handle.result()
row = import_read_status_row(import_run_id)
assert row is not None
message = row['error_message'] or ''
for forbidden in (
object_key,
IMPORT_BUCKET,
IMPORT_PREFIX,
'experiment_run',
'Traceback',
'BundleGateError',
tempfile.gettempdir(),
'gate',
):
assert forbidden not in message, f'{forbidden!r} leaked into error_message: {message!r}'
# The technical truth is not lost — it is on the Temporal error, where support can read it.
assert 'metadata' in failure_text(failure.value).lower()

314
e2e/test_import_model_happy_path.py Normal file
View File

@@ -0,0 +1,314 @@
"""An import that works, twice over: once from the producer's own bytes and once from the factory.
Two origins, because they prove different things. The **golden** bundle is bytes the producer's code
emitted — it decrypts at the producer's real Argon2id cost (256 MiB, 3 passes) and carries the tree
the producer really writes, so it is the only case that would notice the producer's layout moving.
The **factory** bundle is synthesised here, cheaply, with names unique to the test; it is what the
rest of the suite runs on, so a happy path over it is what makes the negative cases' "and nothing
was provisioned" meaningful.
What "it worked" means, in full: the experiment exists, a run exists inside it carrying the derived
run name, the artifact tree is under the run with `prediction_model` and `data_model` at its root,
every parameter the bundle declared is logged plus the origin's identity under `source.*`, a
registry version exists, the MongoDB document exists, and the status row ends `IMPORT_SUCCESS`.
Two things a successful import must **not** have done are asserted here too, because they are the
difference between "imported" and "live": no version is in `Production`, and the document is
`active: false`. Promotion and activation are manual operator steps (QTZPOC-21).
"""
from __future__ import annotations
import json
from typing import Any
import mlflow
import pytest
from e2e.import_harness import (
IMPORT_BUCKET,
IMPORT_PREFIX,
SEEDED_MODEL_IDS,
)
pytestmark = [pytest.mark.import_e2e, pytest.mark.asyncio]
GOLDEN_MODEL_NAME = 'fixture_model'
GOLDEN_EXPERIMENT_NAME = 'fixture_experiment'
GOLDEN_MODEL_VERSION = '3'
GOLDEN_TARGET = 'SE'
REQUIRED_ARTIFACT_ROOTS = {'prediction_model', 'data_model'}
def _mlflow_client(server: dict[str, Any]) -> Any:
return mlflow.tracking.MlflowClient(tracking_uri=server['tracking_uri'])
def _as_dict(value: Any) -> dict[str, Any]:
"""`request_data` / `orchestrator_response_data` come back as a dict or as JSON text."""
return json.loads(value) if isinstance(value, str) else (value or {})
async def _run_import(
*,
payload: bytes,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
password: str,
import_envelope: Any,
) -> tuple[Any, int, str]:
"""Upload, insert the row the BFF would have inserted, run the workflow. Return the outcome."""
object_key, digest = import_upload_bundle(payload)
import_run_id = import_insert_status_row(object_key=object_key, expected_digest=digest)
handle = await import_execute(
import_build_input(
import_run_id=import_run_id,
object_key=object_key,
expected_digest=digest,
password_envelope=import_envelope(password),
)
)
return handle, import_run_id, object_key
def _assert_provisioned(
*,
client: Any,
experiment_name: str,
model_name: str,
run_name: str,
expected_parameters: dict[str, str],
expected_source_fields: dict[str, str],
) -> tuple[str, Any]:
"""The whole positive claim about MLflow, in one place. Returns `(run_id, version)`."""
experiment = client.get_experiment_by_name(experiment_name)
assert experiment is not None, f'no experiment named {experiment_name!r}'
runs = [
run
for run in client.search_runs([experiment.experiment_id])
if run.info.run_name == run_name
]
assert len(runs) == 1, f'expected exactly one run named {run_name!r}, found {len(runs)}'
run = runs[0]
assert run.info.status == 'FINISHED'
artifact_roots = {artifact.path for artifact in client.list_artifacts(run.info.run_id)}
assert REQUIRED_ARTIFACT_ROOTS <= artifact_roots, (
f'the run holds {sorted(artifact_roots)}, missing {sorted(REQUIRED_ARTIFACT_ROOTS - artifact_roots)}'
)
# `prediction_model` is a directory the legacy loader walks, not an opaque blob.
prediction_files = {
artifact.path.rsplit('/', 1)[-1]
for artifact in client.list_artifacts(run.info.run_id, 'prediction_model')
}
assert {'MLmodel', 'model.pkl'} <= prediction_files, prediction_files
for name, value in expected_parameters.items():
assert run.data.params.get(name) == value, (
f'parameter {name!r} is {run.data.params.get(name)!r}, expected {value!r}'
)
for name, value in expected_source_fields.items():
assert run.data.params.get(f'source.{name}') == value, (
f'source.{name} is {run.data.params.get(f"source.{name}")!r}, expected {value!r}'
)
versions = list(client.search_model_versions(f"name='{model_name}'"))
assert len(versions) == 1, f'expected one registered version, found {len(versions)}'
version = versions[0]
assert version.run_id == run.info.run_id
return run.info.run_id, version
class TestGoldenBundle:
"""The producer's own bytes, at the producer's own KDF cost."""
async def test_the_producer_artefact_imports_end_to_end(
self,
import_status_table: Any,
import_models_collection: Any,
import_golden_bundle: dict[str, Any],
import_mlflow_server: dict[str, Any],
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
) -> None:
handle, import_run_id, object_key = await _run_import(
payload=import_golden_bundle['bytes'],
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
password=import_golden_bundle['password'],
import_envelope=import_envelope,
)
result = await handle.result()
assert result['model_name'] == GOLDEN_MODEL_NAME
assert result['import_run_id'] == import_run_id
assert result['version'] == '1', 'the destination numbering starts at 1, not at the bundle'
run_id, version = _assert_provisioned(
client=_mlflow_client(import_mlflow_server),
experiment_name=GOLDEN_EXPERIMENT_NAME,
model_name=GOLDEN_MODEL_NAME,
run_name=f'import-{GOLDEN_MODEL_NAME}-v{GOLDEN_MODEL_VERSION}',
expected_parameters={'target_variable': GOLDEN_TARGET, 'nan_treatment': 'drop'},
expected_source_fields={
'model_name': GOLDEN_MODEL_NAME,
'experiment_name': GOLDEN_EXPERIMENT_NAME,
'model_version': GOLDEN_MODEL_VERSION,
'model_project': 'fixture_project',
},
)
assert result['run_id'] == run_id
document = import_models_collection.find_one({'name': GOLDEN_MODEL_NAME}, {'_id': False})
assert document is not None, 'no model document was written'
assert document['id'] == str(max(int(i) for i in SEEDED_MODEL_IDS) + 1)
assert document['model_config']['target'] == GOLDEN_TARGET
row = import_read_status_row(import_run_id)
assert row is not None
assert row['status'] == 'IMPORT_SUCCESS'
assert row['error_message'] is None
assert row['experiment_name'] == GOLDEN_EXPERIMENT_NAME
assert row['run_name'] == f'import-{GOLDEN_MODEL_NAME}-v{GOLDEN_MODEL_VERSION}'
assert row['file_name'] == object_key
assert row['bucket_name'] == IMPORT_BUCKET
assert object_key.startswith(IMPORT_PREFIX)
# Imported, not live.
assert version.current_stage in (None, 'None'), version.current_stage
assert document['active'] is False
class TestFactoryBundle:
"""A synthesised tree with names unique to this test — the shape the rest of the suite runs on."""
async def test_a_factory_bundle_imports_end_to_end(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_mlflow_server: dict[str, Any],
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
) -> None:
payload = import_bundle_factory(parameters={'target_variable': 'SE'})
handle, import_run_id, _ = await _run_import(
payload=payload,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
password='Factory1Model!Test',
import_envelope=import_envelope,
)
result = await handle.result()
assert result['model_name'] == import_names['model_name']
run_id, version = _assert_provisioned(
client=_mlflow_client(import_mlflow_server),
experiment_name=import_names['experiment_name'],
model_name=import_names['model_name'],
# DEFAULT_METADATA declares model_version '2'.
run_name=f'import-{import_names["model_name"]}-v2',
expected_parameters={'target_variable': 'SE', 'retrain': 'False'},
expected_source_fields={
'model_name': import_names['model_name'],
'experiment_name': import_names['experiment_name'],
'model_version': '2',
},
)
assert result['run_id'] == run_id
assert result['version'] == '1'
document = import_models_collection.find_one(
{'name': import_names['model_name']}, {'_id': False}
)
assert document is not None
assert document['model_config'] == {
'transform_flavor': 'sklearn',
'predict_flavor': 'joblib',
'retention_minutes': 60,
'target': 'SE',
}
row = import_read_status_row(import_run_id)
assert row is not None
assert row['status'] == 'IMPORT_SUCCESS'
detail = _as_dict(row['orchestrator_response_data'])
assert detail['import']['workflow_id'] == handle.id
assert version.current_stage in (None, 'None')
assert document['active'] is False
class TestLegacyLoaderResolution:
"""The registry source an import writes has to be one the *existing* loader can parse.
This is the assertion that could only ever be made against a real registry: the importer writes
the run's resolved artifact URI with `prediction_model` appended, because `get_model_run_id`
extracts the run id as `source.split('/')[2]` — an index that is the run id only in the resolved
form. A literal `runs:/<run_id>/prediction_model` would put `prediction_model` at index 2, and a
`models:/m-*` source would have none of the parts at all.
It is called with the stage the version is actually in rather than `Production`, because an
import deliberately promotes nothing — passing `Production` here would test the promotion step
this feature does not have.
"""
async def test_the_imported_source_resolves_to_the_import_run(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_activities: Any,
import_mlflow_server: dict[str, Any],
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
) -> None:
payload = import_bundle_factory()
handle, _, _ = await _run_import(
payload=payload,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
password='Factory1Model!Test',
import_envelope=import_envelope,
)
result = await handle.result()
model_name = import_names['model_name']
client = _mlflow_client(import_mlflow_server)
version = next(iter(client.search_model_versions(f"name='{model_name}'")))
assert version.source.endswith('/prediction_model'), version.source
# The production method, on the production repository object, against the real registry.
resolved = import_activities.import_mlflow_repository.get_model_run_id(
model_name, stage=version.current_stage
)
assert resolved == result['run_id'], (
f'get_model_run_id resolved {resolved!r} from source {version.source!r}, '
f'but the import created run {result["run_id"]!r}'
)
artifact_roots = {artifact.path for artifact in client.list_artifacts(resolved)}
assert REQUIRED_ARTIFACT_ROOTS <= artifact_roots, sorted(artifact_roots)

364
e2e/test_import_model_provisioning_failures.py Normal file
View File

@@ -0,0 +1,364 @@
"""Each provisioning step fails as itself: its own code, its own sentence, nothing half-built.
The four steps after the gates — experiment creation, artifact upload, registration, the model
document — each get one failure, and every one lands `IMPORT_ERROR` rather than
`ORCHESTRATOR_VALIDATION_ERROR`. That split is not cosmetic: it is the table's existing distinction
between "the person who submitted this can fix it" and "only support can", and a provisioning
failure is always the second kind.
**How each failure is caused**, in design E7's order of preference:
| Step | Cause | Preference |
|---|---|---|
| `experiment_creation` | the MLflow server is genuinely stopped | broken dependency |
| `artifact_upload` | targeted patch on `log_artifacts` | patch, justified in place |
| `registration` | targeted patch on `create_model_version` | patch, justified in place |
| `model_document` | a document already holding the id the activity will compute, against the real unique index | broken dependency |
Two patches rather than the one design E7 anticipated. The reason is the same for both and is
recorded in each test: with one server serving both tracking and registry, and MLflow validating
nothing about a registered model name except that it is non-empty (which gate 6 already requires),
there is no way to make the world hostile to *just* one of these two steps. Stopping the server
fails experiment creation first; a malformed name is not malformed to MLflow. The patch is on the
client call the step makes, not on the activity or the workflow, so what is being tested is still
"the workflow attributes this step's failure to this step".
"""
from __future__ import annotations
import json
import tempfile
from typing import Any
import pytest
from temporalio.client import WorkflowFailureError
from e2e.import_harness import IMPORT_BUCKET, IMPORT_PREFIX, SEEDED_MODEL_IDS
from laborious.activities.model_import_errors import IMPORT_FAILURE_CATALOG
from laborious.utils.bundle.steps import ImportStep
pytestmark = [pytest.mark.import_e2e, pytest.mark.asyncio]
PLATFORM_ERROR_STATUS = 'IMPORT_ERROR'
FACTORY_PASSWORD = 'Factory1Model!Test'
def _as_dict(value: Any) -> dict[str, Any]:
return json.loads(value) if isinstance(value, str) else (value or {})
def _assert_platform_failure(
*,
row: dict[str, Any] | None,
step: ImportStep,
workflow_id: str,
object_key: str,
) -> None:
"""The shared claim of task 7.5, asserted by every case in this module.
`error_message` is the catalog sentence *exactly* — not "contains" — which is the strongest
possible form of "carries no path, table name, object key or traceback": there is nothing in it
that did not come from the catalog.
"""
assert row is not None, 'the failure lost the row it was recorded on'
assert row['status'] == PLATFORM_ERROR_STATUS, (
f"status is {row['status']!r}; a provisioning failure is not the user's to fix"
)
code, reason = IMPORT_FAILURE_CATALOG[step]
detail = _as_dict(row['orchestrator_response_data'])['import']
assert detail['step'] == step.value, detail
assert detail['code'] == code.value, detail
assert detail['workflow_id'] == workflow_id, detail
assert row['error_message'] == reason
message = row['error_message'] or ''
for forbidden in (
object_key,
IMPORT_BUCKET,
IMPORT_PREFIX,
'experiment_run',
'Traceback',
'Error',
'mlflow',
'Mongo',
tempfile.gettempdir(),
):
assert forbidden not in message, f'{forbidden!r} leaked into error_message: {message!r}'
async def _upload_and_start(
*,
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
**factory_kwargs: Any,
) -> tuple[Any, int, str]:
object_key, digest = import_upload_bundle(import_bundle_factory(**factory_kwargs))
import_run_id = import_insert_status_row(object_key=object_key, expected_digest=digest)
handle = await import_execute(
import_build_input(
import_run_id=import_run_id,
object_key=object_key,
expected_digest=digest,
password_envelope=import_envelope(FACTORY_PASSWORD),
)
)
return handle, import_run_id, object_key
class TestExperimentCreation:
"""The tracking server is not there. No patch, no mock — the process is stopped."""
async def test_a_stopped_tracking_server_fails_at_experiment_creation(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
import_mlflow_stopped: Any,
import_assert_nothing_provisioned: Any,
) -> None:
with import_mlflow_stopped():
handle, import_run_id, object_key = await _upload_and_start(
import_bundle_factory=import_bundle_factory,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
with pytest.raises(WorkflowFailureError):
await handle.result()
row = import_read_status_row(import_run_id)
_assert_platform_failure(
row=row,
step=ImportStep.EXPERIMENT_CREATION,
workflow_id=handle.id,
object_key=object_key,
)
# The bundle *was* opened before MLflow was reached, so the names are on the row — which is
# what task 9.3 asks for, observed from a failure that stopped exactly there.
assert row is not None
assert row['experiment_name'] == import_names['experiment_name']
assert row['run_name'] == f'import-{import_names["model_name"]}-v2'
# Nothing in Mongo: the document is the last step and never ran.
documents = list(import_models_collection.find({}, {'_id': False}))
assert {str(d['id']) for d in documents} == set(SEEDED_MODEL_IDS)
import_assert_nothing_provisioned(
experiment_name=import_names['experiment_name'],
model_name=import_names['model_name'],
object_key=object_key,
)
class TestArtifactUpload:
"""The step design E7 names as the one a patch is needed for."""
async def test_a_failing_artifact_upload_is_recorded_as_its_own_step(
self,
monkeypatch: pytest.MonkeyPatch,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_activities: Any,
import_mlflow_server: dict[str, Any],
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
) -> None:
"""Why a patch: with the server up and the bundle valid, nothing else fails only this step.
Hostile input cannot reach here — a bundle that gets past gate 7 has a readable tree, and
one that does not never reaches artifact upload. A broken dependency cannot isolate it
either: the only dependency this step has is the tracking server, and stopping that fails
experiment creation one step earlier. So the patch is on the single client call the step
makes, `log_artifacts`; everything around it — the run creation, the parameters, the
workflow's attribution of the failure — is the real code.
"""
client = import_activities.import_mlflow_repository.client
def refuse(*_args: Any, **_kwargs: Any) -> None:
raise OSError('artifact store is unreachable')
monkeypatch.setattr(client, 'log_artifacts', refuse)
handle, import_run_id, object_key = await _upload_and_start(
import_bundle_factory=import_bundle_factory,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
with pytest.raises(WorkflowFailureError):
await handle.result()
_assert_platform_failure(
row=import_read_status_row(import_run_id),
step=ImportStep.ARTIFACT_UPLOAD,
workflow_id=handle.id,
object_key=object_key,
)
# The experiment exists — it was the previous step and it succeeded. Nothing after does.
mlflow_client = _fresh_client(import_mlflow_server)
assert mlflow_client.get_experiment_by_name(import_names['experiment_name']) is not None
assert not list(mlflow_client.search_model_versions(f"name='{import_names['model_name']}'"))
assert import_models_collection.find_one({'name': import_names['model_name']}) is None
class TestRegistration:
"""The artifacts are stored and the model still cannot be registered."""
async def test_a_failing_registration_is_recorded_as_its_own_step(
self,
monkeypatch: pytest.MonkeyPatch,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_activities: Any,
import_mlflow_server: dict[str, Any],
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
) -> None:
"""Why a patch here too, contrary to design E7's expectation that this step needs none.
Hostile input was the first thing tried: a registered model name MLflow refuses. It does not
exist — `_validate_model_name` rejects only the empty string, which gate 6 already refuses,
and the sqlite backend store does not enforce the `VARCHAR(256)` the schema declares, so
even a 300-character name is accepted. A broken dependency was the second: one process
serves both tracking and the registry, so stopping it fails experiment creation three steps
earlier. The patch is on `create_model_version`, the one call this step is.
"""
client = import_activities.import_mlflow_repository.client
def refuse(*_args: Any, **_kwargs: Any) -> None:
raise RuntimeError('registry write rejected')
monkeypatch.setattr(client, 'create_model_version', refuse)
handle, import_run_id, object_key = await _upload_and_start(
import_bundle_factory=import_bundle_factory,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
with pytest.raises(WorkflowFailureError):
await handle.result()
_assert_platform_failure(
row=import_read_status_row(import_run_id),
step=ImportStep.REGISTRATION,
workflow_id=handle.id,
object_key=object_key,
)
# The run and its artifacts survive — "the model's files were stored, but the model itself
# could not be registered" is exactly what the catalog sentence for this step says.
mlflow_client = _fresh_client(import_mlflow_server)
experiment = mlflow_client.get_experiment_by_name(import_names['experiment_name'])
assert experiment is not None
runs = mlflow_client.search_runs([experiment.experiment_id])
assert len(runs) == 1
assert {a.path for a in mlflow_client.list_artifacts(runs[0].info.run_id)} >= {
'prediction_model',
'data_model',
}
assert import_models_collection.find_one({'name': import_names['model_name']}) is None
class TestModelDocument:
"""The last step, failed by a real unique index on a real collection."""
async def test_a_duplicate_id_fails_at_model_document(
self,
import_status_table: Any,
import_models_collection: Any,
import_names: dict[str, str],
import_bundle_factory: Any,
import_mlflow_server: dict[str, Any],
import_upload_bundle: Any,
import_insert_status_row: Any,
import_build_input: Any,
import_execute: Any,
import_envelope: Any,
import_read_status_row: Any,
) -> None:
"""A document already holding the id `max(int(id)) + 1` will compute, under another name.
Under another name on purpose: a document with the *same* name would make the activity skip
the insert entirely, which is a different behaviour (task 8.6) and would not fail. This one
collides only on `id`, so the insert is attempted and the index refuses it.
No patch and no mock: the failure is `pymongo`'s duplicate-key error from a real index,
which is the thing that would happen in production if two imports raced.
"""
collision_id = str(max(int(seeded) for seeded in SEEDED_MODEL_IDS) + 1)
import_models_collection.insert_one(
{'id': collision_id, 'name': 'a_different_model', 'active': True, 'model_config': {}}
)
handle, import_run_id, object_key = await _upload_and_start(
import_bundle_factory=import_bundle_factory,
import_upload_bundle=import_upload_bundle,
import_insert_status_row=import_insert_status_row,
import_build_input=import_build_input,
import_execute=import_execute,
import_envelope=import_envelope,
)
with pytest.raises(WorkflowFailureError):
await handle.result()
_assert_platform_failure(
row=import_read_status_row(import_run_id),
step=ImportStep.MODEL_DOCUMENT,
workflow_id=handle.id,
object_key=object_key,
)
# No document for this model, and the colliding one is untouched.
assert import_models_collection.find_one({'name': import_names['model_name']}) is None
blocker = import_models_collection.find_one({'id': collision_id}, {'_id': False})
assert blocker == {
'id': collision_id,
'name': 'a_different_model',
'active': True,
'model_config': {},
}
# Everything before it stands: "the model was registered, but the settings ... could not be
# saved" is the catalog sentence, and this is what makes it true rather than merely stored.
mlflow_client = _fresh_client(import_mlflow_server)
versions = list(mlflow_client.search_model_versions(f"name='{import_names['model_name']}'"))
assert len(versions) == 1
def _fresh_client(server: dict[str, Any]) -> Any:
"""An MlflowClient bound to the test server, independent of the one the activities hold."""
import mlflow
return mlflow.tracking.MlflowClient(tracking_uri=server['tracking_uri'])

13
encode.sh
View File

@@ -1,13 +0,0 @@
source ./venv/bin/activate
pip install pathspec
pip install pyyaml
echo "
.git" >> .gitignore
python encrypt.py ./ code --ignore .gitignore --chunk-size 100000
sed -i '/.git/d' .gitignore
xdg-open .

7
laborious/activities/activities.py
View File

@@ -181,8 +181,11 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API, ModelImport):
import_config=import_config or build_import_config(),
minio_repository=minio_repository,
# The MLflow repository is shared rather than rebuilt: setting the tracking URI is a
# process-wide side effect, so two instances would be two chances to disagree.
mlflow_repository=self.model_monitoring_repository,
# process-wide side effect, so two instances would be two chances to disagree. Read
# through `getattr` because `MLFlow.__init__` is what creates it — if that ever stops
# running before this line, the import activities refuse with "MLflow repository not
# initialized" at the point of use rather than silently provisioning nothing.
mlflow_repository=getattr(self, 'model_monitoring_repository', None),
mongo_repository=mongo_repository,
logger=logger,
notification_handler=notification_handler,

13
laborious/activities/model_import.py
View File

@@ -853,8 +853,15 @@ class ModelImport(SientiaMonitoring):
async def register_import_model_version(self, input_data: dict[str, Any]) -> dict[str, Any]:
"""Register a version whose source the legacy loader can parse. No stage change.
The source is `runs:/<run_id>/prediction_model`, because `get_model_run_id` extracts the run
id as `source.split('/')[2]` — a `models:/m-*` source, what the new stack emits, breaks it.
The source is the run's **resolved artifact URI** with `prediction_model` appended —
`mlflow-artifacts:/<experiment_id>/<run_id>/artifacts/prediction_model` on this deployment —
because `get_model_run_id` extracts the run id as `source.split('/')[2]`. That index is the
run id only in the resolved form: a literal `runs:/<run_id>/prediction_model` has just two
slashes, so index 2 is the string `prediction_model`, and a `models:/m-*` source (what the
new stack emits) has none of the parts at all. This is the same value the training flow
stores, because `mlflow.register_model` resolves the `runs:/` URI before writing it
(`model_repository.py`, `register_model`) — `create_model_version` does not.
The stage is left untouched: a `COMPLETED` import is a registered model, not a servable one,
and promoting it to `Production` is a manual operator step outside this workflow.
@@ -880,7 +887,7 @@ class ModelImport(SientiaMonitoring):
# lands as a new version of the same registered model, never a differently named one.
client.create_registered_model(model_name)
source = f'runs:/{run_id}/prediction_model'
source = f'{str(client.get_run(run_id).info.artifact_uri).rstrip("/")}/prediction_model'
version = client.create_model_version(name=model_name, source=source, run_id=run_id)
self.info(
f'import registered {model_name} version {version.version}',

15
laborious/worker/worker.py
View File

@@ -77,6 +77,19 @@ with workflow.unsafe.imports_passed_through():
POD_ID = os.getenv('HOSTNAME')
SDK_METRICS_PORT = int(os.getenv('HTTP_SDK_METRICS_PORT', '9091'))
# The values that turn the import worker on. Anything else — unset, empty, `0`, `false`, a typo —
# leaves the four default workers alone and polls no import queue at all.
IMPORT_ENABLED_VALUES = ('1', 'true', 'yes', 'on')
def import_worker_enabled() -> bool:
"""Whether this runtime registers the fifth (import) worker.
Read at worker start, not per workflow: containment (QTZPOC-19) wants the importer in its own
runtime with its own credentials, and this flag is what makes that a deployment decision.
"""
return os.getenv('IMPORT_MODEL_ENABLED', '').strip().lower() in IMPORT_ENABLED_VALUES
async def main():
"""
@@ -250,7 +263,7 @@ async def main():
# its own credentials, and a flag lets that be a deployment decision instead of forcing every
# Laborious runtime to poll the import queue today. Unset, the four workers above are unaffected
# and no import queue is polled at all.
if os.getenv('IMPORT_MODEL_ENABLED', '').strip().lower() in ('1', 'true', 'yes', 'on'):
if import_worker_enabled():
logger.custom_info('Import model worker enabled', metadata_runtime)
workers.append(
prepare_worker(

18
laborious/workflows/import_model.py
View File

@@ -356,7 +356,9 @@ class ImportModel:
start_to_close_timeout=STATUS_WRITE_TIMEOUT,
schedule_to_close_timeout=STATUS_WRITE_ENVELOPE,
)
except BaseException as error: # NOSONAR - the outcome stands even when it cannot be written
except (
BaseException
) as error: # NOSONAR - the outcome stands even when it cannot be written
await self._report_status_write_failure(
base,
error,
@@ -377,7 +379,9 @@ class ImportModel:
retry_policy=retry_policy,
start_to_close_timeout=STATUS_WRITE_TIMEOUT,
)
except BaseException: # NOSONAR - reporting the report is where this stops
except BaseException: # noqa: S110 # NOSONAR - the report of a report is where this stops
# Deliberately silent: the channel this was trying to reach is the thing that logs, so
# there is nowhere left to say it, and an import that worked must not fail over it.
pass
# ----------------------------------------------------------- side-effect steps
@@ -389,8 +393,8 @@ class ImportModel:
The rule is applied by the importer rather than assumed from a Helm value, so the acceptance
criterion is verifiable from the code that depends on it. But a bucket policy the importer
cannot set is an operational problem, not a bad import: failing a valid model import because
a lifecycle API refused would be the wrong trade, so this is reported and dropped like a
lost status hint.
a lifecycle API refused would be the wrong trade, so the failure is dropped here and left
where Temporal already records it — the activity's own failure in the event history.
"""
try:
await workflow.execute_activity_method(
@@ -399,7 +403,11 @@ class ImportModel:
retry_policy=retry_policy,
start_to_close_timeout=RETENTION_TIMEOUT,
)
except BaseException: # NOSONAR - retention is not a precondition of a correct import
except BaseException: # noqa: S110 # NOSONAR - retention is not a precondition of an import
# Dropped, not reported: the activity's own failure is already in the event history,
# and a bucket policy the importer cannot set is an operational problem rather than a
# bad import. Failing a valid model import over a lifecycle API would be the wrong
# trade.
pass
@staticmethod

9
pyproject.toml
View File

@@ -65,6 +65,11 @@ ignore = [
# must be in sys.modules before Activities is imported, so they cannot move up.
"E402",
]
"scripts/**/*.py" = [
# Notebook-style scripts split into `# %%` cells: each cell imports what it needs
# where it needs it, so imports legitimately appear below other statements.
"E402",
]
[tool.ruff.lint.mccabe]
max-complexity = 15
@@ -133,6 +138,10 @@ markers = [
"integration: marks tests as integration tests",
"unit: marks tests as unit tests",
"opc: marks tests that use the in-process OPC UA server (OpcRepository E2E)",
# The .sientia import e2e suite. Outside `testpaths`, so `validate --only-tests` and the
# coverage gate never collect it; run it on request with `pytest e2e -m import_e2e`.
"import_e2e: marks the model-import end-to-end tests (needs Docker and an MLflow server)",
"slow: marks tests that run on a real clock and take tens of seconds",
]
[tool.coverage.run]

3
requirements-dev.txt
View File

@@ -13,9 +13,10 @@ types-requests>=2.31.0 # Type stubs for requests
pytest>=7.4.0 # Testing framework
pytest-cov>=4.1.0 # Coverage plugin for pytest
pytest-asyncio>=0.21.0 # Async test support (already in main requirements)
testcontainers[postgres,minio] # PostgreSQL and MinIO containers for E2E tests
testcontainers[postgres,minio,mongodb] # PostgreSQL, MinIO and MongoDB containers for E2E tests
# Development Tools
ipython>=8.12.0 # Enhanced Python shell
ipdb>=0.13.13 # IPython debugger
ipykernel==6.30.1 # IPython kernel for Jupyter notebooks
python-dotenv>=1.0.0 # .env loading for scripts/import_model_manual_run.py

2
requirements-local.txt
View File

@@ -4,7 +4,7 @@ sqlalchemy
asyncua==1.0.6
redis
git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.12.1
/home/grezewave/Documents/projects/sientia/sientia-mlops-library
git+ssh://git@github.com/Aignosi/sientia-mlops-library.git@0.41.1
prometheus-client
# fsspec/s3fs/gcsfs pinned as a set: nannyml (via the mlops library) pulls both s3fs and
# gcsfs, and each pins fsspec exactly, so a free `s3fs` makes the resolver fail against

2
requirements.txt
View File

@@ -4,7 +4,7 @@ sqlalchemy
asyncua==1.0.6
redis
sientia_do>=1.12.1
sientia>=0.41.0
sientia>=0.41.1
prometheus-client
# fsspec/s3fs/gcsfs pinned as a set: nannyml (via sientia) pulls both s3fs and gcsfs, and
# each pins fsspec exactly, so a free `s3fs` makes the resolver fail against evidently's

20
scripts/encode.sh Executable file
View File

@@ -0,0 +1,20 @@
#!/bin/bash
# Resolve paths from the script's own location so this works whether it is run
# from the repository root or from inside scripts/.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
source "${REPO_ROOT}/venv/bin/activate"
pip install pathspec
pip install pyyaml
echo "
.git" >> "${REPO_ROOT}/.gitignore"
python "${SCRIPT_DIR}/encrypt.py" "${REPO_ROOT}" "${REPO_ROOT}/code" --ignore "${REPO_ROOT}/.gitignore" --chunk-size 100000
sed -i '/.git/d' "${REPO_ROOT}/.gitignore"
xdg-open "${REPO_ROOT}"

62
encrypt.py → scripts/encrypt.py
View File

@@ -1,22 +1,23 @@
import os
import argparse
from pathspec import PathSpec
import yaml # type: ignore
import os
from typing import Any
'''
import yaml # type: ignore
from pathspec import PathSpec
r"""
Usage:
python .\encrypt.py path_to_dir output_file --ignore ignore_file --chunk-size 100000
'''
"""
def load_ignore_patterns(ignore_file, include_library):
# Ensure the .gitignore file exists
if not os.path.exists(ignore_file):
raise FileNotFoundError(f"Ignore file not found at {ignore_file}")
raise FileNotFoundError(f'Ignore file not found at {ignore_file}')
# Load and parse the .gitignore patterns
with open(ignore_file, 'r') as file:
with open(ignore_file) as file:
patterns = file.readlines()
if not include_library:
patterns.append('**/deploy/library/')
@@ -32,14 +33,12 @@ def is_ignored(file_path, spec):
def encode_file_tree_to_yaml(directory, ignore_file, include_library):
"""Encode the file tree into a single YAML file."""
ignore_patterns = load_ignore_patterns(
ignore_file, include_library) if ignore_file else None
ignore_patterns = load_ignore_patterns(ignore_file, include_library) if ignore_file else None
file_tree: dict[str, Any] = {}
for root, dirs, files in os.walk(directory):
# Skip ignored directories
dirs[:] = [d for d in dirs if not is_ignored(
os.path.join(root, d), ignore_patterns)]
dirs[:] = [d for d in dirs if not is_ignored(os.path.join(root, d), ignore_patterns)]
for file in files:
file_path = os.path.join(root, file)
@@ -50,10 +49,10 @@ def encode_file_tree_to_yaml(directory, ignore_file, include_library):
# Read file content
try:
with open(file_path, 'r', encoding='utf-8') as f:
with open(file_path, encoding='utf-8') as f:
content = f.read()
except Exception as e:
print(f"Error reading file {file_path}: {e}")
print(f'Error reading file {file_path}: {e}')
raise
# Create nested dictionary structure
@@ -72,27 +71,30 @@ def encode_file_tree_to_yaml(directory, ignore_file, include_library):
def chunk_and_write_file_tree_to_yaml(yaml_content, output_file, chunk_size=None):
"""Chunk the YAML content and write it to the output file."""
chunks = [yaml_content] if chunk_size is None else [
yaml_content[i:i + chunk_size] for i in range(0, len(yaml_content), chunk_size)]
chunks = (
[yaml_content]
if chunk_size is None
else [yaml_content[i : i + chunk_size] for i in range(0, len(yaml_content), chunk_size)]
)
for i, chunk in enumerate(chunks):
chunk_file = f"{output_file}_{i}.yaml"
chunk_file = f'{output_file}_{i}.yaml'
# Write the file tree to the output YAML file
with open(chunk_file, 'w', encoding='utf-8') as yaml_file:
yaml_file.write(chunk)
def main():
parser = argparse.ArgumentParser(
description="Encrypts file tree to yaml file")
parser.add_argument("input_directory", help="Directory to encode")
parser.add_argument("output_yaml_file", help="Output YAML file")
parser.add_argument("--ignore", default=None,
help="Path to the ignore file")
parser.add_argument("--chunk-size", type=int, default=None,
help="Chunk size for the output YAML file")
parser.add_argument("--library", type=bool, default=False,
help="Incude the library in the output YAML file")
parser = argparse.ArgumentParser(description='Encrypts file tree to yaml file')
parser.add_argument('input_directory', help='Directory to encode')
parser.add_argument('output_yaml_file', help='Output YAML file')
parser.add_argument('--ignore', default=None, help='Path to the ignore file')
parser.add_argument(
'--chunk-size', type=int, default=None, help='Chunk size for the output YAML file'
)
parser.add_argument(
'--library', type=bool, default=False, help='Incude the library in the output YAML file'
)
# Parse arguments
args = parser.parse_args()
@@ -103,11 +105,9 @@ def main():
output_yaml_file = args.output_yaml_file
include_library = args.library
content = encode_file_tree_to_yaml(
directory_to_encode, ignore_file_path, include_library)
chunk_and_write_file_tree_to_yaml(
content, output_yaml_file, args.chunk_size)
content = encode_file_tree_to_yaml(directory_to_encode, ignore_file_path, include_library)
chunk_and_write_file_tree_to_yaml(content, output_yaml_file, args.chunk_size)
if __name__ == "__main__":
if __name__ == '__main__':
main()

497
scripts/import_model_manual_run.py Normal file
View File

@@ -0,0 +1,497 @@
"""Run one `.sientia` model import by hand, standing in for the frontend that does not exist yet.
**What this is.** The Angular screen plus the Spring Boot BFF will, between them, pick a `.sientia`
file, hash it, wrap the bundle password for the worker, upload the object and insert one `PENDING`
row into `public.experiment_run` before starting `import_model`. None of that exists yet. This
script does exactly those four things, against whatever environment `.env` points at, so the import
path can be exercised for real and so whoever writes the frontend has a working reference for the
wire shapes (the digest, the envelope, the object key, the row).
**What it writes.** Two things, and it can undo both: one object under
`{IMPORT_BUNDLE_BUCKET}/{IMPORT_BUNDLE_PREFIX}` in MinIO, and one row in `public.experiment_run` in
the BFF Postgres database. Everything else it does is a read. If the workflow runs, *it* creates the
MLflow experiment, run and registered version and the MongoDB model document — cell 10 can remove
those too, behind its own flag.
**The safety rule.** Every target comes from the environment, and this file contains no host, user,
password or key literal. A `.env` inherited from a cluster deployment can perfectly well name a
production database and a production bucket, so cell 5 and cell 6 — the only cells that write — are
gated on cell 2, where the operator has to restate the Postgres host that cell 1 printed. This is a
development tool. It is not for production, and the guard is a mitigation, not a guarantee.
**Cells.** Run them one at a time in the VS Code interactive window, or run the module top to
bottom. Cell order is the contract:
1 resolve every target and print it 6 insert the PENDING row -> import_run_id
2 confirmation guard (first write is after) 7 print the workflow input
3 pick the bundle, compute expected_digest 8 optionally start the workflow and wait
4 build the password envelope 9 read the outcome back
5 upload the object, verify the stored bytes 10 undo
**Boundaries.** This script imports nothing from `laborious.activities` or `laborious.workflows` —
it stands in for the frontend, so it must not borrow the consumer's code. The one exception is
`laborious.utils.bundle.format`, which is pure `struct`/`hashlib` over bytes and lets cell 3 print
the header of the file it is about to upload. The envelope is built here in four visible lines
rather than imported, because those four lines are what the frontend has to transliterate to
WebCrypto; `laborious/utils/import_password.py` is the decrypt counterpart.
"""
# %%
# Cell 1 — resolve every target from the environment and print what we are pointed at.
# Reads only. No secret is printed here or anywhere else in this script.
from __future__ import annotations
import os
import sys
from pathlib import Path
from typing import Any
from dotenv import load_dotenv
REPO_ROOT = Path(__file__).resolve().parent.parent
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
load_dotenv(REPO_ROOT / '.env')
def env(name: str, default: str = '') -> str:
"""The environment variable, stripped, or `default`."""
return os.getenv(name, default).strip()
def env_required(name: str) -> str:
"""The environment variable, or a `RuntimeError` naming it."""
value = env(name)
if not value:
raise RuntimeError(f'{name} is not set — see .env.example')
return value
def _line(label: str, value: object) -> str:
"""One aligned `label value` line, for output that stays readable in a terminal."""
return f' {label:<24} {value}'
# Every connection target below is *required* rather than defaulted. The worker's own config falls
# back to localhost, which is right for a service that must boot; it is wrong for a script that
# writes, because a silent fallback is a write to somewhere nobody read out loud.
#
# The import log database: a second Postgres connection, independent of the POSTGRES_* one above it
# in .env. The schema and table are constants of the worker's code, never configuration.
STATUS_DB_HOST = env_required('IMPORT_STATUS_DB_HOST')
STATUS_DB_PORT = int(env_required('IMPORT_STATUS_DB_PORT'))
STATUS_DB_NAME = env_required('IMPORT_STATUS_DB_NAME')
STATUS_DB_USER = env_required('IMPORT_STATUS_DB_USER')
STATUS_DB_PASSWORD = os.getenv('IMPORT_STATUS_DB_PASSWORD', '')
STATUS_TABLE = 'public.experiment_run'
# Where the frontend uploads the bundle. The prefix keeps the worker's own default, which is a
# format decision rather than a target.
MINIO_ENDPOINT_URL = env_required('MINIO_ENDPOINT_URL')
MINIO_ACCESS_KEY = env_required('MINIO_ACCESS_KEY')
MINIO_SECRET_KEY = env_required('MINIO_SECRET_KEY')
BUNDLE_BUCKET = env_required('IMPORT_BUNDLE_BUCKET')
_raw_prefix = env('IMPORT_BUNDLE_PREFIX', 'imported_models/')
BUNDLE_PREFIX = _raw_prefix if _raw_prefix.endswith('/') else f'{_raw_prefix}/'
# Read-back targets. This script never writes to these; the workflow does.
MONGODB_URL = env_required('MONGODB_URL')
MONGODB_USERNAME = env('MONGODB_USERNAME')
MONGODB_PASSWORD = os.getenv('MONGODB_PASSWORD', '')
MONGODB_DATABASE = env_required('MONGODB_DATABASE')
MODELS_COLLECTION = env('IMPORT_MODELS_COLLECTION', 'models')
MLFLOW_HOST = env_required('MLFLOW_HOST')
MLFLOW_PORT = env_required('MLFLOW_PORT')
MLFLOW_TRACKING_URI = f'{MLFLOW_HOST}:{MLFLOW_PORT}'
TEMPORAL_HOST = env_required('TEMPORAL_HOST')
TEMPORAL_NAMESPACE = env_required('TEMPORAL_NAMESPACE')
RUNTIME = env_required('RUNTIME')
IMPORT_TASK_QUEUE = f'import_model-{RUNTIME}-queue'
# This script's own inputs.
BUNDLE_PATH = Path(
env('IMPORT_SCRIPT_BUNDLE_PATH', 'tests/fixtures/bundle/golden_model_v3.sientia')
)
if not BUNDLE_PATH.is_absolute():
BUNDLE_PATH = REPO_ROOT / BUNDLE_PATH
IMPORT_USERNAME = env_required('IMPORT_SCRIPT_USERNAME')
print('resolved targets — nothing has been written yet')
print(_line('status postgres', f'{STATUS_DB_HOST}:{STATUS_DB_PORT}/{STATUS_DB_NAME}'))
print(_line('status user', STATUS_DB_USER))
print(_line('status table', STATUS_TABLE))
print(_line('minio endpoint', MINIO_ENDPOINT_URL))
print(_line('minio bucket/prefix', f'{BUNDLE_BUCKET}/{BUNDLE_PREFIX}'))
print(_line('mongo', f'{MONGODB_URL}/{MONGODB_DATABASE}.{MODELS_COLLECTION}'))
print(_line('mlflow', MLFLOW_TRACKING_URI))
print(_line('temporal', f'{TEMPORAL_HOST} ns={TEMPORAL_NAMESPACE}'))
print(_line('task queue', IMPORT_TASK_QUEUE))
print(_line('bundle', BUNDLE_PATH))
print(_line('username', IMPORT_USERNAME))
print()
print(f'to continue, set the cell 2 confirmation to: {STATUS_DB_HOST}')
# %%
# Cell 2 — the confirmation guard. Cells 5 and 6 write; this is the last cell before them.
#
# Restate the Postgres host cell 1 printed. Either set IMPORT_SCRIPT_CONFIRM_TARGET in .env, or
# replace the right-hand side below with the value cell 1 told you. It is a read-then-restate on
# purpose: a boolean flag can be flipped without ever looking at what it is pointed at.
CONFIRM_TARGET = env('IMPORT_SCRIPT_CONFIRM_TARGET')
if CONFIRM_TARGET != STATUS_DB_HOST:
raise RuntimeError(
'refusing to write: the confirmation does not match the resolved status database host. '
f'confirmation={CONFIRM_TARGET!r}, resolved IMPORT_STATUS_DB_HOST={STATUS_DB_HOST!r}. '
'Read the targets cell 1 printed, then restate the host here.'
)
print(f'target confirmed: {STATUS_DB_HOST} — the write cells may run')
# %%
# Cell 3 — pick the bundle, compute the digest the frontend must send, print its header.
#
# `expected_digest` is the SHA-256 of the exact bytes that get uploaded, computed the way the
# frontend computes it: over the file, not over anything derived from it. The header fields printed
# here are plaintext at the front of the file — reading them decrypts nothing.
import hashlib
from laborious.utils.bundle.format import DIGEST_OFFSET, HEADER_SIZE, parse_header
bundle_bytes = BUNDLE_PATH.read_bytes()
expected_digest = hashlib.sha256(bundle_bytes).hexdigest()
object_name = BUNDLE_PATH.name
object_key = f'{BUNDLE_PREFIX}{object_name}'
header = parse_header(bundle_bytes[:HEADER_SIZE])
print(_line('file', BUNDLE_PATH))
print(_line('size', f'{len(bundle_bytes)} bytes'))
print(_line('expected_digest', expected_digest))
print(_line('object key', f'{BUNDLE_BUCKET}/{object_key}'))
print(_line('format version', header.format_version))
print(_line('aead / kdf id', f'{header.aead_id} / {header.kdf_id}'))
print(_line('kdf memlimit', f'{header.kdf_memlimit_bytes} bytes'))
print(_line('kdf opslimit', header.kdf_opslimit))
print(_line('kdf parallelism', header.kdf_parallelism))
print(_line('salt', header.salt.hex()))
print(_line('ciphertext digest', header.ciphertext_digest.hex()))
print(_line('chunk stream from', f'offset {DIGEST_OFFSET}'))
# %%
# Cell 4 — build the password envelope, exactly as the frontend must build it.
#
# Wire shape: base64( nonce(12) || ciphertext || GCM tag(16) ) under AES-256-GCM with the 32-byte
# key that `IMPORT_PASSWORD_KEY` carries base64-encoded. The four lines below are the whole of it,
# written out rather than imported so the Angular side can transliterate them to WebCrypto:
#
# const k = await crypto.subtle.importKey('raw', keyBytes, 'AES-GCM', false, ['encrypt']);
# const nonce = crypto.getRandomValues(new Uint8Array(12));
# const sealed = await crypto.subtle.encrypt({name: 'AES-GCM', iv: nonce}, k, passwordBytes);
# const envelope = base64(concat(nonce, new Uint8Array(sealed)));
#
# WebCrypto appends the GCM tag to the ciphertext, and so does `cryptography`'s AESGCM, so the two
# produce byte-identical payloads. The decrypt counterpart the worker runs is
# `laborious/utils/import_password.py` (`decrypt_password_envelope`).
#
# Neither the key nor the password nor the plaintext is printed — only the envelope's length.
import base64
import secrets
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
envelope_key = base64.b64decode(env_required('IMPORT_PASSWORD_KEY'), validate=True)
nonce = secrets.token_bytes(12)
sealed = AESGCM(envelope_key).encrypt(
nonce, env_required('IMPORT_SCRIPT_BUNDLE_PASSWORD').encode('utf-8'), None
)
password_envelope = base64.b64encode(nonce + sealed).decode('ascii')
print(_line('envelope key', f'{len(envelope_key)} bytes (not printed)'))
print(_line('envelope', f'{len(password_envelope)} base64 characters (not printed)'))
# %%
# Cell 5 — FIRST WRITE. Upload the object, then prove the stored bytes are the ones we hashed.
#
# The re-read is the point: `expected_digest` is a promise about what is in the bucket, and the
# worker's gate 1 holds the upload to it. If the digest of the stored object differs, the frontend
# contract is already broken and there is no reason to insert a row.
import io
from urllib.parse import urlparse
from minio import Minio
_endpoint = urlparse(MINIO_ENDPOINT_URL)
minio_client = Minio(
_endpoint.netloc or MINIO_ENDPOINT_URL,
access_key=MINIO_ACCESS_KEY,
secret_key=MINIO_SECRET_KEY,
secure=_endpoint.scheme == 'https',
)
minio_client.put_object(
BUNDLE_BUCKET,
object_key,
io.BytesIO(bundle_bytes),
length=len(bundle_bytes),
content_type='application/octet-stream',
)
stat = minio_client.stat_object(BUNDLE_BUCKET, object_key)
_response = minio_client.get_object(BUNDLE_BUCKET, object_key)
try:
stored_bytes = _response.read()
finally:
_response.close()
_response.release_conn()
stored_digest = hashlib.sha256(stored_bytes).hexdigest()
if stored_digest != expected_digest:
raise RuntimeError(
f'the stored object does not hash to the digest we computed: stored {stored_digest}, '
f'expected {expected_digest} — do not insert a row for this upload'
)
print(_line('uploaded', f'{BUNDLE_BUCKET}/{object_key}'))
print(_line('reported size', f'{stat.size} bytes'))
print(_line('etag', stat.etag))
print(_line('re-read digest', f'{stored_digest} (matches expected_digest)'))
# %%
# Cell 6 — SECOND WRITE. Insert the import log row: the BFF's half of the contract.
#
# `URL.create` rather than a URL string, so a password containing `@ : /` survives. `experiment_name`
# is a placeholder: the column is NOT NULL and the frontend cannot see inside an encrypted bundle,
# so the worker overwrites it with the bundle's own experiment name as soon as the bundle is open
# (`record_import_names`). `file_name` is the object key, because the worker compares the two and
# refuses the import at the `received` step when they disagree.
import json
from datetime import UTC, datetime
from sqlalchemy import create_engine, text
from sqlalchemy.engine import URL
status_engine = create_engine(
URL.create(
'postgresql',
username=STATUS_DB_USER,
password=STATUS_DB_PASSWORD,
host=STATUS_DB_HOST,
port=STATUS_DB_PORT,
database=STATUS_DB_NAME,
),
pool_pre_ping=True,
)
request_data = {
'expected_digest': expected_digest,
'bucket': BUNDLE_BUCKET,
'object_key': object_key,
'file_name': object_name,
'source': 'scripts/import_model_manual_run.py',
}
_now = datetime.now(UTC).replace(tzinfo=None)
_placeholder_experiment = f'import-{Path(object_name).stem}'[:50]
INSERT_ROW_SQL = text(
'INSERT INTO public.experiment_run '
'(experiment_name, username, status, created_at, updated_at, '
' bucket_name, file_name, request_data, run_type) '
'VALUES (:experiment_name, :username, :status, :created_at, :updated_at, '
' :bucket_name, :file_name, CAST(:request_data AS JSONB), :run_type) '
'RETURNING id'
)
with status_engine.begin() as connection:
import_run_id = connection.execute(
INSERT_ROW_SQL,
{
'experiment_name': _placeholder_experiment,
'username': IMPORT_USERNAME,
'status': 'PENDING',
'created_at': _now,
'updated_at': _now,
'bucket_name': BUNDLE_BUCKET,
'file_name': object_key,
'request_data': json.dumps(request_data),
'run_type': 'IMPORT',
},
).scalar_one()
print(_line('import_run_id', import_run_id))
print(_line('status / run_type', 'PENDING / IMPORT'))
print(_line('experiment_name', f'{_placeholder_experiment} (the worker overwrites this)'))
print(_line('file_name', object_key))
# %%
# Cell 7 — the workflow input, ready to paste into the Temporal UI.
#
# These five keys are the whole contract. `bucket` is optional — the worker falls back to
# IMPORT_BUNDLE_BUCKET — and is sent explicitly so the printed input is self-describing. No model
# name is sent: the worker takes it from the bundle's own metadata, verbatim.
workflow_input = {
'import_run_id': import_run_id,
'bucket': BUNDLE_BUCKET,
'object_key': object_key,
'expected_digest': expected_digest,
'password_envelope': password_envelope,
}
print(f'workflow: import_model task queue: {IMPORT_TASK_QUEUE}')
print(json.dumps(workflow_input, indent=2))
# %%
# Cell 8 — optionally start the workflow and wait for it.
#
# Skipping this cell is normal: the operator may prefer to start the run from the Temporal UI with
# the JSON cell 7 printed. Cell 9 works either way, because it looks the outcome up by row id.
#
# The coroutine runs on a loop of its own in a worker thread rather than through `asyncio.run`, so
# this cell behaves the same in the interactive window (which already has a running loop) as it
# does when the module is executed top to bottom (which does not).
START_THE_WORKFLOW = False
if START_THE_WORKFLOW:
import asyncio
from concurrent.futures import ThreadPoolExecutor
from temporalio.client import Client
async def start_and_wait() -> Any:
"""Start `import_model` on the import queue and return its result."""
client = await Client.connect(TEMPORAL_HOST, namespace=TEMPORAL_NAMESPACE)
handle = await client.start_workflow(
'import_model',
workflow_input,
id=f'manual-import-{import_run_id}',
task_queue=IMPORT_TASK_QUEUE,
)
print(_line('workflow id', handle.id))
print(_line('run id', handle.result_run_id))
return await handle.result()
with ThreadPoolExecutor(max_workers=1) as _pool:
workflow_result = _pool.submit(asyncio.run, start_and_wait()).result()
print(json.dumps(workflow_result, indent=2, default=str))
else:
print('not started — paste the input from cell 7 into the Temporal UI, or set')
print(f'START_THE_WORKFLOW = True and re-run this cell. Task queue: {IMPORT_TASK_QUEUE}')
# %%
# Cell 9 — read the outcome back from all three places it lands.
#
# The row is authoritative for the verdict; MLflow and MongoDB are what the verdict is about. Each
# block is labelled, so the output can be pasted into a ticket as it is.
print('--- import log row ---')
READ_ROW_SQL = text(
'SELECT id, experiment_name, run_name, status, error_message, run_type, '
' bucket_name, file_name, created_at, updated_at, '
' request_data, orchestrator_response_data '
'FROM public.experiment_run WHERE id = :id'
)
with status_engine.begin() as connection:
_result = connection.execute(READ_ROW_SQL, {'id': import_run_id})
row = _result.mappings().one_or_none()
if row is None:
print(f' no row with id {import_run_id}')
else:
for _column, _value in row.items():
print(_line(_column, _value))
print()
print('--- mlflow ---')
import mlflow
os.environ['MLFLOW_TRACKING_USERNAME'] = env('MLFLOW_USERNAME')
os.environ['MLFLOW_TRACKING_PASSWORD'] = os.getenv('MLFLOW_PASSWORD', '')
mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)
mlflow_client = mlflow.tracking.MlflowClient()
_experiment_name = row['experiment_name'] if row is not None else ''
_experiment = mlflow_client.get_experiment_by_name(_experiment_name) if _experiment_name else None
if _experiment is None:
print(f' no experiment named {_experiment_name!r}')
else:
print(_line('experiment', f'{_experiment.name} ({_experiment.experiment_id})'))
for _run in mlflow_client.search_runs([_experiment.experiment_id]):
print(_line('run', f'{_run.info.run_name} ({_run.info.run_id}) {_run.info.status}'))
for _artifact in mlflow_client.list_artifacts(_run.info.run_id):
print(_line(' artifact', _artifact.path))
# The run name is `import-{model_name}-v{version}` (`derive_run_name`), so the registered model
# name is what sits between the prefix and the version suffix.
_run_name = (row['run_name'] if row is not None else '') or ''
_registered = _run_name.removeprefix('import-').rsplit('-v', 1)[0] if _run_name else ''
if _registered:
try:
for _version in mlflow_client.search_model_versions(f"name='{_registered}'"):
print(
_line(
'version',
f'{_version.name} v{_version.version} stage={_version.current_stage} '
f'run={_version.run_id}',
)
)
print(_line(' source', _version.source))
except Exception as error:
# A read-back cell reports what it could not read; it does not fail the inspection.
print(f' registry read failed: {type(error).__name__}: {error}')
else:
print(' no run_name on the row yet, so no registered model to look up')
print()
print('--- mongodb model document ---')
from pymongo import MongoClient
_credentials = f'{MONGODB_USERNAME}:{MONGODB_PASSWORD}@' if MONGODB_USERNAME else ''
mongo_client: MongoClient = MongoClient(f'mongodb://{_credentials}{MONGODB_URL}')
_collection = mongo_client[MONGODB_DATABASE][MODELS_COLLECTION]
for _document in _collection.find({}, {'_id': False}).sort('id', 1):
print(_line('document', _document))
# %%
# Cell 10 — undo.
#
# The object and the row always: they are what this script created, and leaving them behind leaves a
# PENDING import nobody will ever run. What MLflow and MongoDB hold is left alone unless
# DELETE_PROVISIONING is set, because after a successful run that is the thing you wanted to look at.
DELETE_PROVISIONING = False
try:
minio_client.remove_object(BUNDLE_BUCKET, object_key)
print(_line('deleted object', f'{BUNDLE_BUCKET}/{object_key}'))
except Exception as error:
# Undo reports what it could not remove and carries on to the row.
print(f' could not delete the object: {type(error).__name__}: {error}')
with status_engine.begin() as connection:
_deleted = connection.execute(
text('DELETE FROM public.experiment_run WHERE id = :id'),
{'id': import_run_id},
).rowcount
print(_line('deleted rows', _deleted))
if DELETE_PROVISIONING:
if _experiment is not None:
mlflow_client.delete_experiment(_experiment.experiment_id)
print(_line('deleted experiment', _experiment.experiment_id))
if _registered:
try:
mlflow_client.delete_registered_model(_registered)
print(_line('deleted model', _registered))
except Exception as error:
# Same rule as above: report and carry on to the document.
print(f' could not delete the registered model: {type(error).__name__}: {error}')
_removed = _collection.delete_many({'name': _registered}).deleted_count
print(_line('deleted documents', _removed))
else:
print(' provisioning kept — set DELETE_PROVISIONING = True to remove it too')
status_engine.dispose()
mongo_client.close()

4
inter_arrival.py → scripts/inter_arrival.py
View File

@@ -1,11 +1,12 @@
# %%
# Load logs.txt
with open('logs.txt', 'r') as file:
with open('logs.txt') as file:
lines = file.readlines()
# %%
import re
# Grep "inter-arrival_s=number" with regex
intervals = []
for line in lines:
@@ -17,6 +18,7 @@ for line in lines:
print(intervals)
# %%
import matplotlib.pyplot as plt
plt.plot(intervals)
plt.ylabel('Inter-arrival time (s)')
plt.xlabel('Sample')

55
model_convert.ipynb → scripts/model_convert.ipynb
View File

@@ -9,6 +9,7 @@
"source": [
"import csv\n",
"\n",
"\n",
"def csv_to_tag_lists(csv_path: str) -> dict:\n",
" read_tags = []\n",
" write_tags = []\n",
@@ -19,48 +20,48 @@
" except Exception:\n",
" return None\n",
"\n",
" with open(csv_path, newline=\"\", encoding=\"utf-8\") as f:\n",
" with open(csv_path, newline='', encoding='utf-8') as f:\n",
" reader = csv.DictReader(f)\n",
" for row in reader:\n",
" # Basic normalization\n",
" op = (row.get(\"operation\") or \"\").strip()\n",
" op = (row.get('operation') or '').strip()\n",
"\n",
" if op == \"READ\":\n",
" if op == 'READ':\n",
" # Build common tag payload with required mappings\n",
" tag = {\n",
" \"server_id\": \"1\",\n",
" \"tag_address\": row.get(\"opc_tag\"),\n",
" \"tag_name\": row.get(\"name\"),\n",
" \"data_range\": [to_float(row.get(\"min_value\")), to_float(row.get(\"max_value\"))],\n",
" \"aggr_func\": row.get(\"aggregation_func\").lower(),\n",
" 'server_id': '1',\n",
" 'tag_address': row.get('opc_tag'),\n",
" 'tag_name': row.get('name'),\n",
" 'data_range': [to_float(row.get('min_value')), to_float(row.get('max_value'))],\n",
" 'aggr_func': row.get('aggregation_func').lower(),\n",
" # keep other fields with their original names\n",
" \"frequency\": row.get(\"frequency\"),\n",
" \"local\": row.get(\"local\"),\n",
" \"area\": row.get(\"area\"),\n",
" \"description\": row.get(\"description\"),\n",
" 'frequency': row.get('frequency'),\n",
" 'local': row.get('local'),\n",
" 'area': row.get('area'),\n",
" 'description': row.get('description'),\n",
" }\n",
"\n",
" read_tags.append(tag)\n",
"\n",
" else:\n",
" tag = {\n",
" \"server_id\": \"1\",\n",
" \"addr\": row.get(\"opc_tag\"),\n",
" \"tag_name\": row.get(\"name\"),\n",
" \"local\": row.get(\"local\"),\n",
" \"area\": row.get(\"area\"),\n",
" \"description\": row.get(\"description\"),\n",
" 'server_id': '1',\n",
" 'addr': row.get('opc_tag'),\n",
" 'tag_name': row.get('name'),\n",
" 'local': row.get('local'),\n",
" 'area': row.get('area'),\n",
" 'description': row.get('description'),\n",
" }\n",
" \n",
" if op == \"WRITE_PREDICTION\":\n",
" tag[\"type\"] = \"prediction\"\n",
"\n",
" if op == 'WRITE_PREDICTION':\n",
" tag['type'] = 'prediction'\n",
" write_tags.append(tag)\n",
" elif op == \"WRITE_CONFIDENCE\":\n",
" tag[\"type\"] = \"confidence\"\n",
" elif op == 'WRITE_CONFIDENCE':\n",
" tag['type'] = 'confidence'\n",
" write_tags.append(tag)\n",
" # ignore any other operation values silently\n",
"\n",
" return {\"read_tags\": read_tags, \"write_tags\": write_tags}"
" return {'read_tags': read_tags, 'write_tags': write_tags}"
]
},
{
@@ -72,12 +73,12 @@
"source": [
"import json\n",
"\n",
"file_names = [\"Courier - Página1.csv\"]\n",
"file_names = ['Courier - Página1.csv']\n",
"\n",
"for file_name in file_names:\n",
" write_file = file_name.replace(\".csv\", \".json\")\n",
" write_file = file_name.replace('.csv', '.json')\n",
"\n",
" with open(write_file, \"w\", encoding=\"utf-8\") as f:\n",
" with open(write_file, 'w', encoding='utf-8') as f:\n",
" json.dump(csv_to_tag_lists(file_name), f, indent=2, ensure_ascii=False)"
]
}

0
run_coverage.sh → scripts/run_coverage.sh
View File

0
run_local.sh → scripts/run_local.sh
View File

938
tests.ipynb
View File

File diff suppressed because one or more lines are too long

174
tests/helpers/status_engine.py Normal file
View File

@@ -0,0 +1,174 @@
"""A recording stand-in for the import log's SQLAlchemy engine.
The import's status writes are the part of the feature with the strictest contract: which table they
name, which predicate they carry, which columns they touch, how many transactions they open and
which of them may fail without failing the import. Asserting that against a mock whose `execute`
returns a `MagicMock` proves nothing — the assertions need the statement text and the parameters.
So this double records every statement it is handed, answers reads from a queue the test fills, and
lets a test make a write fail on a chosen attempt. It deliberately understands nothing about SQL
beyond the first word of the statement.
"""
from __future__ import annotations
from typing import Any
class RecordedStatement:
"""One statement as the engine saw it: its normalised text and its bound parameters."""
def __init__(self, statement: Any, parameters: dict[str, Any] | None) -> None:
self.sql = ' '.join(str(statement).split())
self.parameters: dict[str, Any] = dict(parameters or {})
@property
def verb(self) -> str:
return self.sql.split(' ', 1)[0].upper()
@property
def is_write(self) -> bool:
return self.verb in ('UPDATE', 'INSERT', 'DELETE')
@property
def table(self) -> str:
"""The relation the statement names, for the "one table only" assertions."""
words = self.sql.split()
if self.verb == 'UPDATE':
return words[1]
for index, word in enumerate(words):
if word.upper() in ('FROM', 'INTO') and index + 1 < len(words):
return words[index + 1]
return ''
@property
def assignments(self) -> set[str]:
"""The column names a `SET` clause assigns, so a test can assert what is never written."""
upper = self.sql.upper()
if ' SET ' not in upper:
return set()
body = self.sql[upper.index(' SET ') + 5 :]
where = body.upper().find(' WHERE ')
if where != -1:
body = body[:where]
return {part.split('=')[0].strip() for part in body.split(',') if '=' in part}
def __repr__(self) -> str: # pragma: no cover - debugging aid
return f'RecordedStatement({self.sql!r}, {self.parameters!r})'
class FakeResult:
"""What `Connection.execute` returns: a rowcount for writes, mappings for reads."""
def __init__(self, rowcount: int = 1, rows: list[dict[str, Any]] | None = None) -> None:
self.rowcount = rowcount
self._rows = rows or []
def mappings(self) -> FakeResult:
return self
def first(self) -> dict[str, Any] | None:
return self._rows[0] if self._rows else None
def all(self) -> list[dict[str, Any]]:
return list(self._rows)
class FakeConnection:
"""A connection that records and then answers from the engine's queues."""
def __init__(self, engine: FakeStatusEngine) -> None:
self._engine = engine
def execute(self, statement: Any, parameters: dict[str, Any] | None = None) -> FakeResult:
recorded = RecordedStatement(statement, parameters)
self._engine.statements.append(recorded)
if recorded.is_write:
return self._engine._answer_write()
return self._engine._answer_read()
class _Scope:
"""The context manager `engine.begin()` and `engine.connect()` both hand out."""
def __init__(self, engine: FakeStatusEngine, transactional: bool) -> None:
self._engine = engine
self._transactional = transactional
def __enter__(self) -> FakeConnection:
self._engine.open_scopes += 1
if self._transactional:
self._engine.transactions += 1
else:
self._engine.connections += 1
return FakeConnection(self._engine)
def __exit__(self, *_exc: object) -> bool:
self._engine.open_scopes -= 1
return False
class FakeStatusEngine:
"""An engine double that records statements and can be told to fail a write.
Attributes:
statements: every statement executed, in order.
rowcounts: rowcounts for writes, consumed in order; the last one repeats.
rows: rows for reads, consumed in order; the last one repeats. `None` means "no such row".
write_errors: exceptions to raise instead of executing a write, consumed in order. A `None`
entry means "this attempt succeeds", which is how a retry is modelled.
transactions: how many `begin()` scopes were opened — one per write, or the writes are
sharing a transaction.
open_scopes: scopes currently open, so a test can assert none is left open.
"""
def __init__(
self,
rowcounts: list[int] | None = None,
rows: list[dict[str, Any] | None] | None = None,
write_errors: list[BaseException | None] | None = None,
) -> None:
self.statements: list[RecordedStatement] = []
self.rowcounts: list[int] = list(rowcounts or [])
self.rows: list[dict[str, Any] | None] = list(rows or [])
self.write_errors: list[BaseException | None] = list(write_errors or [])
self.transactions = 0
self.connections = 0
self.open_scopes = 0
self.disposed = False
# ------------------------------------------------------------------ engine API
def begin(self) -> _Scope:
return _Scope(self, transactional=True)
def connect(self) -> _Scope:
return _Scope(self, transactional=False)
def dispose(self) -> None:
self.disposed = True
# ------------------------------------------------------------------- answering
def _answer_write(self) -> FakeResult:
if self.write_errors:
error = self.write_errors.pop(0)
if error is not None:
raise error
rowcount = self.rowcounts.pop(0) if len(self.rowcounts) > 1 else (self.rowcounts or [1])[0]
return FakeResult(rowcount=rowcount)
def _answer_read(self) -> FakeResult:
row = self.rows.pop(0) if len(self.rows) > 1 else (self.rows or [None])[0]
return FakeResult(rows=[row] if row is not None else [])
# ---------------------------------------------------------------- convenience
@property
def writes(self) -> list[RecordedStatement]:
return [statement for statement in self.statements if statement.is_write]
@property
def reads(self) -> list[RecordedStatement]:
return [statement for statement in self.statements if not statement.is_write]
@property
def tables(self) -> set[str]:
return {statement.table for statement in self.statements if statement.table}

115
tests/laborious/activities/test_activities.py
View File

@@ -156,6 +156,121 @@ def test___init__(
)
BASE_CONFIGS = {
'postgres_config': {
'host': 'localhost',
'port': 5432,
'user': 'postgres',
'password': 'postgres',
'dbname': 'postgres',
'min_connections': 1,
'max_connections': 10,
},
'minio_config': {
'endpoint_url': 'localhost:9000',
'access_key': 'minio',
'secret_key': 'minio123',
'default_bucket': 'test',
'retention_hours': 24,
'secure': False,
},
'mlflow_config': {'host': 'localhost', 'port': 5000, 'username': 'm', 'password': 'm'},
'opc_config': {},
'pi_web_api_config': {'base_url': 'https://pi', 'auth_type': 'bearer', 'auth_token': 't'},
}
IMPORT_CONFIG = {'bucket': 'sientia', 'prefix': 'imported_models/'}
IMPORT_STATUS_CONFIG = {
'host': 'bff-db',
'port': 5432,
'dbname': 'sientia-core-mlops-bff',
'user': 'u',
'password': 'p',
'schema': 'public',
}
@patch('laborious.activities.activities.ModelImport.__init__')
@patch('laborious.activities.activities.Storage.__init__')
@patch('laborious.activities.activities.MLFlow.__init__')
@patch('laborious.activities.activities.OPC.__init__')
@patch('laborious.activities.activities.Gates.__init__')
@patch('laborious.activities.activities.ModelMetrics.__init__')
@patch('laborious.activities.activities.API.__init__')
@patch('laborious.activities.activities.MongoDBRepository')
@patch('laborious.activities.activities.MinioRepository')
@patch('laborious.activities.activities.MetricsController')
def test_the_import_activities_share_the_worker_s_mlflow_repository( # noqa: PLR0913
_mock_metrics_controller,
mock_minio_repository,
mock_mongo_repository,
_mock_api_init,
_mock_model_metrics_init,
_mock_gates_init,
_mock_opc_init,
mock_mlflow_init,
_mock_storage_init,
mock_model_import_init,
):
"""Setting the MLflow tracking URI is a process-wide side effect, so two repository instances
would be two chances to disagree about where MLflow is."""
repository = MagicMock(name='model_monitoring_repository')
mock_mlflow_init.side_effect = lambda instance, **_: setattr(
instance, 'model_monitoring_repository', repository
)
Activities(
**BASE_CONFIGS,
logger=MagicMock(),
notification_handler=MagicMock(),
mongo_config={'connection_string': 'mongodb://mongo', 'database_name': 'sientia'},
import_config=IMPORT_CONFIG,
import_status_config=IMPORT_STATUS_CONFIG,
)
passed = mock_model_import_init.call_args.kwargs
assert passed['mlflow_repository'] is repository
assert passed['minio_repository'] is mock_minio_repository.return_value
assert passed['mongo_repository'] is mock_mongo_repository.return_value
assert passed['import_config'] is IMPORT_CONFIG
assert passed['status_config'] is IMPORT_STATUS_CONFIG
@patch('laborious.activities.activities.ModelImport.__init__')
@patch('laborious.activities.activities.Storage.__init__')
@patch('laborious.activities.activities.MLFlow.__init__')
@patch('laborious.activities.activities.OPC.__init__')
@patch('laborious.activities.activities.Gates.__init__')
@patch('laborious.activities.activities.ModelMetrics.__init__')
@patch('laborious.activities.activities.API.__init__')
@patch('laborious.activities.activities.MongoDBRepository')
@patch('laborious.activities.activities.MinioRepository')
@patch('laborious.activities.activities.MetricsController')
def test_without_a_mongo_config_the_import_gets_no_repository( # noqa: PLR0913
_mock_metrics_controller,
_mock_minio_repository,
mock_mongo_repository,
_mock_api_init,
_mock_model_metrics_init,
_mock_gates_init,
_mock_opc_init,
_mock_mlflow_init,
_mock_storage_init,
mock_model_import_init,
):
"""It refuses rather than guessing: the notification handler keeps its own client either way."""
Activities(
**BASE_CONFIGS,
logger=MagicMock(),
notification_handler=MagicMock(),
import_config=IMPORT_CONFIG,
import_status_config=IMPORT_STATUS_CONFIG,
)
assert mock_model_import_init.call_args.kwargs['mongo_repository'] is None
mock_mongo_repository.assert_not_called()
@mark.asyncio
@patch('laborious.activities.activities.Storage')
@patch('laborious.activities.activities.MLFlow')

565
tests/laborious/activities/test_model_import_bundle.py Normal file
View File

@@ -0,0 +1,565 @@
"""Downloading the object and opening the bundle: the two activities that touch the secret.
Three things are asserted here:
- **the digest is enforced against the bytes that arrived**, before the header is parsed, with no
path that skips the comparison — a mismatch means the object behind that key is not the file the
user chose, and no amount of well-formedness makes it the right file;
- **the envelope is opened inside the activity that uses the password**, so a plaintext password is
never an activity input, an activity result, or a value the workflow holds;
- **nothing observable carries the secret** — for every failure path, the password, the envelope key
and the derived key are absent from the log, the notification, the metric labels, the exception and
the return value.
The bundles here are real: the shared factory writes them field for field like the producer, at the
test cost parameters, and the production reader opens them with real Argon2id and real
`secretstream`.
"""
import hashlib
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
from pytest import fixture, mark, raises
from laborious import metrics
from laborious.activities.model_import import ModelImport
from laborious.activities.model_import_errors import ImportGateRejectedError
from laborious.utils.bundle.steps import ImportStep
from tests.helpers import bundle_factory as factory
from tests.helpers.status_engine import FakeStatusEngine
# Distinctive enough that a substring search for it in a log line cannot match by accident.
BUNDLE_PASSWORD = 'Zk9-bundle-secret-Passw0rd'
ENVELOPE_KEY = bytes(range(100, 132))
OTHER_ENVELOPE_KEY = bytes(range(200, 232))
METADATA = {
'model_id': '77',
'model_name': 'factory_model',
'workflow_name': 'import_model',
'schedule_name': '-',
}
WORKFLOW_ID = 'import_model-77-abc'
OBJECT_KEY = 'imported_models/factory_model_v2.sientia'
def import_config(work_dir: Path, **overrides) -> dict:
return {
'password_key': factory.envelope_key_b64(ENVELOPE_KEY),
'bucket': 'sientia',
'prefix': 'imported_models/',
'work_dir': str(work_dir),
'max_object_bytes': 8 * 1024 * 1024,
'max_entries': 5000,
'max_uncompressed_bytes': 4 * 1024 * 1024,
'max_compression_ratio': 200.0,
'models_collection': 'models',
'retention_days': 7,
**overrides,
}
@fixture
def minio_client() -> MagicMock:
client = MagicMock()
client.stat_object.return_value = MagicMock(size=1024)
return client
@fixture
def import_activity(tmp_path: Path, minio_client: MagicMock) -> ModelImport:
repository = MagicMock()
repository.minio_client = minio_client
with patch('laborious.activities.model_import.create_engine', return_value=FakeStatusEngine()):
activity = ModelImport(
status_config={
'host': 'h',
'port': 5432,
'dbname': 'sientia-core-mlops-bff',
'user': 'u',
'password': 'p',
'schema': 'public',
},
import_config=import_config(tmp_path),
minio_repository=repository,
mlflow_repository=MagicMock(),
mongo_repository=AsyncMock(),
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
)
activity.emit_metric = AsyncMock()
activity.send_notification_async = AsyncMock()
return activity
def serve_object(minio_client: MagicMock, payload: bytes) -> None:
"""Make `fget_object` write `payload` where the activity asked for it."""
def write(_bucket, _key, destination):
Path(destination).write_bytes(payload)
minio_client.stat_object.return_value = MagicMock(size=len(payload))
minio_client.fget_object.side_effect = write
def download_input(digest: str, **overrides) -> dict:
return {
'metadata': METADATA,
'import_run_id': 77,
'workflow_id': WORKFLOW_ID,
'object_key': OBJECT_KEY,
'expected_digest': digest,
**overrides,
}
def open_input(path: str, envelope: str | None = None, **overrides) -> dict:
return {
'metadata': METADATA,
'import_run_id': 77,
'workflow_id': WORKFLOW_ID,
'path': path,
'password_envelope': envelope
if envelope is not None
else factory.build_envelope(BUNDLE_PASSWORD, ENVELOPE_KEY),
**overrides,
}
def observable_text(activity: ModelImport) -> str:
"""Everything this activity said out loud: logs, notifications and metric labels."""
logger = activity.logger
parts = [
str(call) for call in logger.custom_info.call_args_list + logger.custom_error.call_args_list
]
parts += [str(call) for call in activity.send_notification_async.await_args_list]
parts += [str(call) for call in activity.emit_metric.await_args_list]
return '\n'.join(parts)
# ------------------------------------------------------------------------- the download
@mark.asyncio
async def test_a_matching_digest_yields_the_downloaded_file(
import_activity: ModelImport, minio_client: MagicMock, tmp_path: Path
):
payload = factory.make_bundle(password=BUNDLE_PASSWORD)
serve_object(minio_client, payload)
digest = hashlib.sha256(payload).hexdigest()
result = await import_activity.download_import_bundle(download_input(digest))
assert Path(result['path']).read_bytes() == payload
assert result['digest'] == digest
assert result['size'] == len(payload)
assert Path(result['path']).parent == tmp_path / WORKFLOW_ID
@mark.asyncio
async def test_the_configured_bucket_is_used_when_the_request_names_none(
import_activity: ModelImport, minio_client: MagicMock
):
payload = factory.make_bundle(password=BUNDLE_PASSWORD)
serve_object(minio_client, payload)
await import_activity.download_import_bundle(
download_input(hashlib.sha256(payload).hexdigest())
)
assert minio_client.stat_object.call_args.args[0] == 'sientia'
@mark.asyncio
async def test_the_request_may_name_another_bucket(
import_activity: ModelImport, minio_client: MagicMock
):
payload = factory.make_bundle(password=BUNDLE_PASSWORD)
serve_object(minio_client, payload)
await import_activity.download_import_bundle(
download_input(hashlib.sha256(payload).hexdigest(), bucket='other-bucket')
)
assert minio_client.stat_object.call_args.args[0] == 'other-bucket'
@mark.asyncio
async def test_a_digest_from_another_file_fails_before_the_header_is_parsed(
import_activity: ModelImport, minio_client: MagicMock
):
payload = factory.make_bundle(password=BUNDLE_PASSWORD)
other = factory.make_bundle(password=BUNDLE_PASSWORD, salt=b'\x01' * 16)
serve_object(minio_client, payload)
with raises(ImportGateRejectedError) as failure:
await import_activity.download_import_bundle(
download_input(hashlib.sha256(other).hexdigest())
)
assert failure.value.details[0]['step'] == ImportStep.DOWNLOAD.value
assert failure.value.details[0]['gate'] == 1
@mark.asyncio
async def test_a_file_that_does_not_match_is_not_left_on_the_worker(
import_activity: ModelImport, minio_client: MagicMock, tmp_path: Path
):
serve_object(minio_client, factory.make_bundle(password=BUNDLE_PASSWORD))
with raises(ImportGateRejectedError):
await import_activity.download_import_bundle(download_input('0' * 64))
assert not (tmp_path / WORKFLOW_ID / 'bundle.sientia').exists()
@mark.asyncio
async def test_the_digest_comparison_is_case_insensitive_on_the_input(
import_activity: ModelImport, minio_client: MagicMock
):
payload = factory.make_bundle(password=BUNDLE_PASSWORD)
serve_object(minio_client, payload)
result = await import_activity.download_import_bundle(
download_input(hashlib.sha256(payload).hexdigest().upper())
)
assert result['digest'] == hashlib.sha256(payload).hexdigest()
@mark.asyncio
async def test_an_object_over_the_ceiling_is_refused_before_its_body_is_fetched(
import_activity: ModelImport, minio_client: MagicMock
):
minio_client.stat_object.return_value = MagicMock(size=9 * 1024 * 1024)
with raises(ImportGateRejectedError) as failure:
await import_activity.download_import_bundle(download_input('a' * 64))
assert failure.value.details[0]['step'] == ImportStep.DOWNLOAD.value
minio_client.fget_object.assert_not_called()
@mark.asyncio
@mark.parametrize(
'object_key',
['other_prefix/model.sientia', 'imported_models/model.zip'],
)
async def test_a_key_of_the_wrong_shape_is_refused(
import_activity: ModelImport, minio_client: MagicMock, object_key: str
):
with raises(ImportGateRejectedError):
await import_activity.download_import_bundle(
download_input('a' * 64, object_key=object_key)
)
minio_client.fget_object.assert_not_called()
@mark.asyncio
async def test_an_unreadable_object_is_a_download_failure(
import_activity: ModelImport, minio_client: MagicMock
):
minio_client.stat_object.side_effect = OSError('no such object')
with raises(ImportGateRejectedError) as failure:
await import_activity.download_import_bundle(download_input('a' * 64))
assert failure.value.details[0]['step'] == ImportStep.DOWNLOAD.value
@mark.asyncio
async def test_a_failed_transfer_is_a_download_failure(
import_activity: ModelImport, minio_client: MagicMock
):
minio_client.fget_object.side_effect = OSError('connection reset')
with raises(ImportGateRejectedError) as failure:
await import_activity.download_import_bundle(download_input('a' * 64))
assert failure.value.details[0]['step'] == ImportStep.DOWNLOAD.value
@mark.asyncio
async def test_a_missing_minio_repository_is_a_configuration_error(tmp_path: Path):
with patch('laborious.activities.model_import.create_engine', return_value=FakeStatusEngine()):
activity = ModelImport(
status_config={
'host': 'h',
'port': 5432,
'dbname': 'd',
'user': 'u',
'password': 'p',
'schema': 'public',
},
import_config=import_config(tmp_path),
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
)
with raises(ValueError, match='Minio repository not initialized'):
await activity.download_import_bundle(download_input('a' * 64))
# ----------------------------------------------------------------------- opening the bundle
@mark.asyncio
async def test_a_well_formed_bundle_opens_and_returns_nothing_secret(
import_activity: ModelImport, tmp_path: Path
):
path = tmp_path / 'bundle.sientia'
path.write_bytes(factory.make_bundle(password=BUNDLE_PASSWORD))
result = await import_activity.open_import_bundle(open_input(str(path)))
assert result['metadata']['model_name'] == 'factory_model'
assert result['parameters']['target_variable'] == 'SE'
assert Path(result['extracted_dir']).is_dir()
assert (Path(result['extracted_dir']) / 'artifacts' / 'prediction_model' / 'MLmodel').is_file()
assert set(result) == {'extracted_dir', 'metadata', 'parameters', 'digest'}
assert BUNDLE_PASSWORD not in str(result)
@mark.asyncio
async def test_opening_a_bundle_measures_the_kdf(import_activity: ModelImport, tmp_path: Path):
path = tmp_path / 'bundle.sientia'
path.write_bytes(factory.make_bundle(password=BUNDLE_PASSWORD))
await import_activity.open_import_bundle(open_input(str(path)))
observed = [
call.kwargs
for call in import_activity.emit_metric.await_args_list
if call.kwargs.get('metric_object') is metrics.MODEL_IMPORT_KDF_DURATION
]
assert len(observed) == 1
assert observed[0]['method'] == 'observe'
assert observed[0]['value'] > 0
@mark.asyncio
async def test_a_wrong_envelope_key_is_not_confused_with_a_wrong_bundle_password(
import_activity: ModelImport, tmp_path: Path
):
path = tmp_path / 'bundle.sientia'
path.write_bytes(factory.make_bundle(password=BUNDLE_PASSWORD))
envelope = factory.build_envelope(BUNDLE_PASSWORD, OTHER_ENVELOPE_KEY)
with raises(ImportGateRejectedError) as failure:
await import_activity.open_import_bundle(open_input(str(path), envelope))
details = failure.value.details[0]
assert details['step'] == ImportStep.DECRYPTION.value
assert details['gate'] == 3
# The platform's own transport failed, not the user's password. Same step — the step is where the
# import stopped, not who is at fault — but a distinguishable cause.
assert details['cause'] == 'envelope'
@mark.asyncio
async def test_a_wrong_bundle_password_is_a_decryption_failure_of_its_own(
import_activity: ModelImport, tmp_path: Path
):
path = tmp_path / 'bundle.sientia'
path.write_bytes(factory.make_bundle(password='a-different-password'))
with raises(ImportGateRejectedError) as failure:
await import_activity.open_import_bundle(open_input(str(path)))
details = failure.value.details[0]
assert details['step'] == ImportStep.DECRYPTION.value
assert details['cause'] != 'envelope'
@mark.asyncio
async def test_a_missing_password_key_stops_the_import_at_decryption(
import_activity: ModelImport, tmp_path: Path
):
import_activity.import_config['password_key'] = None
path = tmp_path / 'bundle.sientia'
path.write_bytes(factory.make_bundle(password=BUNDLE_PASSWORD))
with raises(ImportGateRejectedError) as failure:
await import_activity.open_import_bundle(open_input(str(path)))
assert failure.value.details[0]['cause'] == 'envelope'
@mark.asyncio
@mark.parametrize(
'envelope',
['', 'not-base64!!', 'AAAA', 'bm90LWFuLWVudmVsb3BlLWF0LWFsbA=='],
ids=['empty', 'not-base64', 'too-short', 'wrong-bytes'],
)
async def test_a_tampered_or_malformed_envelope_is_refused(
import_activity: ModelImport, tmp_path: Path, envelope: str
):
path = tmp_path / 'bundle.sientia'
path.write_bytes(factory.make_bundle(password=BUNDLE_PASSWORD))
with raises(ImportGateRejectedError) as failure:
await import_activity.open_import_bundle(open_input(str(path), envelope))
assert failure.value.details[0]['cause'] == 'envelope'
def bad_magic() -> bytes:
return factory.make_bundle(password=BUNDLE_PASSWORD, magic=b'NOTSIEN1')
def wrong_digest() -> bytes:
return factory.make_bundle(password=BUNDLE_PASSWORD, digest_override=bytes(32))
def tampered() -> bytes:
return factory.tamper_ciphertext(factory.make_bundle(password=BUNDLE_PASSWORD))
def traversal() -> bytes:
return factory.make_bundle(
factory.ZipSpec(traversal_entry='artifacts/../../escaped.csv'), password=BUNDLE_PASSWORD
)
def absolute() -> bytes:
return factory.make_bundle(
factory.ZipSpec(absolute_entry='/etc/passwd'), password=BUNDLE_PASSWORD
)
def missing_data_model() -> bytes:
files = {
name: content
for name, content in factory.default_tree().items()
if not name.startswith('artifacts/data_model/')
}
return factory.make_bundle(factory.ZipSpec(files=files), password=BUNDLE_PASSWORD)
def missing_target_variable() -> bytes:
document = factory.build_metadata_document(drop_parameter='target_variable')
return factory.make_bundle(factory.ZipSpec(document=document), password=BUNDLE_PASSWORD)
def script_in_the_tree() -> bytes:
return factory.make_bundle(
factory.ZipSpec(extra_files={'artifacts/run.sh': b'#!/bin/sh\nrm -rf /\n'}),
password=BUNDLE_PASSWORD,
)
@mark.asyncio
@mark.parametrize(
('build', 'step', 'gate'),
[
(bad_magic, ImportStep.DOWNLOAD, 2),
(wrong_digest, ImportStep.DOWNLOAD, 2),
(tampered, ImportStep.DECRYPTION, 3),
(traversal, ImportStep.ARCHIVE_INSPECTION, 4),
(absolute, ImportStep.ARCHIVE_INSPECTION, 4),
(missing_data_model, ImportStep.STRUCTURE_VALIDATION, 6),
(missing_target_variable, ImportStep.STRUCTURE_VALIDATION, 6),
(script_in_the_tree, ImportStep.CONTENT_POLICY, 7),
],
ids=lambda value: getattr(value, '__name__', str(value)),
)
async def test_a_gate_failure_reaches_the_activity_as_its_own_step(
import_activity: ModelImport, tmp_path: Path, build, step: ImportStep, gate: int
):
path = tmp_path / 'bundle.sientia'
path.write_bytes(build())
with raises(ImportGateRejectedError) as failure:
await import_activity.open_import_bundle(open_input(str(path)))
assert failure.value.details[0]['step'] == step.value
assert failure.value.details[0]['gate'] == gate
@mark.asyncio
async def test_a_rejected_bundle_leaves_no_extracted_tree(
import_activity: ModelImport, tmp_path: Path
):
path = tmp_path / 'bundle.sientia'
path.write_bytes(traversal())
with raises(ImportGateRejectedError):
await import_activity.open_import_bundle(open_input(str(path)))
extract_root = tmp_path / WORKFLOW_ID / 'extract'
assert not extract_root.exists() or list(extract_root.rglob('*')) == []
# ---------------------------------------------------------------------------- secrecy
SECRET_PATHS = [
('happy', lambda: factory.make_bundle(password=BUNDLE_PASSWORD), None),
('wrong-bundle-password', lambda: factory.make_bundle(password='another-password'), None),
('wrong-envelope-key', lambda: factory.make_bundle(password=BUNDLE_PASSWORD), 'other-key'),
('malformed-envelope', lambda: factory.make_bundle(password=BUNDLE_PASSWORD), 'garbage'),
('gate-2', bad_magic, None),
('gate-3', tampered, None),
('gate-4', traversal, None),
('gate-6', missing_data_model, None),
('gate-7', script_in_the_tree, None),
]
@mark.asyncio
@mark.parametrize(
('build', 'envelope_kind'),
[(build, kind) for _, build, kind in SECRET_PATHS],
ids=[name for name, _, _ in SECRET_PATHS],
)
async def test_no_failure_path_says_the_password_out_loud(
import_activity: ModelImport, tmp_path: Path, build, envelope_kind: str | None
):
"""Every path through this activity, checked against the three things that must never surface."""
path = tmp_path / 'bundle.sientia'
path.write_bytes(build())
if envelope_kind == 'other-key':
envelope = factory.build_envelope(BUNDLE_PASSWORD, OTHER_ENVELOPE_KEY)
elif envelope_kind == 'garbage':
envelope = 'not-base64!!'
else:
envelope = factory.build_envelope(BUNDLE_PASSWORD, ENVELOPE_KEY)
raised: BaseException | None = None
result = None
try:
result = await import_activity.open_import_bundle(open_input(str(path), envelope))
except BaseException as error: # noqa: BLE001 - the point is that *whatever* it is says nothing
raised = error
secrets = [
BUNDLE_PASSWORD,
'another-password',
factory.envelope_key_b64(ENVELOPE_KEY),
ENVELOPE_KEY.hex(),
]
surfaces = [observable_text(import_activity), str(result)]
if raised is not None:
surfaces += [str(raised), str(getattr(raised, 'details', '')), repr(raised)]
for secret in secrets:
for surface in surfaces:
assert secret not in surface
@mark.asyncio
async def test_the_envelope_never_leaves_the_activity_that_opens_it(
import_activity: ModelImport, tmp_path: Path
):
"""The result is what crosses back into the workflow, and therefore into the event history."""
path = tmp_path / 'bundle.sientia'
path.write_bytes(factory.make_bundle(password=BUNDLE_PASSWORD))
envelope = factory.build_envelope(BUNDLE_PASSWORD, ENVELOPE_KEY)
result = await import_activity.open_import_bundle(open_input(str(path), envelope))
assert envelope not in str(result)
assert 'password' not in str(result).lower()

410
tests/laborious/activities/test_model_import_errors.py Normal file
View File

@@ -0,0 +1,410 @@
"""The failure catalog and the one channel every import failure passes through.
Two contracts are asserted here, and both are about what a person is allowed to see:
- **the catalog is closed and total** — every pipeline step that can fail has exactly one code and
one sentence, the sentence never carries a path, a table name, an exception class or a traceback,
and the classification is a function of the *step* alone, so rewording a sentence cannot move a
code the frontend switches on;
- **the channel is the only reporter** — one call reaches the log, the metric and the notification,
the exception's text reaches nothing but the log, and no other module in the import path logs an
error or notifies (asserted from the source, so the rule cannot rot).
"""
import ast
from pathlib import Path
from typing import Any
from pytest import mark, raises
from laborious import metrics
from laborious.activities.model_import_errors import (
IMPORT_FAILURE_CATALOG,
STEPS_WITHOUT_A_USER_FACING_REASON,
UNCLASSIFIED_FAILURE,
ImportErrorCode,
ImportGateRejectedError,
ImportInputError,
ImportLogRowNotClaimableError,
ImportStatusRejectedError,
classify_import_failure,
failure_location,
report_import_failure,
)
from laborious.utils.bundle.steps import GATE_TO_STEP, ImportStep
REPOSITORY_ROOT = Path(__file__).resolve().parents[3]
# The modules that make up the import path. The channel lives in the third one; the other two may
# not report a failure of their own.
IMPORT_MODULES = (
REPOSITORY_ROOT / 'laborious/activities/model_import.py',
REPOSITORY_ROOT / 'laborious/workflows/import_model.py',
)
# What a leaked exception message looks like: a work directory, the table, an exception class and a
# traceback line. None of it may reach a recorded reason.
LEAKY_MESSAGE = (
"KeyError: 'target_variable' while writing public.experiment_run from "
'/tmp/sientia-import/import_model-77-abc/extract\n' # noqa: S108 - the leak under test
' File "laborious/activities/model_import.py", line 1, in write_import_model_document'
)
# Every fragment of the leaky message that must not reach a recorded reason.
LEAKS_THAT_MUST_NOT_SURFACE = (
'sientia-import',
'public.experiment_run',
'KeyError',
'File "',
'line 1',
)
STEPS_THAT_ONLY_SUPPORT_CAN_ACT_ON = (
ImportStep.EXPERIMENT_CREATION,
ImportStep.ARTIFACT_UPLOAD,
ImportStep.REGISTRATION,
ImportStep.MODEL_DOCUMENT,
)
class FakeSinks:
"""The observability surface the channel writes to, recorded rather than performed."""
def __init__(self) -> None:
self.logged: list[tuple[str, dict]] = []
self.metrics: list[tuple[Any, dict]] = []
self.notifications: list[dict] = []
def error(self, message: str, metadata: dict | None = None) -> None:
self.logged.append((message, metadata or {}))
def get_core_labels(self, metadata: dict | None = None, operation_type: str = '-') -> dict:
return {
'model_name': (metadata or {}).get('model_name', '-'),
'operation_type': operation_type,
}
async def emit_metric(
self,
metric_object: Any,
tags: dict[str, Any],
method: str = 'inc',
value: float | None = None,
) -> None:
self.metrics.append((metric_object, tags))
async def send_notification_async(
self,
metadata: dict,
notification_id: str,
message: str,
block: str,
level: str = 'INFO',
attachment_content: str | None = None,
) -> None:
self.notifications.append(
{
'metadata': metadata,
'notification_id': notification_id,
'message': message,
'block': block,
'level': level,
}
)
# ------------------------------------------------------------------------- the catalog
def test_every_failing_step_has_exactly_one_entry():
"""A step added without a sentence fails here rather than reaching a user as `None`."""
expected = set(ImportStep) - STEPS_WITHOUT_A_USER_FACING_REASON
assert set(IMPORT_FAILURE_CATALOG) == expected
def test_cleanup_is_the_only_step_without_a_sentence():
"""Cleanup failing is a log line on a successful import, never the reported failure."""
assert STEPS_WITHOUT_A_USER_FACING_REASON == frozenset({ImportStep.CLEANUP})
assert ImportStep.CLEANUP not in IMPORT_FAILURE_CATALOG
def test_every_gate_maps_to_a_step_that_has_a_sentence():
for step in GATE_TO_STEP.values():
assert step in IMPORT_FAILURE_CATALOG
def test_each_step_has_its_own_code():
codes = [code for code, _ in IMPORT_FAILURE_CATALOG.values()]
assert len(codes) == len(set(codes))
def test_the_codes_that_never_reach_a_record_are_absent_from_the_catalog():
catalogued = {code for code, _ in IMPORT_FAILURE_CATALOG.values()}
for code in (
ImportErrorCode.LOG_ROW_NOT_CLAIMABLE,
ImportErrorCode.STATUS_NOT_RECORDED,
ImportErrorCode.CLEANUP_INCOMPLETE,
):
assert code not in catalogued
@mark.parametrize('step', list(IMPORT_FAILURE_CATALOG))
def test_a_reason_never_carries_the_exception(step: ImportStep):
failure = classify_import_failure(step, RuntimeError(LEAKY_MESSAGE))
for leak in LEAKS_THAT_MUST_NOT_SURFACE:
assert leak not in failure.reason
assert failure.reason == IMPORT_FAILURE_CATALOG[step][1]
@mark.parametrize('step', list(IMPORT_FAILURE_CATALOG))
def test_the_classification_is_a_function_of_the_step_alone(step: ImportStep):
"""Which is what makes a reworded sentence keep its code: neither is read from the exception."""
first = classify_import_failure(step, RuntimeError(LEAKY_MESSAGE))
second = classify_import_failure(step, KeyError('target_variable'))
third = classify_import_failure(step, None)
assert first == second == third
assert first.code is IMPORT_FAILURE_CATALOG[step][0]
def test_the_decryption_reason_does_not_say_which_of_the_two_it_was():
reason = IMPORT_FAILURE_CATALOG[ImportStep.DECRYPTION][1]
# Both causes are named as possibilities; neither is asserted, because the AEAD cannot tell them
# apart and guessing would be a hint about somebody else's password.
assert 'password is wrong' in reason
assert 'damaged' in reason
assert 'altered' in reason
@mark.parametrize('step', STEPS_THAT_ONLY_SUPPORT_CAN_ACT_ON)
def test_a_reason_the_user_cannot_act_on_names_the_import_identifier(step: ImportStep):
assert "Contact support with this import's identifier." in IMPORT_FAILURE_CATALOG[step][1]
def test_a_failure_with_no_step_is_unclassified():
failure = classify_import_failure(None, RuntimeError('boom'))
assert failure.step is None
assert failure.code is ImportErrorCode.UNEXPECTED_ERROR
assert failure.reason == UNCLASSIFIED_FAILURE[1]
def test_every_reason_is_long_enough_for_the_column():
"""`error_message` carries a `LENGTH(...) >= 3` CHECK; every sentence here clears it."""
for _, reason in IMPORT_FAILURE_CATALOG.values():
assert len(reason) >= 3
# ----------------------------------------------------------------- the typed failures
def test_the_two_unrepeatable_failures_are_non_retryable():
assert ImportLogRowNotClaimableError('x').non_retryable is True
assert ImportStatusRejectedError('x').non_retryable is True
assert ImportInputError('x').non_retryable is True
def test_a_gate_rejection_carries_its_step_and_gate_across_the_boundary():
"""An exception loses its Python type crossing an activity boundary but keeps its details."""
error = ImportGateRejectedError('rejected', step=ImportStep.ARCHIVE_INSPECTION, gate=4)
assert failure_location(error) == ('archive_inspection', 4)
def test_an_input_error_reports_the_received_step():
assert failure_location(ImportInputError('missing object_key')) == ('received', None)
def test_a_located_failure_is_found_through_the_activity_wrapper():
wrapper = RuntimeError('activity failed')
wrapper.__cause__ = ImportGateRejectedError('nope', step=ImportStep.EXTRACTION, gate=5)
assert failure_location(wrapper) == ('extraction', 5)
def test_an_unlocated_failure_says_nothing_rather_than_guessing():
assert failure_location(RuntimeError('boom')) == (None, None)
assert failure_location(None) == (None, None)
def test_a_cause_chain_cannot_loop_forever():
first = RuntimeError('a')
second = RuntimeError('b')
first.__cause__ = second
second.__cause__ = first
assert failure_location(first) == (None, None)
# ------------------------------------------------------------------------- the channel
@mark.asyncio
async def test_the_channel_hits_every_sink_once():
sinks = FakeSinks()
failure = await report_import_failure(
sinks,
ImportStep.REGISTRATION,
RuntimeError(LEAKY_MESSAGE),
import_run_id=77,
workflow_id='import_model-77-abc',
metadata={'model_name': 'imported_model'},
)
assert failure.code is ImportErrorCode.MODEL_NOT_REGISTERED
assert len(sinks.logged) == 1
assert len(sinks.metrics) == 1
assert len(sinks.notifications) == 1
assert sinks.metrics[0][0] is metrics.MODEL_IMPORT_ERROR_COUNT
assert sinks.metrics[0][1]['operation_type'] == ImportErrorCode.MODEL_NOT_REGISTERED.value
@mark.asyncio
async def test_the_exception_text_reaches_the_log_and_nothing_else():
sinks = FakeSinks()
try:
raise RuntimeError(LEAKY_MESSAGE)
except RuntimeError as raised:
caught = raised
await report_import_failure(
sinks,
ImportStep.MODEL_DOCUMENT,
caught,
import_run_id=77,
workflow_id='import_model-77-abc',
)
logged, _ = sinks.logged[0]
assert 'KeyError' in logged
assert 'public.experiment_run' in logged
assert 'Traceback' in logged
notification = sinks.notifications[0]
assert notification['message'] == IMPORT_FAILURE_CATALOG[ImportStep.MODEL_DOCUMENT][1]
assert 'KeyError' not in notification['message']
assert 'KeyError' not in str(sinks.metrics[0][1])
@mark.asyncio
async def test_the_log_line_correlates_the_record_and_the_workflow():
sinks = FakeSinks()
await report_import_failure(
sinks,
ImportStep.DOWNLOAD,
RuntimeError('boom'),
import_run_id=77,
workflow_id='import_model-77-abc',
)
logged, _ = sinks.logged[0]
assert 'import_run_id=77' in logged
assert 'workflow_id=import_model-77-abc' in logged
assert 'step=download' in logged
@mark.asyncio
async def test_a_failure_that_is_not_a_step_reports_under_its_marker():
sinks = FakeSinks()
failure = await report_import_failure(
sinks,
None,
None,
import_run_id=77,
workflow_id='wf',
code=ImportErrorCode.STATUS_NOT_RECORDED,
marker='status_write',
verdict='succeeded=True step=model_document gate=None',
detail_text='OperationalError: connection refused',
)
logged, _ = sinks.logged[0]
assert failure.code is ImportErrorCode.STATUS_NOT_RECORDED
assert 'step=status_write' in logged
assert 'verdict_not_recorded=succeeded=True step=model_document gate=None' in logged
assert 'connection refused' in logged
assert sinks.notifications[0]['block'] == 'status_write'
@mark.asyncio
async def test_a_report_without_an_exception_object_says_so():
sinks = FakeSinks()
await report_import_failure(sinks, None, None, import_run_id=77, workflow_id='wf')
assert 'no exception object available' in sinks.logged[0][0]
@mark.asyncio
async def test_an_explicit_code_keeps_the_step_s_sentence():
"""The cleanup case: its own code, and a sentence a person can still read."""
sinks = FakeSinks()
failure = await report_import_failure(
sinks,
ImportStep.CLEANUP,
OSError('device busy'),
import_run_id=77,
workflow_id='wf',
code=ImportErrorCode.CLEANUP_INCOMPLETE,
marker=ImportStep.CLEANUP.value,
)
assert failure.code is ImportErrorCode.CLEANUP_INCOMPLETE
assert failure.reason == UNCLASSIFIED_FAILURE[1]
assert sinks.notifications[0]['level'] == 'ERROR'
# ------------------------------------------------- the channel is the only reporter
def _reporting_calls(path: Path) -> list[str]:
"""Every call in `path` that would log an error or send a notification."""
forbidden = {
'error',
'custom_error',
'critical',
'custom_critical',
'send_notification',
'send_notification_async',
'publish_notification',
}
tree = ast.parse(path.read_text())
found = []
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
continue
target = node.func
name = target.attr if isinstance(target, ast.Attribute) else getattr(target, 'id', '')
if name in forbidden:
found.append(f'{path.name}:{node.lineno}:{name}')
return found
@mark.parametrize('module_path', IMPORT_MODULES, ids=lambda path: path.name)
def test_no_import_module_reports_a_failure_of_its_own(module_path: Path):
"""So no failure is reported twice, or half-reported by whoever noticed it first."""
assert _reporting_calls(module_path) == []
def test_the_channel_itself_is_where_the_reporting_lives():
channel = REPOSITORY_ROOT / 'laborious/activities/model_import_errors.py'
calls = _reporting_calls(channel)
assert [call.split(':')[2] for call in calls] == ['error', 'send_notification_async']
def test_the_reader_neither_logs_nor_notifies():
"""The bundle reader raises; it is the activity layer that decides what a failure means."""
for module in ('laborious/utils/bundle/reader.py', 'laborious/utils/bundle/format.py'):
assert _reporting_calls(REPOSITORY_ROOT / module) == []
def test_an_unclassified_step_cannot_be_recorded_as_a_known_one():
with raises(ValueError, match='not a valid ImportStep'):
ImportStep('provisioning')

656
tests/laborious/activities/test_model_import_provisioning.py Normal file
View File

@@ -0,0 +1,656 @@
"""Provisioning: the experiment, the run, the registered version, the document, the cleanup.
The boundary this file defends is where the import stops. A finished import leaves a version whose
stage was never touched and a document that is inert — promoting and activating are an operator's
decisions, owned by another card — so the tests assert not only what is written but what is *not*:
no stage transition, no alias, no `active: true`, no update of a document already there.
The rest is the model listing's arithmetic (`max(int(id)) + 1`, which is not `count` and not a
lexicographic sort) and the two operations that must never fail an import that otherwise worked:
removing the scratch directory and putting an expiry rule on the bundle prefix.
"""
import json
import shutil
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
import mlflow as mlflow_lib
from pytest import fixture, mark, raises
from laborious.activities.model_import import ModelImport
from laborious.utils.repository.model_repository import MLFlowRepository
from tests.helpers.status_engine import FakeStatusEngine
METADATA = {
'model_id': '77',
'model_name': 'factory_model',
'workflow_name': 'import_model',
'schedule_name': '-',
}
WORKFLOW_ID = 'import_model-77-abc'
RUN_ID = '0123456789abcdef0123456789abcdef'
# What a tracking server with proxied artifacts reports as a run's artifact root, and therefore what
# the version's `source` is built from. The run id sits at index 2 of this URI — the whole reason the
# registration cannot store a literal `runs:/…` source.
ARTIFACT_URI = f'mlflow-artifacts:/17/{RUN_ID}/artifacts'
RESOLVED_SOURCE = f'{ARTIFACT_URI}/prediction_model'
BUNDLE_PARAMETERS = {
'model_name': 'factory_model',
'target_variable': 'SE',
'input_variables': "['feature_a', 'feature_b']",
'retrain': 'False',
}
BUNDLE_METADATA = {
'model_name': 'factory_model',
'experiment_name': 'factory_experiment',
'run_id': 'aa11bb22cc33dd44ee55ff6677889900',
'model_version': '2',
'model_project': 'factory_project',
'export_timestamp': '2026-08-25T12:00:00+00:00',
}
# The whole vocabulary of promotion. None of it may be reached by this change.
PROMOTION_CALLS = (
'transition_model_version_stage',
'set_registered_model_alias',
'set_model_version_tag',
'update_model_version',
)
@fixture
def mlflow_client() -> MagicMock:
client = MagicMock()
client.create_run.return_value.info.run_id = RUN_ID
client.get_run.return_value.info.artifact_uri = ARTIFACT_URI
client.create_model_version.return_value.version = 4
return client
@fixture
def mongo() -> AsyncMock:
repository = AsyncMock()
repository.find_one.return_value = None
repository.distinct.return_value = []
return repository
@fixture
def import_activity(tmp_path: Path, mlflow_client: MagicMock, mongo: AsyncMock) -> ModelImport:
minio_repository = MagicMock()
mlflow_repository = MagicMock()
mlflow_repository.client = mlflow_client
with patch('laborious.activities.model_import.create_engine', return_value=FakeStatusEngine()):
activity = ModelImport(
status_config={
'host': 'h',
'port': 5432,
'dbname': 'sientia-core-mlops-bff',
'user': 'u',
'password': 'p',
'schema': 'public',
},
import_config={
'password_key': None,
'bucket': 'sientia',
'prefix': 'imported_models/',
'work_dir': str(tmp_path),
'max_object_bytes': 1024,
'max_entries': 10,
'max_uncompressed_bytes': 4096,
'max_compression_ratio': 200.0,
'models_collection': 'models',
'retention_days': 7,
},
minio_repository=minio_repository,
mlflow_repository=mlflow_repository,
mongo_repository=mongo,
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
)
activity.emit_metric = AsyncMock()
activity.send_notification_async = AsyncMock()
return activity
def base(**overrides) -> dict:
return {
'metadata': METADATA,
'import_run_id': 77,
'workflow_id': WORKFLOW_ID,
**overrides,
}
# ---------------------------------------------------------------------- the experiment
@mark.asyncio
async def test_the_experiment_keeps_the_origin_s_name(import_activity: ModelImport):
repository = import_activity.import_mlflow_repository
repository.get_experiment.return_value = MagicMock(experiment_id='17')
result = await import_activity.create_import_experiment(
base(experiment_name='factory_experiment')
)
repository.get_experiment.assert_called_once_with(
'factory_experiment', create_if_not_exists=True
)
assert result == {'experiment_id': '17', 'experiment_name': 'factory_experiment'}
@mark.asyncio
async def test_a_missing_mlflow_repository_is_a_configuration_error(import_activity: ModelImport):
import_activity.import_mlflow_repository = None
with raises(ValueError, match='MLflow repository not initialized'):
await import_activity.create_import_experiment(base(experiment_name='e'))
# ------------------------------------------------------------------- the run and its tree
def upload_input(extracted_dir: Path, **overrides) -> dict:
return base(
experiment_id='17',
run_name='import-factory_model-v2',
extracted_dir=str(extracted_dir),
parameters=BUNDLE_PARAMETERS,
source=BUNDLE_METADATA,
**overrides,
)
@mark.asyncio
async def test_every_import_creates_a_new_run_with_the_recorded_name(
import_activity: ModelImport, mlflow_client: MagicMock, tmp_path: Path
):
"""The name is passed in, never recomputed, so the record and the run cannot diverge."""
result = await import_activity.upload_import_artifacts(upload_input(tmp_path / 'extract'))
mlflow_client.create_run.assert_called_once_with(
experiment_id='17', run_name='import-factory_model-v2'
)
assert result == {'run_id': RUN_ID, 'run_name': 'import-factory_model-v2'}
@mark.asyncio
async def test_the_artifact_tree_is_logged_from_the_bundle_s_artifacts_directory(
import_activity: ModelImport, mlflow_client: MagicMock, tmp_path: Path
):
extracted = tmp_path / 'extract'
await import_activity.upload_import_artifacts(upload_input(extracted))
mlflow_client.log_artifacts.assert_called_once_with(RUN_ID, str(extracted / 'artifacts'))
@mark.asyncio
async def test_the_bundle_s_parameters_are_logged_verbatim(
import_activity: ModelImport, mlflow_client: MagicMock, tmp_path: Path
):
await import_activity.upload_import_artifacts(upload_input(tmp_path / 'extract'))
logged = {
call.args[1]: call.args[2]
for call in mlflow_client.log_param.call_args_list
if not call.args[1].startswith('source.')
}
assert logged == BUNDLE_PARAMETERS
assert logged['target_variable'] == 'SE'
@mark.asyncio
async def test_the_origin_is_logged_as_source_parameters(
import_activity: ModelImport, mlflow_client: MagicMock, tmp_path: Path
):
await import_activity.upload_import_artifacts(upload_input(tmp_path / 'extract'))
logged = {
call.args[1]: call.args[2]
for call in mlflow_client.log_param.call_args_list
if call.args[1].startswith('source.')
}
assert logged == {f'source.{key}': value for key, value in BUNDLE_METADATA.items()}
# A record of where the model came from, not a link: the origin run id is never dereferenced.
assert logged['source.run_id'] == BUNDLE_METADATA['run_id']
mlflow_client.get_run.assert_not_called()
mlflow_client.download_artifacts.assert_not_called()
@mark.asyncio
async def test_the_run_is_closed_when_its_tree_is_in(
import_activity: ModelImport, mlflow_client: MagicMock, tmp_path: Path
):
await import_activity.upload_import_artifacts(upload_input(tmp_path / 'extract'))
mlflow_client.set_terminated.assert_called_once_with(RUN_ID, 'FINISHED')
# ------------------------------------------------------------------------ registration
@mark.asyncio
async def test_the_version_source_is_the_shape_the_legacy_loader_parses(
import_activity: ModelImport, mlflow_client: MagicMock
):
result = await import_activity.register_import_model_version(
base(model_name='factory_model', run_id=RUN_ID)
)
assert result == {'model_name': 'factory_model', 'version': '4', 'source': RESOLVED_SOURCE}
mlflow_client.create_model_version.assert_called_once_with(
name='factory_model', source=RESOLVED_SOURCE, run_id=RUN_ID
)
# The one property the legacy loader depends on, asserted directly.
assert RESOLVED_SOURCE.split('/')[2] == RUN_ID
@mark.asyncio
async def test_a_literal_runs_uri_would_not_be_parseable_by_the_legacy_loader():
"""The trap this registration avoids: `create_model_version` stores `source` verbatim, and a
`runs:/` URI has only two slashes, so the loader's index 2 is the artifact path."""
assert f'runs:/{RUN_ID}/prediction_model'.split('/')[2] == 'prediction_model'
assert 'models:/m-9f1c'.split('/')[1:] == ['m-9f1c']
@mark.asyncio
async def test_a_registered_model_is_created_only_when_it_is_absent(
import_activity: ModelImport, mlflow_client: MagicMock
):
mlflow_client.get_registered_model.side_effect = RuntimeError('not found')
await import_activity.register_import_model_version(base(model_name='new_model', run_id=RUN_ID))
mlflow_client.create_registered_model.assert_called_once_with('new_model')
@mark.asyncio
async def test_a_second_import_reuses_the_registered_model(
import_activity: ModelImport, mlflow_client: MagicMock
):
mlflow_client.get_registered_model.return_value = MagicMock(name='factory_model')
await import_activity.register_import_model_version(
base(model_name='factory_model', run_id=RUN_ID)
)
mlflow_client.create_registered_model.assert_not_called()
@mark.asyncio
async def test_the_import_never_promotes_anything(
import_activity: ModelImport, mlflow_client: MagicMock, tmp_path: Path
):
"""A `COMPLETED` import is a registered model, not a servable one."""
await import_activity.create_import_experiment(base(experiment_name='factory_experiment'))
await import_activity.upload_import_artifacts(upload_input(tmp_path / 'extract'))
await import_activity.register_import_model_version(
base(model_name='factory_model', run_id=RUN_ID)
)
for call_name in PROMOTION_CALLS:
getattr(mlflow_client, call_name).assert_not_called()
@mark.asyncio
async def test_no_provisioning_call_passes_a_stage(
import_activity: ModelImport, mlflow_client: MagicMock, tmp_path: Path
):
await import_activity.upload_import_artifacts(upload_input(tmp_path / 'extract'))
result = await import_activity.register_import_model_version(
base(model_name='factory_model', run_id=RUN_ID)
)
for call in mlflow_client.method_calls:
assert 'stage' not in call.kwargs
assert 'alias' not in call.kwargs
assert 'stage' not in result
# ------------------------------------------------- the legacy loader reads what we wrote
def legacy_repository() -> MLFlowRepository:
"""A real `MLFlowRepository`, with the process-wide side effects of its constructor stubbed."""
with (
patch.object(mlflow_lib, 'set_tracking_uri'),
patch.object(mlflow_lib.tracking, 'MlflowClient'),
):
return MLFlowRepository(
host='http://localhost:5000',
username='admin',
password='admin',
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
)
def registry_holding(source: str) -> MagicMock:
client = MagicMock()
client.search_registered_models.return_value = [MagicMock()]
version = MagicMock(version='4', source=source)
version.current_stage = 'Production'
client.search_model_versions.return_value = [version]
return client
def test_the_legacy_loader_resolves_the_run_the_import_created():
"""The contract the source shape exists for: the real production code, fed a version this
import would create, returns this import's run id."""
repository = legacy_repository()
repository.client = registry_holding(RESOLVED_SOURCE)
assert repository.get_model_run_id('factory_model') == RUN_ID
def test_the_legacy_loader_would_read_a_runs_uri_as_the_artifact_path():
"""Left as an executable warning: registering with the unresolved URI silently breaks loading,
because nothing raises — the loader just returns the wrong string."""
repository = legacy_repository()
repository.client = registry_holding(f'runs:/{RUN_ID}/prediction_model')
assert repository.get_model_run_id('factory_model') == 'prediction_model'
def test_the_legacy_loader_cannot_read_a_logged_model_uri():
repository = legacy_repository()
repository.client = registry_holding('models:/m-9f1c')
with raises(mlflow_lib.exceptions.MlflowException, match='invalid source URI'):
repository.get_model_run_id('factory_model')
# ------------------------------------------------------------------- the model document
@mark.asyncio
async def test_the_document_id_continues_the_collection_across_gaps(
import_activity: ModelImport, mongo: AsyncMock
):
mongo.distinct.return_value = [str(value) for value in [*range(1, 19), *range(20, 25), 30]]
result = await import_activity.write_import_model_document(
base(model_name='factory_model', target='SE')
)
assert result['id'] == '31'
@mark.asyncio
async def test_the_document_id_is_numeric_not_lexicographic(
import_activity: ModelImport, mongo: AsyncMock
):
mongo.distinct.return_value = ['9', '30']
result = await import_activity.write_import_model_document(
base(model_name='factory_model', target='SE')
)
assert result['id'] == '31'
@mark.asyncio
async def test_an_empty_collection_starts_at_one(import_activity: ModelImport, mongo: AsyncMock):
result = await import_activity.write_import_model_document(
base(model_name='factory_model', target='SE')
)
assert result['id'] == '1'
@mark.asyncio
async def test_the_document_is_exactly_the_contracted_shape(
import_activity: ModelImport, mongo: AsyncMock
):
mongo.distinct.return_value = ['7']
await import_activity.write_import_model_document(base(model_name='factory_model', target='SE'))
collection, document, metadata = mongo.insert.await_args.args
assert collection == 'models'
assert metadata == METADATA
assert document == {
'id': '8',
'name': 'factory_model',
'active': False,
'model_config': {
'transform_flavor': 'sklearn',
'predict_flavor': 'joblib',
'retention_minutes': 60,
'target': 'SE',
},
}
@mark.asyncio
async def test_the_document_carries_no_field_the_runtime_does_not_read(
import_activity: ModelImport, mongo: AsyncMock
):
await import_activity.write_import_model_document(base(model_name='factory_model', target='SE'))
_, document, _ = mongo.insert.await_args.args
for absent in ('is_compressed', 'retention_target', 'stage', 'source', 'project'):
assert absent not in document
assert absent not in document['model_config']
@mark.asyncio
async def test_a_name_already_in_the_collection_is_left_exactly_as_it_is(
import_activity: ModelImport, mongo: AsyncMock
):
"""`active`, `target` and the flavors of a model already on the platform are operator-owned."""
mongo.find_one.return_value = {'id': '12', 'name': 'factory_model', 'active': True}
result = await import_activity.write_import_model_document(
base(model_name='factory_model', target='SE')
)
assert result == {'written': False, 'skipped': True, 'id': '12'}
mongo.insert.assert_not_awaited()
mongo.distinct.assert_not_awaited()
@mark.asyncio
async def test_unusable_ids_are_ignored_and_counted(import_activity: ModelImport, mongo: AsyncMock):
mongo.distinct.return_value = ['1', 'legacy-model', None, '3']
result = await import_activity.write_import_model_document(
base(model_name='factory_model', target='SE')
)
assert result['id'] == '4'
assert result['ignored_ids'] == 2
assert any(
'ignored 2 non-numeric' in str(call)
for call in import_activity.logger.custom_info.call_args_list
)
@mark.asyncio
async def test_a_duplicate_id_fails_the_step_without_recomputing_it(
import_activity: ModelImport, mongo: AsyncMock
):
"""No retry and no second guess: the unique index says the collection moved under us."""
mongo.distinct.return_value = ['1']
mongo.insert.side_effect = RuntimeError('E11000 duplicate key error: id_1')
with raises(RuntimeError, match='duplicate key'):
await import_activity.write_import_model_document(
base(model_name='factory_model', target='SE')
)
assert mongo.distinct.await_count == 1
assert mongo.insert.await_count == 1
@mark.asyncio
async def test_the_import_only_reads_and_inserts_documents(
import_activity: ModelImport, mongo: AsyncMock
):
"""No update, no replace, no upsert: an import cannot change a model already on the platform."""
await import_activity.write_import_model_document(base(model_name='factory_model', target='SE'))
used = {name for name, _, _ in mongo.mock_calls}
assert used <= {'find_one', 'distinct', 'insert'}
@mark.asyncio
async def test_a_missing_mongo_repository_is_a_configuration_error(
import_activity: ModelImport,
):
import_activity.mongo_repository = None
with raises(ValueError, match='MongoDB repository not initialized'):
await import_activity.write_import_model_document(base(model_name='m', target='SE'))
def test_no_import_module_writes_an_active_document():
"""Asserted from the source too, so `active: false` cannot be flipped by a later edit."""
source = Path('laborious/activities/model_import.py').read_text()
assert "'active': False" in source
assert "'active': True" not in source
for forbidden in ('update_one', 'replace_one', 'update_many', 'upsert'):
assert forbidden not in source
# ------------------------------------------------------------------ cleanup and retention
@mark.asyncio
async def test_cleanup_removes_this_import_s_scratch_directory(
import_activity: ModelImport, tmp_path: Path
):
work_dir = tmp_path / WORKFLOW_ID
(work_dir / 'extract' / 'artifacts').mkdir(parents=True)
(work_dir / 'bundle.sientia').write_bytes(b'downloaded')
result = await import_activity.cleanup_import_files(base())
assert result == {'cleaned': True, 'existed': True}
assert not work_dir.exists()
@mark.asyncio
async def test_cleanup_of_a_run_that_never_downloaded_anything_is_not_a_failure(
import_activity: ModelImport,
):
result = await import_activity.cleanup_import_files(base())
assert result == {'cleaned': True, 'existed': False}
@mark.asyncio
async def test_cleanup_that_fails_is_reported_and_swallowed(
import_activity: ModelImport, tmp_path: Path
):
"""The user's model was imported; a leaked directory is a log line, not an ERROR."""
(tmp_path / WORKFLOW_ID).mkdir()
with patch.object(shutil, 'rmtree', side_effect=OSError('device or resource busy')):
result = await import_activity.cleanup_import_files(base())
assert result['cleaned'] is False
assert result['reason'] == 'OSError'
logged = import_activity.logger.custom_error.call_args_list[0].args[0]
assert 'IMPORT_CLEANUP_INCOMPLETE' in logged
import_activity.send_notification_async.assert_awaited_once()
@mark.asyncio
async def test_cleanup_never_touches_the_uploaded_object(
import_activity: ModelImport, tmp_path: Path
):
(tmp_path / WORKFLOW_ID).mkdir()
await import_activity.cleanup_import_files(base())
client = import_activity.import_minio_repository.minio_client
client.remove_object.assert_not_called()
client.remove_objects.assert_not_called()
@mark.asyncio
async def test_the_expiry_rule_is_applied_to_the_import_prefix(import_activity: ModelImport):
client = import_activity.import_minio_repository.minio_client
client.get_bucket_lifecycle.return_value = None
result = await import_activity.ensure_import_bundle_retention(base())
assert result == {'bucket': 'sientia', 'prefix': 'imported_models/', 'days': 7, 'rules': 1}
config = client.set_bucket_lifecycle.call_args.args[1]
rule = config.rules[0]
assert rule.rule_id == 'sientia-import-bundle-expiry'
assert rule.expiration.days == 7
assert rule.rule_filter.prefix == 'imported_models/'
@mark.asyncio
async def test_applying_the_rule_twice_leaves_one_rule(import_activity: ModelImport):
client = import_activity.import_minio_repository.minio_client
client.get_bucket_lifecycle.return_value = None
await import_activity.ensure_import_bundle_retention(base())
applied = client.set_bucket_lifecycle.call_args.args[1]
client.get_bucket_lifecycle.return_value = applied
await import_activity.ensure_import_bundle_retention(base())
config = client.set_bucket_lifecycle.call_args.args[1]
assert [rule.rule_id for rule in config.rules] == ['sientia-import-bundle-expiry']
@mark.asyncio
async def test_a_rule_that_belongs_to_somebody_else_is_kept(import_activity: ModelImport):
client = import_activity.import_minio_repository.minio_client
existing = MagicMock()
existing.rules = [MagicMock(rule_id='predictions-offload-expiry')]
client.get_bucket_lifecycle.return_value = existing
await import_activity.ensure_import_bundle_retention(base())
config = client.set_bucket_lifecycle.call_args.args[1]
assert [rule.rule_id for rule in config.rules] == [
'predictions-offload-expiry',
'sientia-import-bundle-expiry',
]
@mark.asyncio
async def test_a_bucket_with_no_lifecycle_configuration_at_all_is_handled(
import_activity: ModelImport,
):
"""Some deployments answer with an error, others with `None`; both mean "no rule yet"."""
client = import_activity.import_minio_repository.minio_client
client.get_bucket_lifecycle.side_effect = RuntimeError('NoSuchLifecycleConfiguration')
result = await import_activity.ensure_import_bundle_retention(base())
assert result['rules'] == 1
@mark.asyncio
async def test_the_retention_rule_can_be_applied_to_a_requested_bucket(
import_activity: ModelImport,
):
client = import_activity.import_minio_repository.minio_client
client.get_bucket_lifecycle.return_value = None
result = await import_activity.ensure_import_bundle_retention(base(bucket='other-bucket'))
assert result['bucket'] == 'other-bucket'
assert client.set_bucket_lifecycle.call_args.args[0] == 'other-bucket'
def test_the_model_document_shape_is_documented_in_one_place():
"""A guard against the shape drifting between the code and what the runtime reads."""
source = Path('laborious/activities/model_import.py').read_text()
for key in ('transform_flavor', 'predict_flavor', 'retention_minutes', 'target'):
assert f"'{key}'" in source
assert json.dumps({'transform_flavor': 'sklearn'}) # sanity: the flavors are plain strings

754
tests/laborious/activities/test_model_import_status.py Normal file
View File

@@ -0,0 +1,754 @@
"""The import log row: the claim, the hints, the names and the one authoritative write.
What these tests defend, in the order the workflow does it:
- the **state mapping** is total — every `ImportStep` has a status, so a new step cannot be added
without deciding what the record says when it fails;
- the **claim** is the only statement that checks ownership, and a claim that matches nothing writes
nothing anywhere and fails loudly everywhere else;
- every **later** write keys on the id alone, never rewrites a column the inserting side owns, and
never drops the workflow identifier out of the JSON detail;
- the **terminal** write is the single authority: one classification, `NULL` (never `''`) on success,
and a failure of its own that neither invents a failed import nor masks a real one.
"""
import json
from datetime import datetime
from unittest.mock import AsyncMock, MagicMock, patch
from pytest import fixture, mark, raises
from sqlalchemy.exc import DataError, IntegrityError, OperationalError, ProgrammingError
from laborious import metrics
from laborious.activities.model_import import (
IMPORT_LOG_TABLE,
IMPORT_STATUS_HINT_RETRY_POLICY,
IMPORT_STATUS_RETRY_POLICY,
RUN_NAME_MAX_LENGTH,
ExperimentRunStatus,
ImportState,
ModelImport,
build_import_detail,
derive_run_name,
map_state_to_status,
)
from laborious.activities.model_import_errors import (
ImportErrorCode,
ImportLogRowNotClaimableError,
ImportStatusRejectedError,
)
from laborious.utils.bundle.steps import ImportStep
from tests.helpers.status_engine import FakeStatusEngine
STATUS_CONFIG = {
'host': 'bff-db.internal',
'port': 5432,
'dbname': 'sientia-core-mlops-bff',
'user': 'mlops_bff_user',
'password': 'bff-password',
'schema': 'public',
}
IMPORT_CONFIG = {
'password_key': None,
'bucket': 'sientia',
'prefix': 'imported_models/',
'work_dir': '/tmp/sientia-import', # noqa: S108 - never written to in this module's tests
'max_object_bytes': 1024,
'max_entries': 10,
'max_uncompressed_bytes': 4096,
'max_compression_ratio': 200.0,
'models_collection': 'models',
'retention_days': 7,
}
METADATA = {
'model_id': '77',
'model_name': 'imported_model',
'workflow_name': 'import_model',
'schedule_name': '-',
}
# The columns the inserting side owns. No write in this module may ever assign one of them: the
# record is shared with the training flow and with whoever created the row.
BFF_OWNED_COLUMNS = frozenset(
{'username', 'request_data', 'bucket_name', 'file_name', 'run_type', 'created_at'}
)
# What the record's status must be for a failure at each step. Written out rather than derived, so a
# new step forces a decision here instead of silently inheriting one.
EXPECTED_ERROR_STATUS = {
ImportStep.RECEIVED: ExperimentRunStatus.ORCHESTRATOR_VALIDATION_ERROR,
ImportStep.DOWNLOAD: ExperimentRunStatus.ORCHESTRATOR_VALIDATION_ERROR,
ImportStep.DECRYPTION: ExperimentRunStatus.ORCHESTRATOR_VALIDATION_ERROR,
ImportStep.ARCHIVE_INSPECTION: ExperimentRunStatus.ORCHESTRATOR_VALIDATION_ERROR,
ImportStep.EXTRACTION: ExperimentRunStatus.ORCHESTRATOR_VALIDATION_ERROR,
ImportStep.STRUCTURE_VALIDATION: ExperimentRunStatus.ORCHESTRATOR_VALIDATION_ERROR,
ImportStep.CONTENT_POLICY: ExperimentRunStatus.ORCHESTRATOR_VALIDATION_ERROR,
ImportStep.EXPERIMENT_CREATION: ExperimentRunStatus.IMPORT_ERROR,
ImportStep.ARTIFACT_UPLOAD: ExperimentRunStatus.IMPORT_ERROR,
ImportStep.REGISTRATION: ExperimentRunStatus.IMPORT_ERROR,
ImportStep.MODEL_DOCUMENT: ExperimentRunStatus.IMPORT_ERROR,
ImportStep.CLEANUP: ExperimentRunStatus.IMPORT_ERROR,
}
def build_activity(engine: FakeStatusEngine) -> ModelImport:
"""A `ModelImport` whose only connection is the recording engine double."""
with patch('laborious.activities.model_import.create_engine', return_value=engine):
activity = ModelImport(
status_config=STATUS_CONFIG,
import_config=IMPORT_CONFIG,
minio_repository=MagicMock(),
mlflow_repository=MagicMock(),
mongo_repository=AsyncMock(),
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
)
activity.emit_metric = AsyncMock()
activity.send_notification_async = AsyncMock()
return activity
@fixture
def engine() -> FakeStatusEngine:
return FakeStatusEngine()
@fixture
def import_activity(engine: FakeStatusEngine) -> ModelImport:
return build_activity(engine)
def detail_of(statement) -> dict:
"""The `import` object a write put into `orchestrator_response_data`."""
return json.loads(statement.parameters['detail'])['import']
def claim_input(**overrides) -> dict:
return {
'metadata': METADATA,
'import_run_id': 77,
'workflow_id': 'import_model-77-abc',
**overrides,
}
# --------------------------------------------------------------------- the state mapping
def test_running_is_the_orchestrator_holding_the_row():
assert map_state_to_status(ImportState.RUNNING) == ExperimentRunStatus.ORCHESTRATOR_WAITING_PROC
# The step must not change a running row's status: the row is running whatever it is doing.
for step in ImportStep:
assert (
map_state_to_status(ImportState.RUNNING, step)
== ExperimentRunStatus.ORCHESTRATOR_WAITING_PROC
)
def test_completed_is_import_success():
assert map_state_to_status(ImportState.COMPLETED) == ExperimentRunStatus.IMPORT_SUCCESS
assert (
map_state_to_status(ImportState.COMPLETED, ImportStep.MODEL_DOCUMENT)
== ExperimentRunStatus.IMPORT_SUCCESS
)
@mark.parametrize('step', list(ImportStep))
def test_every_step_has_an_error_status(step: ImportStep):
assert map_state_to_status(ImportState.ERROR, step) == EXPECTED_ERROR_STATUS[step]
# The value form is what the workflow actually passes across the activity boundary.
assert map_state_to_status(ImportState.ERROR, step.value) == EXPECTED_ERROR_STATUS[step]
def test_the_error_mapping_covers_the_whole_enum():
"""A step added without a decision here fails this test rather than defaulting quietly."""
assert set(EXPECTED_ERROR_STATUS) == set(ImportStep)
def test_an_unclassified_error_is_a_platform_error():
assert map_state_to_status(ImportState.ERROR) == ExperimentRunStatus.IMPORT_ERROR
# ------------------------------------------------------------------------- the run name
def test_run_name_has_the_documented_shape():
assert derive_run_name('sales_forecast', '3') == 'import-sales_forecast-v3'
def test_run_name_truncates_inside_the_name_segment():
name = derive_run_name('a' * 120, '12')
assert name.startswith('import-')
assert name.endswith('-v12')
assert len(name) == RUN_NAME_MAX_LENGTH
def test_run_name_survives_a_version_that_leaves_no_room():
name = derive_run_name('model', 'v' * 80)
assert name.startswith('import-')
assert len(name) <= RUN_NAME_MAX_LENGTH
def test_run_name_is_deterministic():
assert derive_run_name('model', '4') == derive_run_name('model', '4')
def test_run_name_is_never_shorter_than_the_column_floor():
# `error_message` is not the only column with a length CHECK; the prefix is what guarantees it.
assert len(derive_run_name('', '')) >= 3
# --------------------------------------------------------------------- the JSON detail
def test_the_detail_always_carries_the_workflow_id():
detail = json.loads(build_import_detail(workflow_id='wf-1'))['import']
assert detail['workflow_id'] == 'wf-1'
def test_the_detail_omits_a_code_when_there_is_none():
detail = json.loads(build_import_detail(workflow_id='wf-1', step=ImportStep.RECEIVED))['import']
assert 'code' not in detail
assert detail['step'] == 'received'
def test_the_detail_records_a_cleanup_failure_without_a_code():
detail = json.loads(build_import_detail(workflow_id='wf-1', cleanup_failed=True))['import']
assert detail['cleanup_failed'] is True
assert 'code' not in detail
# ------------------------------------------------------------------------------ the claim
@mark.asyncio
async def test_claim_writes_the_running_status_and_the_workflow_id(
import_activity: ModelImport, engine: FakeStatusEngine
):
engine.rows = [{'run_type': 'IMPORT', 'bucket_name': 'sientia', 'file_name': None}]
result = await import_activity.claim_import_status(claim_input())
assert result['claimed'] is True
write = engine.writes[0]
assert write.table == IMPORT_LOG_TABLE
assert write.parameters['status'] == ExperimentRunStatus.ORCHESTRATOR_WAITING_PROC.value
assert detail_of(write) == {
'workflow_id': 'import_model-77-abc',
'gate': None,
'step': 'received',
}
@mark.asyncio
async def test_claim_is_the_only_statement_that_guards_ownership(
import_activity: ModelImport, engine: FakeStatusEngine
):
engine.rows = [{'run_type': 'IMPORT'}]
await import_activity.claim_import_status(claim_input())
sql = engine.writes[0].sql
assert "run_type = 'IMPORT'" in sql
assert "status = 'PENDING'" in sql
# The idempotency disjunct: a re-execution of the same workflow re-claims its own row instead of
# colliding with itself. It reads the identifier out of the JSON detail because the change adds
# no column for it.
assert "orchestrator_response_data -> 'import' ->> 'workflow_id' = :workflow_id" in sql
@mark.asyncio
async def test_claim_returns_the_values_the_input_guards_are_checked_against(
import_activity: ModelImport, engine: FakeStatusEngine
):
engine.rows = [
{
'run_type': 'IMPORT',
'bucket_name': 'sientia',
'file_name': 'imported_models/model.sientia',
'request_data': {'expected_digest': 'AB' * 32},
}
]
result = await import_activity.claim_import_status(claim_input())
assert result['bucket_name'] == 'sientia'
assert result['file_name'] == 'imported_models/model.sientia'
assert result['request_digest'] == 'ab' * 32
@mark.asyncio
@mark.parametrize('key', ['expected_digest', 'digest', 'sha256', 'file_digest'])
async def test_claim_finds_the_digest_wherever_the_bff_recorded_it(key: str):
engine = FakeStatusEngine(rows=[{'run_type': 'IMPORT', 'request_data': {key: 'f' * 64}}])
activity = build_activity(engine)
result = await activity.claim_import_status(claim_input())
assert result['request_digest'] == 'f' * 64
@mark.asyncio
@mark.parametrize('request_data', [None, 'not-a-dict', {}, {'expected_digest': ''}])
async def test_claim_treats_an_absent_digest_as_absent(request_data):
engine = FakeStatusEngine(rows=[{'run_type': 'IMPORT', 'request_data': request_data}])
activity = build_activity(engine)
result = await activity.claim_import_status(claim_input())
assert result['request_digest'] is None
@mark.asyncio
async def test_claim_reclaimed_by_the_same_workflow_succeeds(
import_activity: ModelImport, engine: FakeStatusEngine
):
engine.rows = [
{'run_type': 'IMPORT', 'status': 'ORCHESTRATOR_WAITING_PROC', 'claimed_by': 'wf-mine'}
]
result = await import_activity.claim_import_status(claim_input(workflow_id='wf-mine'))
assert result['claimed'] is True
@mark.asyncio
async def test_claim_opens_one_transaction_and_leaves_none_open(
import_activity: ModelImport, engine: FakeStatusEngine
):
engine.rows = [{'run_type': 'IMPORT'}]
await import_activity.claim_import_status(claim_input())
assert engine.transactions == 1
assert engine.open_scopes == 0
@mark.asyncio
async def test_the_claim_writes_before_it_reads(
import_activity: ModelImport, engine: FakeStatusEngine
):
"""A read-then-write claim would race two workflows into the same row; this one cannot."""
engine.rows = [{'run_type': 'IMPORT'}]
await import_activity.claim_import_status(claim_input())
assert engine.statements[0].is_write is True
assert engine.statements[1].verb == 'SELECT'
@mark.asyncio
@mark.parametrize(
('row', 'expected'),
[
(None, 'no such import log row'),
({'run_type': 'TRAIN', 'status': 'PENDING'}, 'is a TRAIN run'),
(
{
'run_type': 'IMPORT',
'status': 'ORCHESTRATOR_WAITING_PROC',
'claimed_by': 'another-workflow',
},
'already claimed by workflow another-workflow',
),
({'run_type': 'IMPORT', 'status': 'IMPORT_SUCCESS'}, 'in status IMPORT_SUCCESS'),
],
)
async def test_an_unclaimable_row_names_its_cause(row, expected: str):
engine = FakeStatusEngine(rowcounts=[0], rows=[row])
activity = build_activity(engine)
with raises(ImportLogRowNotClaimableError) as failure:
await activity.claim_import_status(claim_input())
assert expected in str(failure.value)
assert failure.value.import_run_id == 77
@mark.asyncio
async def test_an_unclaimable_row_is_not_retried():
engine = FakeStatusEngine(rowcounts=[0], rows=[None])
activity = build_activity(engine)
with raises(ImportLogRowNotClaimableError) as failure:
await activity.claim_import_status(claim_input())
assert failure.value.non_retryable is True
assert failure.value.type == 'ImportLogRowNotClaimableError'
# And the policy that governs the claim agrees, so the ten attempts are not spent on a verdict
# that cannot change.
assert 'ImportLogRowNotClaimableError' in IMPORT_STATUS_RETRY_POLICY.non_retryable_error_types
assert (
'ImportLogRowNotClaimableError' in IMPORT_STATUS_HINT_RETRY_POLICY.non_retryable_error_types
)
@mark.asyncio
async def test_an_unclaimable_row_is_never_written_to():
"""The only statement executed is the guarded claim that matched nothing, and no other write."""
engine = FakeStatusEngine(rowcounts=[0], rows=[{'run_type': 'TRAIN'}])
activity = build_activity(engine)
with raises(ImportLogRowNotClaimableError):
await activity.claim_import_status(claim_input())
assert len(engine.writes) == 1
assert engine.writes[0].sql == engine.statements[0].sql
assert "run_type = 'IMPORT'" in engine.writes[0].sql
assert [statement.verb for statement in engine.statements[1:]] == ['SELECT']
assert engine.tables == {IMPORT_LOG_TABLE}
@mark.asyncio
async def test_an_unclaimable_row_is_reported_through_every_sink():
engine = FakeStatusEngine(rowcounts=[0], rows=[None])
activity = build_activity(engine)
with raises(ImportLogRowNotClaimableError):
await activity.claim_import_status(claim_input())
logged = activity.logger.custom_error.call_args_list[0].args[0]
assert ImportErrorCode.LOG_ROW_NOT_CLAIMABLE.value in logged
assert 'not_claimable' in logged
metric = activity.emit_metric.await_args.kwargs
assert metric['metric_object'] is metrics.MODEL_IMPORT_ERROR_COUNT
assert metric['tags']['operation_type'] == ImportErrorCode.LOG_ROW_NOT_CLAIMABLE.value
notification = activity.send_notification_async.await_args.kwargs
assert notification['notification_id'] == ImportErrorCode.LOG_ROW_NOT_CLAIMABLE.value
assert notification['level'] == 'ERROR'
@mark.asyncio
async def test_a_claimed_row_counts_as_an_import_started(
import_activity: ModelImport, engine: FakeStatusEngine
):
engine.rows = [{'run_type': 'IMPORT'}]
await import_activity.claim_import_status(claim_input())
metric = import_activity.emit_metric.await_args.kwargs
assert metric['metric_object'] is metrics.MODEL_IMPORT_STARTED_COUNT
# ------------------------------------------------------------------- progress and names
@mark.asyncio
async def test_a_progress_write_keys_on_the_id_alone(
import_activity: ModelImport, engine: FakeStatusEngine
):
await import_activity.record_import_status({**claim_input(), 'step': ImportStep.DOWNLOAD.value})
write = engine.writes[0]
assert 'WHERE id = :import_run_id' in write.sql
assert 'run_type' not in write.sql
assert write.parameters['status'] == ExperimentRunStatus.ORCHESTRATOR_WAITING_PROC.value
assert detail_of(write)['step'] == 'download'
@mark.asyncio
async def test_a_lost_progress_write_is_not_a_failure(
import_activity: ModelImport, engine: FakeStatusEngine
):
engine.rowcounts = [0]
result = await import_activity.record_import_status(
{**claim_input(), 'step': ImportStep.DOWNLOAD.value}
)
assert result == {'recorded': False, 'rows': 0}
@mark.asyncio
async def test_the_names_write_touches_only_the_two_name_columns(
import_activity: ModelImport, engine: FakeStatusEngine
):
await import_activity.record_import_names(
{
**claim_input(),
'experiment_name': 'sales_forecast',
'run_name': 'import-sales_forecast-v3',
}
)
write = engine.writes[0]
assert write.assignments == {
'experiment_name',
'run_name',
'updated_at',
'orchestrator_response_data',
}
# Deliberately no `status`: the names are written mid-flight and must not disturb the verdict.
assert 'status' not in write.assignments
assert write.parameters['experiment_name'] == 'sales_forecast'
assert write.parameters['run_name'] == 'import-sales_forecast-v3'
@mark.asyncio
async def test_the_names_write_preserves_the_workflow_id(
import_activity: ModelImport, engine: FakeStatusEngine
):
await import_activity.record_import_names(
{**claim_input(), 'experiment_name': 'e', 'run_name': 'import-e-v1'}
)
assert detail_of(engine.writes[0])['workflow_id'] == 'import_model-77-abc'
# ------------------------------------------------------------------- the terminal write
@mark.asyncio
async def test_a_successful_import_writes_a_null_reason_and_no_code(
import_activity: ModelImport, engine: FakeStatusEngine
):
verdict = await import_activity.record_import_terminal_status(
{**claim_input(), 'succeeded': True, 'step': ImportStep.MODEL_DOCUMENT.value}
)
write = engine.writes[0]
assert write.parameters['status'] == ExperimentRunStatus.IMPORT_SUCCESS.value
# `''` would violate the column's `LENGTH(error_message) >= 3` CHECK.
assert write.parameters['error_message'] is None
assert 'code' not in detail_of(write)
assert verdict['status'] == ExperimentRunStatus.IMPORT_SUCCESS.value
assert verdict['code'] is None
@mark.asyncio
@mark.parametrize(
('step', 'status', 'code'),
[
(
ImportStep.ARCHIVE_INSPECTION,
ExperimentRunStatus.ORCHESTRATOR_VALIDATION_ERROR,
ImportErrorCode.BUNDLE_REJECTED,
),
(
ImportStep.STRUCTURE_VALIDATION,
ExperimentRunStatus.ORCHESTRATOR_VALIDATION_ERROR,
ImportErrorCode.BUNDLE_INCOMPLETE,
),
(
ImportStep.REGISTRATION,
ExperimentRunStatus.IMPORT_ERROR,
ImportErrorCode.MODEL_NOT_REGISTERED,
),
(
ImportStep.MODEL_DOCUMENT,
ExperimentRunStatus.IMPORT_ERROR,
ImportErrorCode.MODEL_SETTINGS_NOT_SAVED,
),
],
)
async def test_a_failure_is_recorded_with_its_step_code_and_reason(
step: ImportStep, status: ExperimentRunStatus, code: ImportErrorCode
):
engine = FakeStatusEngine()
activity = build_activity(engine)
verdict = await activity.record_import_terminal_status(
{**claim_input(), 'succeeded': False, 'step': step.value, 'gate': 4}
)
write = engine.writes[0]
assert write.parameters['status'] == status.value
assert write.parameters['error_message'] == verdict['reason']
detail = detail_of(write)
assert detail['step'] == step.value
assert detail['code'] == code.value
assert detail['gate'] == 4
@mark.asyncio
async def test_a_cleanup_failure_does_not_turn_a_success_into_an_error(
import_activity: ModelImport, engine: FakeStatusEngine
):
verdict = await import_activity.record_import_terminal_status(
{**claim_input(), 'succeeded': True, 'cleanup_failed': True}
)
write = engine.writes[0]
assert write.parameters['status'] == ExperimentRunStatus.IMPORT_SUCCESS.value
assert write.parameters['error_message'] is None
assert detail_of(write)['cleanup_failed'] is True
assert verdict['code'] is None
@mark.asyncio
async def test_the_terminal_write_returns_the_verdict_it_wrote(
import_activity: ModelImport,
):
verdict = await import_activity.record_import_terminal_status(
{**claim_input(), 'succeeded': False, 'step': ImportStep.DECRYPTION.value}
)
assert verdict == {
'status': ExperimentRunStatus.ORCHESTRATOR_VALIDATION_ERROR.value,
'code': ImportErrorCode.BUNDLE_CANNOT_BE_OPENED.value,
'reason': verdict['reason'],
'step': ImportStep.DECRYPTION.value,
}
assert 'password' in verdict['reason']
@mark.asyncio
async def test_executing_the_terminal_write_twice_leaves_the_row_the_same(
import_activity: ModelImport, engine: FakeStatusEngine
):
"""What retries must be idempotent: two executions of the same verdict, one row state."""
payload = {**claim_input(), 'succeeded': True, 'step': ImportStep.MODEL_DOCUMENT.value}
await import_activity.record_import_terminal_status(payload)
await import_activity.record_import_terminal_status(payload)
first, second = engine.writes
assert first.sql == second.sql
assert {k: v for k, v in first.parameters.items() if k != 'updated_at'} == {
k: v for k, v in second.parameters.items() if k != 'updated_at'
}
@mark.asyncio
async def test_every_write_advances_updated_at(
import_activity: ModelImport, engine: FakeStatusEngine
):
stamps = [datetime(2026, 8, 25, 12, 0, 0), datetime(2026, 8, 25, 12, 0, 5)]
with patch('laborious.activities.model_import.now', side_effect=stamps):
await import_activity.record_import_status(
{**claim_input(), 'step': ImportStep.DOWNLOAD.value}
)
await import_activity.record_import_terminal_status(
{**claim_input(), 'succeeded': True, 'step': ImportStep.MODEL_DOCUMENT.value}
)
assert engine.writes[0].parameters['updated_at'] < engine.writes[1].parameters['updated_at']
@mark.asyncio
async def test_no_status_write_touches_a_column_the_inserting_side_owns(
import_activity: ModelImport, engine: FakeStatusEngine
):
engine.rows = [{'run_type': 'IMPORT'}]
await import_activity.claim_import_status(claim_input())
await import_activity.record_import_status({**claim_input(), 'step': ImportStep.DOWNLOAD.value})
await import_activity.record_import_names(
{**claim_input(), 'experiment_name': 'e', 'run_name': 'import-e-v1'}
)
await import_activity.record_import_terminal_status({**claim_input(), 'succeeded': True})
assert len(engine.writes) == 4
for write in engine.writes:
assert write.assignments & BFF_OWNED_COLUMNS == set()
assert write.table == IMPORT_LOG_TABLE
# One transaction per write: no status write shares a transaction with anything else.
assert engine.transactions == 4
@mark.asyncio
async def test_every_post_claim_write_keys_on_the_id_alone(
import_activity: ModelImport, engine: FakeStatusEngine
):
"""Ownership was settled by the claim. Re-checking it later can only lose a real verdict."""
await import_activity.record_import_status({**claim_input(), 'step': ImportStep.DOWNLOAD.value})
await import_activity.record_import_names(
{**claim_input(), 'experiment_name': 'e', 'run_name': 'import-e-v1'}
)
await import_activity.record_import_terminal_status({**claim_input(), 'succeeded': True})
for write in engine.writes:
where = write.sql[write.sql.index('WHERE') :]
assert where == 'WHERE id = :import_run_id'
@mark.asyncio
async def test_a_row_edited_by_hand_after_the_claim_still_receives_the_verdict(
import_activity: ModelImport, engine: FakeStatusEngine
):
"""The predicate carries no status, so an operator's edit cannot orphan the record."""
engine.rowcounts = [1]
verdict = await import_activity.record_import_terminal_status(
{**claim_input(), 'succeeded': False, 'step': ImportStep.REGISTRATION.value}
)
assert verdict['status'] == ExperimentRunStatus.IMPORT_ERROR.value
assert 'status' not in engine.writes[0].sql[engine.writes[0].sql.index('WHERE') :]
@mark.asyncio
@mark.parametrize('error_class', [IntegrityError, ProgrammingError, DataError])
async def test_a_refused_statement_fails_on_the_first_attempt(error_class):
"""A missing `V12` is this: the connection worked and the statement was refused."""
engine = FakeStatusEngine(
write_errors=[error_class('UPDATE ...', {}, Exception('refused'))],
)
activity = build_activity(engine)
with raises(ImportStatusRejectedError) as failure:
await activity.record_import_terminal_status({**claim_input(), 'succeeded': True})
assert failure.value.non_retryable is True
assert failure.value.type == 'ImportStatusRejectedError'
assert 'ImportStatusRejectedError' in IMPORT_STATUS_RETRY_POLICY.non_retryable_error_types
@mark.asyncio
async def test_a_connection_failure_is_left_to_the_retry():
"""Repeating it is exactly what fixes it, so it must not be swallowed or reclassified."""
engine = FakeStatusEngine(write_errors=[OperationalError('UPDATE ...', {}, OSError('down'))])
activity = build_activity(engine)
with raises(OperationalError):
await activity.record_import_terminal_status({**claim_input(), 'succeeded': True})
@mark.asyncio
async def test_a_status_write_that_fails_then_succeeds_records_the_right_verdict():
engine = FakeStatusEngine(
write_errors=[OperationalError('UPDATE ...', {}, OSError('down')), None]
)
activity = build_activity(engine)
payload = {**claim_input(), 'succeeded': False, 'step': ImportStep.REGISTRATION.value}
with raises(OperationalError):
await activity.record_import_terminal_status(payload)
verdict = await activity.record_import_terminal_status(payload)
assert verdict['status'] == ExperimentRunStatus.IMPORT_ERROR.value
assert verdict['code'] == ImportErrorCode.MODEL_NOT_REGISTERED.value
assert engine.writes[-1].parameters['status'] == ExperimentRunStatus.IMPORT_ERROR.value
@mark.asyncio
async def test_a_verdict_that_cannot_be_written_is_reported_with_the_verdict():
engine = FakeStatusEngine()
activity = build_activity(engine)
result = await activity.report_import_status_write_failure(
{
**claim_input(),
'detail': 'OperationalError: connection refused',
'verdict': 'succeeded=True step=model_document gate=None',
}
)
assert result == {'reported': True}
# The one failure with no row to land on: nothing is written anywhere.
assert engine.statements == []
logged = activity.logger.custom_error.call_args_list[0].args[0]
assert ImportErrorCode.STATUS_NOT_RECORDED.value in logged
assert 'status_write' in logged
assert 'verdict_not_recorded=succeeded=True step=model_document' in logged
assert 'connection refused' in logged
@mark.asyncio
async def test_the_engine_is_disposed_on_shutdown(import_activity: ModelImport, engine):
import_activity.close()
assert engine.disposed is True

264
tests/laborious/test_import_absence.py Normal file
View File

@@ -0,0 +1,264 @@
"""What the import deliberately does **not** do.
Every absence here was a decision, and every one of them is the kind of thing a later well-meaning
edit adds back: a rollback for a failed import, a resume for a half-finished one, a signal to retry,
a `joblib.load` to "check the model is valid", a reason string built from the exception. None of
those are missing features — a failed import stays failed, and trying again is a new upload — so the
tests are written against the source itself, because there is no behaviour to observe when the
behaviour is the point of not existing.
"""
import ast
from pathlib import Path
from pytest import mark
from laborious.workflows.import_model import ImportModel
REPOSITORY_ROOT = Path(__file__).resolve().parents[2]
WORKFLOW_MODULE = REPOSITORY_ROOT / 'laborious/workflows/import_model.py'
ACTIVITIES_MODULE = REPOSITORY_ROOT / 'laborious/activities/model_import.py'
ERRORS_MODULE = REPOSITORY_ROOT / 'laborious/activities/model_import_errors.py'
READER_MODULE = REPOSITORY_ROOT / 'laborious/utils/bundle/reader.py'
FORMAT_MODULE = REPOSITORY_ROOT / 'laborious/utils/bundle/format.py'
IMPORT_PATH_MODULES = (
WORKFLOW_MODULE,
ACTIVITIES_MODULE,
ERRORS_MODULE,
READER_MODULE,
FORMAT_MODULE,
)
# The vocabulary of recovery. A partial import is not repaired; it is reported and abandoned.
RECOVERY_WORDS = ('rollback', 'compensate', 'resume', 'reconcile', 'undo')
# Loading a model is the *execution* side, owned by another card. The importer moves bytes and
# writes metadata; it never unpickles what it just accepted from a user.
MODEL_LOADING_CALLS = (
'mlflow.pyfunc.load_model',
'mlflow.sklearn.load_model',
'joblib.load',
'pickle.load',
'pickle.loads',
)
# Decryption lives in the activity that uses the password, never in the workflow, which would put
# the plaintext into the event history.
DECRYPTION_PRIMITIVES = (
'AESGCM',
'crypto_pwhash',
'crypto_secretstream',
'decrypt_password_envelope',
'load_envelope_key',
'IMPORT_PASSWORD_KEY',
'Argon2',
)
def parse(path: Path) -> ast.Module:
return ast.parse(path.read_text())
def called_names(tree: ast.Module) -> list[str]:
"""Every called name, as written: `a.b.c(...)` becomes `a.b.c`."""
names = []
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
continue
names.append(ast.unparse(node.func))
return names
def defined_names(tree: ast.Module) -> list[str]:
return [
node.name
for node in ast.walk(tree)
if isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef | ast.ClassDef)
]
# ------------------------------------------------------------- no retry beyond the record
def test_the_workflow_holds_no_recovery_routine():
tree = parse(WORKFLOW_MODULE)
for name in defined_names(tree) + called_names(tree):
for word in RECOVERY_WORDS:
assert word not in name.lower(), f'{name} looks like recovery'
def test_the_activities_hold_no_recovery_routine():
tree = parse(ACTIVITIES_MODULE)
for name in defined_names(tree) + called_names(tree):
for word in RECOVERY_WORDS:
assert word not in name.lower(), f'{name} looks like recovery'
def test_the_workflow_never_continues_as_new():
"""A single import is one workflow execution, start to finish."""
assert not [name for name in called_names(parse(WORKFLOW_MODULE)) if 'continue_as_new' in name]
# ------------------------------------------------------------------ no handlers, no waiting
def test_the_workflow_declares_no_signal_or_query_handler():
"""Nothing outside can steer an import once it starts: no retry signal, no progress query."""
for name in dir(ImportModel):
member = getattr(ImportModel, name, None)
for marker in (
'__temporal_signal_definition',
'__temporal_query_definition',
'__temporal_update_definition',
):
assert not hasattr(member, marker), f'{name} declares {marker}'
def test_the_workflow_module_carries_no_handler_decorator():
tree = parse(WORKFLOW_MODULE)
decorators = [
ast.unparse(decorator)
for node in ast.walk(tree)
if isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef)
for decorator in node.decorator_list
]
assert set(decorators) == {'workflow.run', 'staticmethod'}
assert decorators.count('workflow.run') == 1
def test_the_workflow_never_waits_for_a_condition():
"""No `wait_condition`: there is no external event an import waits on."""
assert not [name for name in called_names(parse(WORKFLOW_MODULE)) if 'wait_condition' in name]
# --------------------------------------------------------------------- nothing is loaded
@mark.parametrize('module_path', IMPORT_PATH_MODULES, ids=lambda path: path.name)
def test_no_module_in_the_import_path_loads_a_model(module_path: Path):
source = module_path.read_text()
for call in MODEL_LOADING_CALLS:
assert call not in source
def test_the_reader_knows_nothing_about_mlflow():
"""It stays liftable into a shared package: the reader is about the file, not the platform."""
source = READER_MODULE.read_text()
assert 'mlflow' not in source
assert 'temporalio' not in source
# ----------------------------------------------------------- the workflow holds no secret
@mark.parametrize('primitive', DECRYPTION_PRIMITIVES)
def test_the_workflow_module_holds_no_decryption_primitive(primitive: str):
assert primitive not in WORKFLOW_MODULE.read_text()
def test_the_workflow_only_ever_holds_the_envelope():
"""The envelope crosses the workflow; the password it wraps never does."""
source = WORKFLOW_MODULE.read_text()
assert 'password_envelope' in source
# The only other mention of a password is the input field the workflow refuses to accept.
assert source.count("'password'") == 1
def test_the_plaintext_password_field_is_refused_by_name():
from laborious.workflows.import_model import REJECTED_INPUT_FIELDS
assert 'password' in REJECTED_INPUT_FIELDS
# ------------------------------------------------------- every reason comes from the catalog
def _error_message_values(tree: ast.Module) -> list[ast.expr]:
"""Every expression written into the record's `error_message`."""
values = []
for node in ast.walk(tree):
if isinstance(node, ast.Dict):
for key, value in zip(node.keys, node.values, strict=True):
if isinstance(key, ast.Constant) and key.value == 'error_message':
values.append(value)
if isinstance(node, ast.keyword) and node.arg == 'error_message':
values.append(node.value)
return values
def test_no_recorded_reason_is_built_from_an_exception():
"""The catalog is the only source of a sentence a user reads."""
values = _error_message_values(parse(ACTIVITIES_MODULE))
assert values, 'no error_message write found — has the terminal statement moved?'
for value in values:
assert isinstance(value, ast.Name | ast.Constant), ast.unparse(value)
assert not isinstance(value, ast.JoinedStr | ast.BinOp)
def test_the_reason_written_is_the_classified_one():
values = _error_message_values(parse(ACTIVITIES_MODULE))
assert {ast.unparse(value) for value in values} == {'reason'}
def test_the_classification_ignores_the_exception_by_construction():
"""`classify_import_failure` deletes its exception argument; nothing can leak from it."""
source = ERRORS_MODULE.read_text()
assert 'del exception' in source
@mark.parametrize('module_path', (WORKFLOW_MODULE, ACTIVITIES_MODULE), ids=lambda p: p.name)
def test_no_status_detail_interpolates_an_exception(module_path: Path):
"""The JSON detail is built from a code and a step, both of them enum values."""
tree = parse(module_path)
for node in ast.walk(tree):
if not isinstance(node, ast.Call) or ast.unparse(node.func) != 'build_import_detail':
continue
for keyword in node.keywords:
rendered = ast.unparse(keyword.value)
assert 'error' not in rendered
assert 'exception' not in rendered
assert not isinstance(keyword.value, ast.JoinedStr)
# ------------------------------------------------------------------ no relational target
def test_the_import_writes_to_one_table_and_names_it_once():
"""D22: no project table, no model table, no second database. One `UPDATE`, one relation."""
statements = []
for node in ast.walk(parse(ACTIVITIES_MODULE)):
if not isinstance(node, ast.Call) or ast.unparse(node.func) != 'text':
continue
argument = node.args[0]
if isinstance(argument, ast.Constant):
statements.append(argument.value)
assert statements, 'no SQL statement found in the import activities'
for statement in statements:
words = statement.split()
relations = {
words[index + 1]
for index, word in enumerate(words)
if word.upper() in ('UPDATE', 'FROM', 'INTO', 'JOIN')
}
assert relations == {'public.experiment_run'}
def test_the_import_never_inserts_a_relational_row():
source = ACTIVITIES_MODULE.read_text().upper()
assert 'INSERT INTO' not in source
assert 'DELETE FROM' not in source
assert 'CREATE TABLE' not in source

100
tests/laborious/worker/test_runtime_task_queues.py
View File

@@ -1,10 +1,27 @@
"""Queue names and how many workers a runtime starts.
Every queue is runtime-scoped, so two runtimes never poll each other's work. The import worker is
the only conditional one: unset, the four default workers are exactly what they were, and no import
queue is polled at all.
"""
import ast
from pathlib import Path
from pytest import mark
from sientia_do.temporal.worker.prepare_worker import build_queue_name
from laborious.worker.worker import import_worker_enabled
from laborious.workflows.drift import Drift
from laborious.workflows.import_model import ImportModel
from laborious.workflows.minimal_retrain import MinimalRetrain
from laborious.workflows.predictions_batch import PredictionsBatch
from laborious.workflows.simple_metrics import SimpleMetrics
WORKER_MODULE = Path(__file__).resolve().parents[3] / 'laborious/worker/worker.py'
DEFAULT_WORKFLOWS = (PredictionsBatch, MinimalRetrain, Drift, SimpleMetrics)
def test_runtime_scoped_queue_names():
runtime = 'prod-a'
@@ -12,3 +29,86 @@ def test_runtime_scoped_queue_names():
assert build_queue_name(MinimalRetrain.__name__, runtime) == 'minimal_retrain-prod-a-queue'
assert build_queue_name(Drift.__name__, runtime) == 'drift-prod-a-queue'
assert build_queue_name(SimpleMetrics.__name__, runtime) == 'simple_metrics-prod-a-queue'
def test_the_import_queue_shares_the_runtime_suffix():
assert build_queue_name(ImportModel.__name__, 'prod-a') == 'import_model-prod-a-queue'
def test_every_queue_carries_the_same_runtime():
runtime = 'prod-a'
queues = [
build_queue_name(workflow.__name__, runtime)
for workflow in (*DEFAULT_WORKFLOWS, ImportModel)
]
assert all(queue.endswith(f'-{runtime}-queue') for queue in queues)
assert len(set(queues)) == 5
@mark.parametrize('value', ['1', 'true', 'TRUE', 'yes', 'on', ' true '])
def test_the_import_worker_is_registered_when_the_flag_says_so(monkeypatch, value: str):
monkeypatch.setenv('IMPORT_MODEL_ENABLED', value)
assert import_worker_enabled() is True
@mark.parametrize('value', ['', '0', 'false', 'no', 'off', 'maybe'])
def test_the_import_worker_stays_out_by_default(monkeypatch, value: str):
monkeypatch.setenv('IMPORT_MODEL_ENABLED', value)
assert import_worker_enabled() is False
def test_an_unset_flag_leaves_the_import_worker_out(monkeypatch):
monkeypatch.delenv('IMPORT_MODEL_ENABLED', raising=False)
assert import_worker_enabled() is False
def _prepare_worker_calls(tree: ast.Module) -> tuple[int, int]:
"""How many workers are built unconditionally, and how many behind the flag."""
conditional = 0
for node in ast.walk(tree):
if not isinstance(node, ast.If):
continue
if 'import_worker_enabled' not in ast.unparse(node.test):
continue
conditional += sum(
1
for inner in ast.walk(node)
if isinstance(inner, ast.Call) and ast.unparse(inner.func) == 'prepare_worker'
)
total = sum(
1
for node in ast.walk(tree)
if isinstance(node, ast.Call) and ast.unparse(node.func) == 'prepare_worker'
)
return total - conditional, conditional
def test_four_workers_by_default_and_five_with_the_flag():
unconditional, conditional = _prepare_worker_calls(ast.parse(WORKER_MODULE.read_text()))
assert unconditional == 4
assert conditional == 1
def test_the_import_worker_registers_only_the_import_activities():
"""The containment boundary: the import runtime polls one queue and holds one workflow."""
tree = ast.parse(WORKER_MODULE.read_text())
activities: list[str] = []
for node in ast.walk(tree):
if not isinstance(node, ast.If) or 'import_worker_enabled' not in ast.unparse(node.test):
continue
for inner in ast.walk(node):
if isinstance(inner, ast.keyword) and inner.arg == 'activities':
activities = [ast.unparse(element) for element in inner.value.elts]
if isinstance(inner, ast.keyword) and inner.arg == 'main_workflow':
assert ast.unparse(inner.value) == 'ImportModel'
if isinstance(inner, ast.keyword) and inner.arg == 'other_workflows':
assert ast.unparse(inner.value) == '[]'
assert len(activities) == 13
assert all(name.startswith('activities.') for name in activities)
assert all('import' in name for name in activities)

781
tests/laborious/workflows/test_import_model.py Normal file
View File

@@ -0,0 +1,781 @@
"""The `import_model` workflow: order, fail-fast, and which calls are allowed to retry.
The workflow itself holds no logic worth testing in isolation except the two things that can only be
wrong here: **the order** — the claim before anything else, the names before MLflow, the document
last, cleanup and then one terminal write in the `finally` — and **the retry split**, where a single
misplaced policy either replays a registry call or spends 151 seconds on a hint nobody reads.
Everything is asserted through the recorded activity calls, because that is exactly what the
workflow is: a sequence of activity invocations with policies attached.
"""
from datetime import timedelta
from types import SimpleNamespace
from unittest.mock import MagicMock, patch
from pytest import fixture, mark, raises
from sientia_do.temporal.policies import retry_policy
from laborious.activities.activities import Activities
from laborious.activities.model_import import (
IMPORT_STATUS_HINT_RETRY_POLICY,
IMPORT_STATUS_RETRY_POLICY,
)
from laborious.activities.model_import_errors import (
ImportGateRejectedError,
ImportInputError,
ImportLogRowNotClaimableError,
)
from laborious.utils.bundle.steps import ImportStep
from laborious.workflows.import_model import (
REJECTED_INPUT_FIELDS,
STATUS_WRITE_ENVELOPE,
STATUS_WRITE_TIMEOUT,
ImportModel,
)
WORKFLOW_ID = 'import_model-77-abc'
RUN_ID = '0123456789abcdef0123456789abcdef'
DIGEST = 'a' * 64
OBJECT_KEY = 'imported_models/factory_model_v2.sientia'
BUNDLE_METADATA = {
'model_name': 'factory_model',
'experiment_name': 'factory_experiment',
'run_id': 'aa11bb22cc33dd44ee55ff6677889900',
'model_version': '2',
'model_project': 'factory_project',
'export_timestamp': '2026-08-25T12:00:00+00:00',
}
BUNDLE_PARAMETERS = {'model_name': 'factory_model', 'target_variable': 'SE'}
# The activities that write the import log row. These four are the only calls allowed to retry.
STATUS_ACTIVITIES = (
Activities.claim_import_status,
Activities.record_import_status,
Activities.record_import_names,
Activities.record_import_terminal_status,
)
CLAIM_AND_TERMINAL = (Activities.claim_import_status, Activities.record_import_terminal_status)
HINTS = (Activities.record_import_status, Activities.record_import_names)
# What a correct import does, in order.
EXPECTED_ORDER = [
Activities.claim_import_status,
Activities.record_import_status,
Activities.ensure_import_bundle_retention,
Activities.download_import_bundle,
Activities.record_import_status,
Activities.open_import_bundle,
Activities.record_import_names,
Activities.record_import_status,
Activities.create_import_experiment,
Activities.record_import_status,
Activities.upload_import_artifacts,
Activities.record_import_status,
Activities.register_import_model_version,
Activities.record_import_status,
Activities.write_import_model_document,
Activities.cleanup_import_files,
Activities.record_import_terminal_status,
]
PROVISIONING = (
Activities.create_import_experiment,
Activities.upload_import_artifacts,
Activities.register_import_model_version,
Activities.write_import_model_document,
)
HAPPY_RESULTS = {
Activities.claim_import_status: {
'claimed': True,
'bucket_name': None,
'file_name': None,
'request_digest': None,
},
Activities.download_import_bundle: {
'path': '/work/bundle.sientia',
'size': 4096,
'digest': DIGEST,
},
Activities.open_import_bundle: {
'extracted_dir': '/work/extract',
'metadata': BUNDLE_METADATA,
'parameters': BUNDLE_PARAMETERS,
'digest': DIGEST,
},
Activities.create_import_experiment: {'experiment_id': '17'},
Activities.upload_import_artifacts: {'run_id': RUN_ID, 'run_name': 'import-factory_model-v2'},
Activities.register_import_model_version: {'model_name': 'factory_model', 'version': '4'},
Activities.write_import_model_document: {'written': True, 'id': '8'},
Activities.cleanup_import_files: {'cleaned': True, 'existed': True},
Activities.record_import_terminal_status: {'status': 'IMPORT_SUCCESS'},
}
class Recorder:
"""Stands in for `workflow.execute_activity_method`, recording every invocation."""
def __init__(self, failures: dict | None = None, results: dict | None = None) -> None:
self.results = {**HAPPY_RESULTS, **(results or {})}
self.failures = dict(failures or {})
self.calls: list[SimpleNamespace] = []
async def __call__(self, method, payload, **kwargs):
self.calls.append(SimpleNamespace(method=method, payload=payload, kwargs=kwargs))
if method in self.failures:
raise self.failures[method]
return self.results.get(method, {})
@property
def order(self) -> list:
return [call.method for call in self.calls]
def payload_of(self, method):
return next(call.payload for call in self.calls if call.method is method)
def last_payload_of(self, method):
return [call.payload for call in self.calls if call.method is method][-1]
def calls_to(self, method) -> list[SimpleNamespace]:
return [call for call in self.calls if call.method is method]
def valid_input(**overrides) -> dict:
return {
'import_run_id': 77,
'object_key': OBJECT_KEY,
'expected_digest': DIGEST,
'password_envelope': 'ZW52ZWxvcGU=',
**overrides,
}
@fixture
def import_workflow() -> ImportModel:
return ImportModel()
def run_with(recorder: Recorder):
"""Patch the workflow module's `workflow` object and hand back the patch context."""
patcher = patch('laborious.workflows.import_model.workflow')
workflow_mock = patcher.start()
workflow_mock.info.return_value = MagicMock(workflow_id=WORKFLOW_ID)
# Assigned directly rather than wrapped in an `AsyncMock`: a mock whose side effect is an
# instance with an `async def __call__` hands back the coroutine instead of awaiting it.
workflow_mock.execute_activity_method = recorder
return patcher, workflow_mock
@fixture
def recorder():
return Recorder()
@fixture
def workflow_env(recorder: Recorder):
patcher, workflow_mock = run_with(recorder)
yield SimpleNamespace(recorder=recorder, workflow=workflow_mock)
patcher.stop()
# ------------------------------------------------------------------------------- order
@mark.asyncio
async def test_a_successful_import_calls_its_activities_in_order(
import_workflow: ImportModel, workflow_env
):
result = await import_workflow.run(valid_input())
assert workflow_env.recorder.order == EXPECTED_ORDER
assert result == {
'import_run_id': 77,
'run_id': RUN_ID,
'model_name': 'factory_model',
'version': '4',
'document_id': '8',
}
@mark.asyncio
async def test_the_claim_is_the_first_thing_that_happens(
import_workflow: ImportModel, workflow_env
):
await import_workflow.run(valid_input())
assert workflow_env.recorder.order[0] is Activities.claim_import_status
@mark.asyncio
async def test_the_names_are_recorded_before_mlflow_is_touched(
import_workflow: ImportModel, workflow_env
):
"""So a person watching the import list sees which model is being imported, while it runs."""
await import_workflow.run(valid_input())
order = workflow_env.recorder.order
assert order.index(Activities.record_import_names) < order.index(
Activities.create_import_experiment
)
names = workflow_env.recorder.payload_of(Activities.record_import_names)
assert names['experiment_name'] == 'factory_experiment'
assert names['run_name'] == 'import-factory_model-v2'
@mark.asyncio
async def test_the_run_is_created_with_exactly_the_recorded_name(
import_workflow: ImportModel, workflow_env
):
await import_workflow.run(valid_input())
recorded = workflow_env.recorder.payload_of(Activities.record_import_names)['run_name']
used = workflow_env.recorder.payload_of(Activities.upload_import_artifacts)['run_name']
assert used == recorded
@mark.asyncio
async def test_the_model_document_is_the_last_thing_provisioned(
import_workflow: ImportModel, workflow_env
):
await import_workflow.run(valid_input())
order = workflow_env.recorder.order
for step in PROVISIONING[:-1]:
assert order.index(step) < order.index(Activities.write_import_model_document)
assert order[-2:] == [Activities.cleanup_import_files, Activities.record_import_terminal_status]
@mark.asyncio
async def test_the_document_target_comes_from_the_bundle_s_parameters(
import_workflow: ImportModel, workflow_env
):
await import_workflow.run(valid_input())
assert (
workflow_env.recorder.payload_of(Activities.write_import_model_document)['target'] == 'SE'
)
@mark.asyncio
async def test_the_terminal_write_reports_a_success_with_its_last_step(
import_workflow: ImportModel, workflow_env
):
await import_workflow.run(valid_input())
terminal = workflow_env.recorder.payload_of(Activities.record_import_terminal_status)
assert terminal['succeeded'] is True
assert terminal['step'] == ImportStep.MODEL_DOCUMENT.value
assert terminal['cleanup_failed'] is False
assert terminal['source'] == BUNDLE_METADATA
@mark.asyncio
async def test_the_download_is_handed_the_digest_the_request_carried(
import_workflow: ImportModel, workflow_env
):
await import_workflow.run(valid_input())
payload = workflow_env.recorder.payload_of(Activities.download_import_bundle)
assert payload['expected_digest'] == DIGEST
assert payload['object_key'] == OBJECT_KEY
# --------------------------------------------------------------------------- fail-fast
@mark.asyncio
@mark.parametrize(
('failing', 'step'),
[
(Activities.download_import_bundle, ImportStep.DOWNLOAD),
(Activities.open_import_bundle, ImportStep.DECRYPTION),
(Activities.create_import_experiment, ImportStep.EXPERIMENT_CREATION),
(Activities.upload_import_artifacts, ImportStep.ARTIFACT_UPLOAD),
(Activities.register_import_model_version, ImportStep.REGISTRATION),
(Activities.write_import_model_document, ImportStep.MODEL_DOCUMENT),
],
ids=lambda value: getattr(value, '__name__', str(value)),
)
async def test_a_failing_step_is_recorded_as_that_step_and_stops_the_import(
import_workflow: ImportModel, failing, step: ImportStep
):
recorder = Recorder(failures={failing: RuntimeError('dependency unavailable')})
patcher, _ = run_with(recorder)
try:
with raises(RuntimeError, match='dependency unavailable'):
await import_workflow.run(valid_input())
finally:
patcher.stop()
terminal = recorder.payload_of(Activities.record_import_terminal_status)
assert terminal['succeeded'] is False
assert terminal['step'] == step.value
# Nothing after the failing step was attempted, and cleanup still ran before the verdict.
order = recorder.order
assert order[-2:] == [Activities.cleanup_import_files, Activities.record_import_terminal_status]
later = EXPECTED_ORDER[EXPECTED_ORDER.index(failing) + 1 :]
for method in later:
if method in (Activities.cleanup_import_files, Activities.record_import_terminal_status):
continue
if method in STATUS_ACTIVITIES:
continue
assert method not in order
@mark.asyncio
async def test_nothing_is_provisioned_before_the_gates_pass(import_workflow: ImportModel):
recorder = Recorder(
failures={
Activities.open_import_bundle: ImportGateRejectedError(
'rejected', step=ImportStep.ARCHIVE_INSPECTION, gate=4
)
}
)
patcher, _ = run_with(recorder)
try:
with raises(ImportGateRejectedError):
await import_workflow.run(valid_input())
finally:
patcher.stop()
for method in PROVISIONING:
assert method not in recorder.order
@mark.asyncio
async def test_a_gate_rejection_is_recorded_with_the_gate_that_rejected_it(
import_workflow: ImportModel,
):
"""Gates 2 to 7 share one activity, so the step travels in the error's details, not in the
step the workflow was attempting."""
recorder = Recorder(
failures={
Activities.open_import_bundle: ImportGateRejectedError(
'rejected', step=ImportStep.CONTENT_POLICY, gate=7
)
}
)
patcher, _ = run_with(recorder)
try:
with raises(ImportGateRejectedError):
await import_workflow.run(valid_input())
finally:
patcher.stop()
terminal = recorder.payload_of(Activities.record_import_terminal_status)
assert terminal['step'] == ImportStep.CONTENT_POLICY.value
assert terminal['gate'] == 7
@mark.asyncio
async def test_an_ambiguous_decryption_failure_keeps_its_step(import_workflow: ImportModel):
recorder = Recorder(
failures={
Activities.open_import_bundle: ImportGateRejectedError(
'the bundle could not be opened', step=ImportStep.DECRYPTION, gate=3
)
}
)
patcher, _ = run_with(recorder)
try:
with raises(ImportGateRejectedError):
await import_workflow.run(valid_input())
finally:
patcher.stop()
terminal = recorder.payload_of(Activities.record_import_terminal_status)
assert terminal['step'] == ImportStep.DECRYPTION.value
assert terminal['gate'] == 3
# ------------------------------------------------------------------- the input contract
@mark.asyncio
@mark.parametrize('field', list(REJECTED_INPUT_FIELDS))
async def test_an_unknown_field_is_rejected_rather_than_ignored(
import_workflow: ImportModel, field: str
):
"""A caller that believes it recorded a project link must be told it did not."""
recorder = Recorder()
patcher, _ = run_with(recorder)
try:
with raises(ImportInputError, match='does not accept'):
await import_workflow.run(valid_input(**{field: 'whatever'}))
finally:
patcher.stop()
# After the claim, so the refusal lands on the record instead of only in the logs.
assert recorder.order[0] is Activities.claim_import_status
assert Activities.download_import_bundle not in recorder.order
terminal = recorder.payload_of(Activities.record_import_terminal_status)
assert terminal['step'] == ImportStep.RECEIVED.value
assert terminal['succeeded'] is False
@mark.asyncio
@mark.parametrize(
'digest',
[None, '', 'not-a-digest', 'a' * 63, 'a' * 65, 'g' * 64],
ids=['missing', 'empty', 'short-text', 'too-short', 'too-long', 'not-hex'],
)
async def test_a_missing_or_malformed_digest_fails_before_anything_is_downloaded(
import_workflow: ImportModel, digest
):
recorder = Recorder()
patcher, _ = run_with(recorder)
try:
with raises(ImportInputError):
await import_workflow.run(valid_input(expected_digest=digest))
finally:
patcher.stop()
assert Activities.download_import_bundle not in recorder.order
assert (
recorder.payload_of(Activities.record_import_terminal_status)['step']
== ImportStep.RECEIVED.value
)
@mark.asyncio
@mark.parametrize('missing', ['object_key', 'password_envelope'])
async def test_a_request_missing_a_required_field_fails_at_received(
import_workflow: ImportModel, missing: str
):
recorder = Recorder()
patcher, _ = run_with(recorder)
payload = valid_input()
payload.pop(missing)
try:
with raises(ImportInputError, match=missing):
await import_workflow.run(payload)
finally:
patcher.stop()
assert Activities.download_import_bundle not in recorder.order
@mark.asyncio
async def test_a_record_naming_another_object_is_a_received_failure(
import_workflow: ImportModel,
):
recorder = Recorder(
results={
Activities.claim_import_status: {
'claimed': True,
'file_name': 'imported_models/somebody_elses.sientia',
'request_digest': None,
}
}
)
patcher, _ = run_with(recorder)
try:
with raises(ImportInputError, match='different object'):
await import_workflow.run(valid_input())
finally:
patcher.stop()
assert Activities.download_import_bundle not in recorder.order
@mark.asyncio
async def test_a_record_naming_another_digest_is_a_received_failure(
import_workflow: ImportModel,
):
recorder = Recorder(
results={
Activities.claim_import_status: {
'claimed': True,
'file_name': OBJECT_KEY,
'request_digest': 'b' * 64,
}
}
)
patcher, _ = run_with(recorder)
try:
with raises(ImportInputError, match='different digest'):
await import_workflow.run(valid_input())
finally:
patcher.stop()
assert Activities.download_import_bundle not in recorder.order
@mark.asyncio
@mark.parametrize('value', [None, 0, -1, '77', 7.0, True], ids=lambda v: f'{type(v).__name__}-{v}')
async def test_an_unusable_record_id_fails_before_any_activity(import_workflow: ImportModel, value):
"""There is no record to claim and none to write: the run ends FAILED and nothing is touched."""
recorder = Recorder()
patcher, _ = run_with(recorder)
try:
with raises(ImportInputError, match='positive integer'):
await import_workflow.run(valid_input(import_run_id=value))
finally:
patcher.stop()
assert recorder.calls == []
@mark.asyncio
async def test_a_claim_that_matches_nothing_writes_nothing_at_all(import_workflow: ImportModel):
recorder = Recorder(
failures={
Activities.claim_import_status: ImportLogRowNotClaimableError(
'import log row 77 could not be claimed: no such import log row'
)
}
)
patcher, _ = run_with(recorder)
try:
with raises(ImportLogRowNotClaimableError):
await import_workflow.run(valid_input())
finally:
patcher.stop()
# The claim is outside the try/finally on purpose: no row is owned, so there is nothing to clean
# up and nothing to record.
assert recorder.order == [Activities.claim_import_status]
# ----------------------------------------------------------- failures that must not fail
@mark.asyncio
async def test_a_lost_hint_does_not_fail_a_working_import(import_workflow: ImportModel):
recorder = Recorder(failures={Activities.record_import_status: RuntimeError('db down')})
patcher, _ = run_with(recorder)
try:
result = await import_workflow.run(valid_input())
finally:
patcher.stop()
assert result['run_id'] == RUN_ID
assert Activities.report_import_status_write_failure in recorder.order
assert recorder.payload_of(Activities.record_import_terminal_status)['succeeded'] is True
@mark.asyncio
async def test_a_lost_names_write_does_not_fail_a_working_import(import_workflow: ImportModel):
recorder = Recorder(failures={Activities.record_import_names: RuntimeError('db down')})
patcher, _ = run_with(recorder)
try:
result = await import_workflow.run(valid_input())
finally:
patcher.stop()
assert result['version'] == '4'
assert Activities.upload_import_artifacts in recorder.order
@mark.asyncio
async def test_a_retention_rule_that_cannot_be_set_does_not_fail_an_import(
import_workflow: ImportModel,
):
recorder = Recorder(
failures={Activities.ensure_import_bundle_retention: RuntimeError('access denied')}
)
patcher, _ = run_with(recorder)
try:
result = await import_workflow.run(valid_input())
finally:
patcher.stop()
assert result['run_id'] == RUN_ID
@mark.asyncio
async def test_a_cleanup_failure_is_carried_into_the_verdict_not_raised(
import_workflow: ImportModel,
):
recorder = Recorder(failures={Activities.cleanup_import_files: RuntimeError('busy')})
patcher, _ = run_with(recorder)
try:
result = await import_workflow.run(valid_input())
finally:
patcher.stop()
terminal = recorder.payload_of(Activities.record_import_terminal_status)
assert result['run_id'] == RUN_ID
assert terminal['succeeded'] is True
assert terminal['cleanup_failed'] is True
@mark.asyncio
async def test_a_terminal_write_that_cannot_land_does_not_invent_a_failed_import(
import_workflow: ImportModel,
):
recorder = Recorder(
failures={Activities.record_import_terminal_status: RuntimeError('still down')}
)
patcher, _ = run_with(recorder)
try:
result = await import_workflow.run(valid_input())
finally:
patcher.stop()
assert result['run_id'] == RUN_ID
reported = recorder.payload_of(Activities.report_import_status_write_failure)
assert reported['verdict'] == 'succeeded=True step=model_document gate=None'
assert 'still down' in reported['detail']
@mark.asyncio
async def test_a_terminal_write_failure_does_not_mask_a_real_failure(
import_workflow: ImportModel,
):
recorder = Recorder(
failures={
Activities.register_import_model_version: RuntimeError('registry refused'),
Activities.record_import_terminal_status: RuntimeError('status db down'),
}
)
patcher, _ = run_with(recorder)
try:
with raises(RuntimeError, match='registry refused'):
await import_workflow.run(valid_input())
finally:
patcher.stop()
reported = recorder.payload_of(Activities.report_import_status_write_failure)
assert reported['verdict'] == 'succeeded=False step=registration gate=None'
@mark.asyncio
async def test_reporting_the_report_is_where_it_stops(import_workflow: ImportModel):
"""A failure in the failure channel cannot become the import's outcome."""
recorder = Recorder(
failures={
Activities.record_import_terminal_status: RuntimeError('down'),
Activities.report_import_status_write_failure: RuntimeError('also down'),
}
)
patcher, _ = run_with(recorder)
try:
result = await import_workflow.run(valid_input())
finally:
patcher.stop()
assert result['run_id'] == RUN_ID
# ------------------------------------------------------------------- the retry split
@mark.asyncio
async def test_only_the_status_writes_retry(import_workflow: ImportModel, workflow_env):
await import_workflow.run(valid_input())
for call in workflow_env.recorder.calls:
if call.method in STATUS_ACTIVITIES:
continue
assert call.kwargs['retry_policy'] is retry_policy
assert retry_policy.maximum_attempts == 1
@mark.asyncio
async def test_the_claim_and_the_terminal_write_use_the_long_envelope(
import_workflow: ImportModel, workflow_env
):
await import_workflow.run(valid_input())
for method in CLAIM_AND_TERMINAL:
for call in workflow_env.recorder.calls_to(method):
assert call.kwargs['retry_policy'] is IMPORT_STATUS_RETRY_POLICY
assert call.kwargs['start_to_close_timeout'] == STATUS_WRITE_TIMEOUT
assert call.kwargs['schedule_to_close_timeout'] == STATUS_WRITE_ENVELOPE
@mark.asyncio
async def test_the_hints_use_the_short_policy(import_workflow: ImportModel, workflow_env):
await import_workflow.run(valid_input())
for method in HINTS:
for call in workflow_env.recorder.calls_to(method):
assert call.kwargs['retry_policy'] is IMPORT_STATUS_HINT_RETRY_POLICY
# No long envelope: a hint that cannot land must not hold the import open.
assert 'schedule_to_close_timeout' not in call.kwargs
@mark.asyncio
async def test_no_provisioning_call_references_a_retrying_policy(
import_workflow: ImportModel, workflow_env
):
"""A repeated registry call or Mongo insert writes twice; that is why they cannot retry."""
await import_workflow.run(valid_input())
for call in workflow_env.recorder.calls:
if call.method in PROVISIONING or call.method is Activities.open_import_bundle:
assert call.kwargs['retry_policy'] not in (
IMPORT_STATUS_RETRY_POLICY,
IMPORT_STATUS_HINT_RETRY_POLICY,
)
@mark.asyncio
async def test_the_only_retrying_calls_are_the_four_status_writes(
import_workflow: ImportModel, workflow_env
):
await import_workflow.run(valid_input())
retrying = {
call.method
for call in workflow_env.recorder.calls
if call.kwargs['retry_policy']
in (IMPORT_STATUS_RETRY_POLICY, IMPORT_STATUS_HINT_RETRY_POLICY)
}
assert retrying == set(STATUS_ACTIVITIES)
def test_the_retry_policies_are_the_values_the_design_chose():
assert IMPORT_STATUS_RETRY_POLICY.maximum_attempts == 10
assert IMPORT_STATUS_RETRY_POLICY.initial_interval == timedelta(seconds=1)
assert IMPORT_STATUS_RETRY_POLICY.backoff_coefficient == 2.0
assert IMPORT_STATUS_RETRY_POLICY.maximum_interval == timedelta(seconds=30)
assert IMPORT_STATUS_HINT_RETRY_POLICY.maximum_attempts == 3
assert IMPORT_STATUS_HINT_RETRY_POLICY.maximum_interval == timedelta(seconds=2)
def test_the_terminal_write_s_envelope_outlasts_its_own_retries():
"""Set this smaller and Temporal cuts the retries short, silently undoing the retry split."""
backoff = sum(
min(
IMPORT_STATUS_RETRY_POLICY.initial_interval.total_seconds()
* IMPORT_STATUS_RETRY_POLICY.backoff_coefficient**attempt,
IMPORT_STATUS_RETRY_POLICY.maximum_interval.total_seconds(),
)
for attempt in range(IMPORT_STATUS_RETRY_POLICY.maximum_attempts - 1)
)
assert backoff == 151
assert STATUS_WRITE_ENVELOPE.total_seconds() > backoff
def test_a_hint_gives_up_inside_a_short_bound():
"""Three attempts capped at two seconds: a lost hint costs the import seconds, not minutes."""
backoff = sum(
min(
IMPORT_STATUS_HINT_RETRY_POLICY.initial_interval.total_seconds()
* IMPORT_STATUS_HINT_RETRY_POLICY.backoff_coefficient**attempt,
IMPORT_STATUS_HINT_RETRY_POLICY.maximum_interval.total_seconds(),
)
for attempt in range(IMPORT_STATUS_HINT_RETRY_POLICY.maximum_attempts - 1)
)
assert backoff <= 3
@mark.asyncio
async def test_no_retrying_call_carries_a_payload_that_would_insert(
import_workflow: ImportModel, workflow_env
):
"""What retries must be idempotent, so the payload may not name a row to create."""
await import_workflow.run(valid_input())
for call in workflow_env.recorder.calls:
if call.method not in STATUS_ACTIVITIES:
continue
assert call.payload['import_run_id'] == 77
for owned in ('username', 'request_data', 'bucket_name', 'file_name', 'run_type'):
assert owned not in call.payload

0
tests/scripts/__init__.py Normal file
View File

217
tests/scripts/test_import_model_manual_run.py Normal file
View File

@@ -0,0 +1,217 @@
"""The anti-rot guard for `scripts/import_model_manual_run.py`.
The script itself is only ever smoke-run by a person (task 13.6 of `import-model-e2e-suite`), so
nothing else would notice the three things most likely to be quietly broken in it:
1. the `# %%` cell layout — the whole contract is "cell N does step N", and a lost or reordered
marker turns a ten-step procedure into an unreadable one;
2. the boundary of design E15 — the script stands in for the frontend, so borrowing
`laborious.activities` or `laborious.workflows` would make it exercise the implementation instead
of the contract, and that is exactly the shortcut somebody reaches for when a cell needs one more
value;
3. the configuration rule — no host, user, password or key literal in the file, every target from
the environment, and every variable it reads documented in `.env.example`.
The wire shapes the script produces (the digest, the envelope, the object key, the row) are asserted
by the e2e suite against the real workflow, so they are deliberately not re-asserted here. This file
is about the script staying the *kind of thing* it is.
"""
from __future__ import annotations
import ast
import re
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
SCRIPT_PATH = REPO_ROOT / 'scripts' / 'import_model_manual_run.py'
ENV_EXAMPLE_PATH = REPO_ROOT / '.env.example'
CELL_MARKER = '# %%'
EXPECTED_CELLS = 10
FORBIDDEN_IMPORT_ROOTS = ('laborious.activities', 'laborious.workflows')
ALLOWED_LABORIOUS_IMPORT = 'laborious.utils.bundle.format'
# Substrings that would mean a target or a credential was written into the file instead of read
# from the environment. `mongodb://` and `postgresql` are schemes, not hosts, so they are not here.
FORBIDDEN_LITERAL_SUBSTRINGS = (
'localhost',
'127.0.0.1',
'0.0.0.0', # noqa: S104 - a denylist entry, not an interface this test binds
'::1',
'.svc.cluster.local',
'minioadmin',
'mlops_bff_user',
)
# A dotted host with a port (`db.example.com:5432`), an IPv4 address, or inline URL credentials.
HOSTLIKE_PATTERNS = (
re.compile(r'[A-Za-z0-9][A-Za-z0-9.-]*\.[A-Za-z]{2,}:\d{2,5}'),
re.compile(r'\b\d{1,3}(?:\.\d{1,3}){3}\b'),
re.compile(r'//[^/\s]+:[^/\s]+@'),
)
@pytest.fixture(scope='module')
def source() -> str:
"""The script's source text."""
return SCRIPT_PATH.read_text(encoding='utf-8')
@pytest.fixture(scope='module')
def tree(source: str) -> ast.Module:
"""The script parsed. A syntax error here is itself the finding — the module must be runnable."""
return ast.parse(source, filename=str(SCRIPT_PATH))
def _string_literals(tree: ast.Module) -> list[str]:
"""Every string constant in the module, comments and docstrings excluded."""
docstrings = set()
for node in ast.walk(tree):
if isinstance(node, ast.Module | ast.FunctionDef | ast.AsyncFunctionDef | ast.ClassDef):
body = getattr(node, 'body', [])
if (
body
and isinstance(body[0], ast.Expr)
and isinstance(body[0].value, ast.Constant)
and isinstance(body[0].value.value, str)
):
docstrings.add(id(body[0].value))
return [
node.value
for node in ast.walk(tree)
if isinstance(node, ast.Constant)
and isinstance(node.value, str)
and id(node) not in docstrings
]
def _env_names(tree: ast.Module) -> set[str]:
"""Every environment variable the script reads, by name."""
names: set[str] = set()
readers = {'env', 'env_required', 'getenv'}
for node in ast.walk(tree):
if not isinstance(node, ast.Call) or not node.args:
continue
target = node.func
called = (
target.attr
if isinstance(target, ast.Attribute)
else target.id
if isinstance(target, ast.Name)
else ''
)
first = node.args[0]
if called in readers and isinstance(first, ast.Constant) and isinstance(first.value, str):
names.add(first.value)
return names
class TestCellLayout:
"""The `# %%` contract of design E15."""
def test_script_exists(self) -> None:
assert SCRIPT_PATH.is_file(), f'{SCRIPT_PATH} is missing'
def test_every_cell_marker_is_present(self, source: str) -> None:
markers = [line for line in source.splitlines() if line.strip() == CELL_MARKER]
assert len(markers) == EXPECTED_CELLS, (
f'expected {EXPECTED_CELLS} `{CELL_MARKER}` cell markers, found {len(markers)}'
)
def test_cells_are_numbered_in_order(self, source: str) -> None:
"""Each cell opens with `# Cell N — ...`, and N counts 1..10 without a gap."""
lines = source.splitlines()
numbers: list[int] = []
for index, line in enumerate(lines):
if line.strip() != CELL_MARKER:
continue
following = lines[index + 1] if index + 1 < len(lines) else ''
match = re.match(r'#\s*Cell\s+(\d+)\b', following.strip())
assert match, f'the line after the marker on line {index + 1} does not name its cell'
numbers.append(int(match.group(1)))
assert numbers == list(range(1, EXPECTED_CELLS + 1))
def test_the_marker_precedes_the_first_statement(self, source: str) -> None:
"""The module docstring comes first, then cell 1 — nothing executable before a marker."""
body = source.split('"""', 2)[-1]
assert body.lstrip().startswith(CELL_MARKER)
class TestBoundary:
"""Design E15: the script stands in for the frontend, so it borrows no consumer code."""
def test_no_import_from_activities_or_workflows(self, tree: ast.Module) -> None:
for node in ast.walk(tree):
if isinstance(node, ast.Import):
for alias in node.names:
assert not alias.name.startswith(FORBIDDEN_IMPORT_ROOTS), (
f'the script imports {alias.name}'
)
elif isinstance(node, ast.ImportFrom) and node.module:
assert not node.module.startswith(FORBIDDEN_IMPORT_ROOTS), (
f'the script imports from {node.module}'
)
def test_the_only_laborious_import_is_the_format_helper(self, tree: ast.Module) -> None:
modules: set[str] = set()
for node in ast.walk(tree):
if isinstance(node, ast.Import):
modules.update(a.name for a in node.names if a.name.startswith('laborious'))
elif (
isinstance(node, ast.ImportFrom)
and node.module
and node.module.startswith('laborious')
):
modules.add(node.module)
assert modules <= {ALLOWED_LABORIOUS_IMPORT}, (
f'unexpected laborious imports: {sorted(modules - {ALLOWED_LABORIOUS_IMPORT})}'
)
class TestConfiguration:
"""No credential and no target literal in the file; every variable documented."""
@pytest.mark.parametrize('forbidden', FORBIDDEN_LITERAL_SUBSTRINGS)
def test_no_forbidden_substring(self, tree: ast.Module, forbidden: str) -> None:
offenders = [text for text in _string_literals(tree) if forbidden in text]
assert not offenders, f'{forbidden!r} appears in a string literal: {offenders}'
def test_no_hostlike_literal(self, tree: ast.Module) -> None:
for text in _string_literals(tree):
for pattern in HOSTLIKE_PATTERNS:
assert not pattern.search(text), (
f'{text!r} looks like a host, an address or inline credentials'
)
def test_every_variable_it_reads_is_documented(self, tree: ast.Module) -> None:
documented = {
line.split('=', 1)[0].strip()
for line in ENV_EXAMPLE_PATH.read_text(encoding='utf-8').splitlines()
if '=' in line and not line.lstrip().startswith('#')
}
undocumented = sorted(
_env_names(tree) - documented - {'MLFLOW_TRACKING_USERNAME', 'MLFLOW_TRACKING_PASSWORD'}
)
assert not undocumented, f'.env.example does not document: {undocumented}'
def test_it_reads_the_variables_the_contract_names(self, tree: ast.Module) -> None:
names = _env_names(tree)
required = {
'IMPORT_PASSWORD_KEY',
'IMPORT_BUNDLE_BUCKET',
'IMPORT_BUNDLE_PREFIX',
'IMPORT_STATUS_DB_HOST',
'IMPORT_STATUS_DB_PORT',
'IMPORT_STATUS_DB_NAME',
'IMPORT_STATUS_DB_USER',
'IMPORT_STATUS_DB_PASSWORD',
'IMPORT_SCRIPT_BUNDLE_PATH',
'IMPORT_SCRIPT_BUNDLE_PASSWORD',
'IMPORT_SCRIPT_USERNAME',
'RUNTIME',
}
assert required <= names, f'the script no longer reads: {sorted(required - names)}'