gitea_admin/sientia-dataops-laborious-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

QTZPOC-20

Browse Source 
QTZPOC-20: Implement model import workflow for .sientia bundles

Introduces the import_model workflow and its associated activities to enable importing .sientia bundles into MLflow and MongoDB. This includes:

*   New core workflow and activity logic for handling bundle download, decryption, validation, MLflow provisioning (experiments, runs, registered versions), and MongoDB model document creation.
*   Comprehensive E2E and unit test suites, including a dedicated test harness and a manual run script for development.
*   Updated .env.example and README.md with new configurations and usage instructions.
*   Ensures correct MLflow model source resolution for legacy loader compatibility and robust MongoDB document management.
*   Refactors existing utility scripts into a scripts/ directory.
This commit is contained in:
Vitor Pimentel dos Santos
2026-08-25 17:07:42 -03:00
parent 0e445de386
commit dac4f1718f
41 changed files with 8168 additions and 1045 deletions
Download Patch File Download Diff File Expand all files Collapse all files

150
README.md
View File

@@ -21,6 +21,7 @@ A comprehensive, Temporal-based ML orchestration system for industrial data proc
- [Minimal Retrain Workflow](#4-minimal-retrain-workflow-minimal_retrainpy)
- [Drift Workflow](#5-drift-workflow-driftpy)
- [Simple Metrics Workflow](#6-simple-metrics-workflow-simple_metricspy)
- [Import Model Workflow](#7-import-model-workflow-import_modelpy)
- [Installation & Setup](#installation--setup)
- [Prerequisites](#prerequisites)
- [Environment Setup](#environment-setup)
@@ -31,6 +32,7 @@ A comprehensive, Temporal-based ML orchestration system for industrial data proc
- [Running Tests and Coverage](#running-tests-and-coverage)
- [Manual Test Execution](#manual-test-execution)
- [Manual Application Execution](#manual-application-execution)
- [Running One Model Import by Hand](#running-one-model-import-by-hand)
- [Code Quality & Validation](#code-quality--validation)
- [Overview](#overview)
- [Validation Tools](#validation-tools)
@@ -126,7 +128,8 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
### Key Components
#### **Worker (`laborious/worker/worker.py`)**
- Temporal client setup, four workers via `sientia_do.temporal.worker.prepare_worker`
- Temporal client setup, four workers via `sientia_do.temporal.worker.prepare_worker`, plus a
fifth registered only when `IMPORT_MODEL_ENABLED` is truthy (`1`, `true`, `yes`, `on`)
- Runtime-scoped task queues: `{workflow}-{RUNTIME}-queue` for all workflows
- Metrics server initialization, notification handler setup
- Graceful shutdown and autoscaling-friendly behavior
@@ -135,6 +138,16 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
`simple_metrics-queue`. Use `drift-{RUNTIME}-queue` and `simple_metrics-{RUNTIME}-queue`
matching the worker pod `RUNTIME` env (same as `predictions_batch` / `minimal_retrain`).
Task queues, one per worker:
| Queue | Worker | Registered |
|---|---|---|
| `predictions_batch-{RUNTIME}-queue` | PredictionsBatch (+ its two sub-workflows) | always |
| `minimal_retrain-{RUNTIME}-queue` | MinimalRetrain | always |
| `drift-{RUNTIME}-queue` | Drift | always |
| `simple_metrics-{RUNTIME}-queue` | SimpleMetrics | always |
| `import_model-{RUNTIME}-queue` | ImportModel | only when `IMPORT_MODEL_ENABLED` is truthy |
#### **Workflows (`laborious/workflows/`)**
- `predictions_batch.py`: Batch prediction entry point
- `sub_workflows/prediction_process.py`: Core prediction pipeline
@@ -142,6 +155,8 @@ matching the worker pod `RUNTIME` env (same as `predictions_batch` / `minimal_re
- `minimal_retrain.py`: Automated model retraining and production update
- `drift.py`: Data drift detection and monitoring
- `simple_metrics.py`: Regression metrics calculation (RMSE, MSE, MAE, R²)
- `import_model.py`: `.sientia` bundle import — registered model + model document, behind
`IMPORT_MODEL_ENABLED`
#### **Activities (`laborious/activities/`)**
- `gates.py`: Data quality validation, filtering, and data formatting operations
@@ -509,6 +524,55 @@ flowchart LR
}
```
### 7. Import Model Workflow (`import_model.py`)
The **ImportModel** workflow turns an encrypted `.sientia` bundle already uploaded to MinIO into a
registered MLflow model plus one document in the MongoDB `models` collection — or into a durable
`ERROR` row carrying a sentence a person can read. It is registered **only** when
`IMPORT_MODEL_ENABLED` is set, so the importer can live in its own runtime with its own credentials.
Full contract for the frontend and the BFF: [`docs/model-import.md`](docs/model-import.md).
The file format: [`docs/sientia-bundle-format.md`](docs/sientia-bundle-format.md).
#### Execution Flow
1. **Claim**: takes ownership of the `public.experiment_run` row the BFF inserted (it never inserts one)
2. **Download**: object limits, then the mandatory `expected_digest` comparison
3. **Open**: seven ordered gates — header, decryption, archive inspection, extraction, structure, content policy
4. **Provision**: experiment, a new run with the recorded name, the artifact tree, a registered version (stage untouched)
5. **List**: the MongoDB model document, `active: false`
6. **Finish**: cleanup, then one terminal write classifying the outcome once
#### Architecture Diagram
```mermaid
flowchart LR
A[1. claim_import_status] --> B[2. download_import_bundle] --> C[3. open_import_bundle]
C --> D[4. create_import_experiment] --> E[5. upload_import_artifacts]
E --> F[6. register_import_model_version] --> G[7. write_import_model_document]
G --> H[8. cleanup_import_files] --> I[9. record_import_terminal_status]
A -.-> BFF[(sientia-core-mlops-bff)]
B -.-> MinIO[(MinIO)]
F -.-> MLflow[(MLflow)]
G -.-> Mongo[(MongoDB)]
I -.-> BFF
```
#### Input Parameters
```json
{
"import_run_id": 41,
"object_key": "imported_models/sales_forecast_v3.sientia",
"expected_digest": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
"password_envelope": "Base64OfNonceCiphertextTag==",
"bucket": "sientia"
}
```
A successful import leaves the registered version **unstaged** and the model document **inactive**:
promoting and activating are manual operator steps (QTZPOC-21). The `V12` migration of
`sientia-core-mlops-bff` must be applied before enabling the worker — see
[`docs/model-import.md`](docs/model-import.md) § 7.
## 📋 Prerequisites
- Python 3.11+
@@ -589,10 +653,10 @@ Use the provided script to run the application locally:
```bash
# Make script executable (first time only)
chmod +x run_local.sh
chmod +x scripts/run_local.sh
# Run the application
./run_local.sh
# Run the application (from the repository root)
./scripts/run_local.sh
```
The script will:
@@ -606,10 +670,10 @@ Use the provided script to run tests with coverage:
```bash
# Make script executable (first time only)
chmod +x run_coverage.sh
chmod +x scripts/run_coverage.sh
# Run tests with coverage
./run_coverage.sh
# Run tests with coverage (from the repository root)
./scripts/run_coverage.sh
```
The script will:
@@ -654,6 +718,62 @@ fi
python -m laborious.worker.worker
```
### Running One Model Import by Hand
[`scripts/import_model_manual_run.py`](scripts/import_model_manual_run.py) stands in for the frontend
that does not exist yet. The Angular screen plus the Spring Boot BFF will, between them, hash a
`.sientia` file, wrap the bundle password, upload the object and insert one `PENDING` row before
starting `import_model`; this script does exactly those four things against whatever `.env` points
at, then reads the outcome back and can undo itself.
It is a **development tool**. It writes to whatever `.env` names — a real object in a real bucket and
a real row in a real `public.experiment_run` — and a `.env` copied from a cluster deployment can
perfectly well name production. It is not for production use.
**Prerequisites**
- A reachable environment: the BFF Postgres, MinIO, MongoDB, MLflow and Temporal that `.env` names.
- The `V12` migration applied to `public.experiment_run` in the BFF database. Without it the import
provisions the model and then cannot record the outcome — see
[`docs/model-import.md`](docs/model-import.md) § 7.
- A worker running with `IMPORT_MODEL_ENABLED` truthy and the same `RUNTIME`, if you want cell 8 (or
the Temporal UI) to actually execute the run.
- The import variables from [`.env.example`](.env.example) filled in, including `IMPORT_PASSWORD_KEY`
and the script's own `IMPORT_SCRIPT_BUNDLE_PATH`, `IMPORT_SCRIPT_BUNDLE_PASSWORD` and
`IMPORT_SCRIPT_USERNAME`.
**Running it.** The file is split into `# %%` cells: run them one at a time in the VS Code
interactive window, or run the module top to bottom.
```bash
source ./venv/bin/activate
# top to bottom (cells 5 and 6 write; cell 2 must pass first)
python scripts/import_model_manual_run.py
```
| Cell | Does | Writes? |
|---|---|---|
| 1 | Resolves and prints every target — Postgres, MinIO, Mongo, MLflow, Temporal, the task queue | no |
| 2 | **Confirmation guard** | no |
| 3 | Picks the bundle, computes `expected_digest`, prints the parsed header | no |
| 4 | Builds the AES-256-GCM password envelope | no |
| 5 | Uploads the object, re-reads it and checks the digest still matches | **yes** |
| 6 | Inserts the `PENDING` / `run_type='IMPORT'` row, prints `import_run_id` | **yes** |
| 7 | Prints the workflow input as JSON, ready for the Temporal UI | no |
| 8 | Optionally starts `import_model` and waits (`START_THE_WORKFLOW`) | starts a run |
| 9 | Reads the outcome back: the row, the MLflow experiment/run/version, the model document | no |
| 10 | Undo: deletes the object and the row always; the provisioning behind `DELETE_PROVISIONING` | deletes |
**The guard.** Cells 5 and 6 are the only cells that write, and neither runs until cell 2 passes.
Cell 2 compares a confirmation value against the Postgres host cell 1 resolved: set
`IMPORT_SCRIPT_CONFIRM_TARGET` in `.env`, or paste the host into the cell. It is a read-then-restate
rather than a boolean on purpose — a flag can be flipped without ever looking at what it points at.
No credential is written in the script and none is printed: it prints hosts, ports, database, bucket
and object names, the row id and the workflow input, and never the bundle password, the envelope key
or the envelope's plaintext.
## Code Quality & Validation
### Overview
@@ -819,6 +939,22 @@ See [OPC UA Communication](#opc-ua-communication) for semantics, concurrency, an
| `MLFLOW_PORT` | MLFlow server port | `5080` | Yes |
| `MLFLOW_USERNAME` | MLFlow username | `aignosi` | Yes |
| `MLFLOW_PASSWORD` | MLFlow password | `aignosi` | Yes |
| `IMPORT_MODEL_ENABLED` | Register the import worker on `import_model-{RUNTIME}-queue` | _(unset)_ | No |
| `IMPORT_PASSWORD_KEY` | base64 of the 32-byte AES-256-GCM key wrapping bundle passwords | _(none)_ | If importing |
| `IMPORT_BUNDLE_BUCKET` | Bucket holding uploaded `.sientia` objects | `sientia` | No |
| `IMPORT_BUNDLE_PREFIX` | Key prefix the uploads must sit under | `imported_models/` | No |
| `IMPORT_BUNDLE_RETENTION_DAYS` | Lifecycle expiry applied to that prefix | `7` | No |
| `IMPORT_WORK_DIR` | Worker scratch directory for bundles | `<tempdir>/sientia-import` | No |
| `IMPORT_MAX_OBJECT_BYTES` | Gate 1 object size ceiling | `1073741824` | No |
| `IMPORT_MAX_ARCHIVE_ENTRIES` | Gate 4 entry count ceiling | `5000` | No |
| `IMPORT_MAX_UNCOMPRESSED_BYTES` | Gate 4 uncompressed size ceiling | `4294967296` | No |
| `IMPORT_MAX_COMPRESSION_RATIO` | Gate 4 compression ratio ceiling | `200` | No |
| `IMPORT_MODELS_COLLECTION` | MongoDB collection holding the model listing | `models` | No |
| `IMPORT_STATUS_DB_HOST` | Import log database host (the BFF's) | `localhost` | If importing |
| `IMPORT_STATUS_DB_PORT` | Import log database port | `5432` | No |
| `IMPORT_STATUS_DB_NAME` | Import log database name | `sientia-core-mlops-bff` | No |
| `IMPORT_STATUS_DB_USER` | Import log database user | `mlops_bff_user` | If importing |
| `IMPORT_STATUS_DB_PASSWORD` | Import log database password | _(empty)_ | If importing |
| `OPC_CONFIG` | OPC server configuration (JSON) | `{}` | No |
| `OPC_ID` | OPC server identifier | `1` | No |
| `OPC_URL` | OPC server URL | `opc.tcp://localhost:4840` | No |