gitea_admin/sientia-dataops-laborious-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

QTZPOC-20

Browse Source 
QTZPOC-20: Implement model import workflow for .sientia bundles

Introduces the import_model workflow and its associated activities to enable importing .sientia bundles into MLflow and MongoDB. This includes:

*   New core workflow and activity logic for handling bundle download, decryption, validation, MLflow provisioning (experiments, runs, registered versions), and MongoDB model document creation.
*   Comprehensive E2E and unit test suites, including a dedicated test harness and a manual run script for development.
*   Updated .env.example and README.md with new configurations and usage instructions.
*   Ensures correct MLflow model source resolution for legacy loader compatibility and robust MongoDB document management.
*   Refactors existing utility scripts into a scripts/ directory.
This commit is contained in:
Vitor Pimentel dos Santos
2026-08-25 17:07:42 -03:00
parent 0e445de386
commit dac4f1718f
41 changed files with 8168 additions and 1045 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
e2e/fixtures/experiment_run.sql
View File

@@ -1,4 +1,4 @@
-- public.experiment_run as migrations V1 … V12 leave it, for the e2e Postgres container.
-- public.experiment_run as migrations V1 … V11 leave it, for the e2e Postgres container.
--
-- Source of truth: sientia-dataops-database-migrations,
-- postgres/sientia-core-mlops-bff/migrations/ — this file mirrors the *final state* of:
@@ -14,7 +14,11 @@
-- V9 remove_file_upload_statuses (status vocabulary)
-- V10 remove_unused_statuses (status vocabulary)
-- V11 add_run_type_to_experiment_run (run_type + reindexed)
-- V12 add_import_statuses_to_experiment_run (IMPORT_SUCCESS / IMPORT_ERROR + pairing)
--
-- V12 (add_import_statuses_to_experiment_run) lives in `experiment_run_v12.sql`, applied on top of
-- this file. It is separate because one e2e test — the evidence for why the migration gates
-- enabling the import worker — has to run against a table that does *not* have it, and a second
-- whole-schema copy with V12 removed would rot the moment either copy were edited.
--
-- The same drift trade already accepted for the bundle fixtures: this is a copy, and the tests
-- that read it are what make the copy auditable. If a migration lands there, update this file and
@@ -48,19 +52,13 @@ CREATE TABLE IF NOT EXISTS public.experiment_run (
CONSTRAINT chk_experiment_run_file_name_pattern
CHECK (file_name ~* '^[a-zA-Z0-9 /._-]+$'),
-- V12: the two shared stage values plus the training and import terminals.
-- The V1…V11 status vocabulary: the two shared stage values plus the training terminals.
-- V12 adds IMPORT_SUCCESS and IMPORT_ERROR to this list.
CONSTRAINT chk_experiment_run_status CHECK (status IN ('PENDING', 'ORCHESTRATOR_REQUEST_ERROR',
'ORCHESTRATOR_WAITING_PROC',
'ORCHESTRATOR_VALIDATION_ERROR',
'TRAINING_SUCCESS', 'TRAINING_ERROR',
'IMPORT_SUCCESS', 'IMPORT_ERROR')),
CONSTRAINT chk_experiment_run_run_type CHECK (run_type IN ('TRAIN', 'IMPORT')),
-- V12: each pair of terminal values belongs to its own run_type. The shared values stay out.
CONSTRAINT chk_experiment_run_terminal_status_run_type CHECK (
(run_type = 'IMPORT' OR status NOT IN ('IMPORT_SUCCESS', 'IMPORT_ERROR'))
AND (run_type = 'TRAIN' OR status NOT IN ('TRAINING_SUCCESS', 'TRAINING_ERROR'))
)
'TRAINING_SUCCESS', 'TRAINING_ERROR')),
CONSTRAINT chk_experiment_run_run_type CHECK (run_type IN ('TRAIN', 'IMPORT'))
);
CREATE INDEX IF NOT EXISTS idx_experiment_run_user_exp_status_date_type

36
e2e/fixtures/experiment_run_v12.sql Normal file
View File

@@ -0,0 +1,36 @@
-- V12 add_import_statuses_to_experiment_run, as a delta applied on top of `experiment_run.sql`.
--
-- Source of truth: sientia-dataops-database-migrations,
-- postgres/sientia-core-mlops-bff/migrations/V12__add_import_statuses_to_experiment_run.sql
--
-- Two things, and nothing else:
--
-- 1. the status vocabulary gains IMPORT_SUCCESS and IMPORT_ERROR;
-- 2. chk_experiment_run_terminal_status_run_type pairs each terminal value with its own
-- run_type, so a TRAIN row can never end IMPORT_SUCCESS and an IMPORT row can never end
-- TRAINING_ERROR. The two shared stage values (PENDING, ORCHESTRATOR_WAITING_PROC) stay out
-- of the pairing — both run types legitimately pass through them.
--
-- This file is applied separately because one e2e test runs an import against a table *without*
-- it: that failure shape — provisioned, then unrecordable, refused on the first attempt — is the
-- evidence for why this migration has to land before IMPORT_MODEL_ENABLED is turned on anywhere.
ALTER TABLE public.experiment_run
DROP CONSTRAINT IF EXISTS chk_experiment_run_status;
ALTER TABLE public.experiment_run
ADD CONSTRAINT chk_experiment_run_status CHECK (status IN ('PENDING',
'ORCHESTRATOR_REQUEST_ERROR',
'ORCHESTRATOR_WAITING_PROC',
'ORCHESTRATOR_VALIDATION_ERROR',
'TRAINING_SUCCESS', 'TRAINING_ERROR',
'IMPORT_SUCCESS', 'IMPORT_ERROR'));
ALTER TABLE public.experiment_run
DROP CONSTRAINT IF EXISTS chk_experiment_run_terminal_status_run_type;
ALTER TABLE public.experiment_run
ADD CONSTRAINT chk_experiment_run_terminal_status_run_type CHECK (
(run_type = 'IMPORT' OR status NOT IN ('IMPORT_SUCCESS', 'IMPORT_ERROR'))
AND (run_type = 'TRAIN' OR status NOT IN ('TRAINING_SUCCESS', 'TRAINING_ERROR'))
);