gitea_admin/sientia-dataops-laborious-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

QTZPOC-20

Browse Source 
QTZPOC-20: Implement explicit joblib model flavor and enhance model import workflow with extensive E2E tests.

This commit introduces a first-class joblib model flavor for direct loading and saving of Python pickle artifacts, including custom code within the artifact's code/ directory, bypassing MLflow's flavor loaders. The model import workflow's status database configuration is now streamlined to default to the POSTGRES_* server, with explicit IMPORT_STATUS_DB_* overrides.

Comprehensive end-to-end tests have been added for the model import workflow, covering success paths, gate rejections, digest enforcement, provisioning failures, status record integrity, cleanup, and retry/recovery scenarios using a real-clock environment. New E2
This commit is contained in:
Vitor Pimentel dos Santos
2026-08-26 15:45:33 -03:00
parent dac4f1718f
commit e95e65b72b
32 changed files with 4247 additions and 1040 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
laborious/utils/connectors_config.py
View File

@@ -136,27 +136,36 @@ def build_import_config() -> dict[str, Any]:
def build_import_status_config() -> dict[str, Any]:
"""
Build the connection to the import log database — a **second** Postgres connection.
Build the connection to the import log database — the POSTGRES_* server, another database.
The import log is `public.experiment_run` in the database owned by the Spring Boot BFF
(`sientia-core-mlops-bff`), not the `sientia` database Laborious itself uses. The table and
schema are constants of the code, never inputs; only the connection is configuration.
(`sientia-core-mlops-bff`), which sits on the same Postgres server as the `sientia` database
Laborious itself uses. Only the database name differs, so only the database name has a variable
of its own; host, port, user and password come from POSTGRES_*. The table and schema are
constants of the code, never inputs.
The four IMPORT_STATUS_DB_* connection variables remain as overrides, for a deployment where the
log genuinely lives on another server — the e2e suite is exactly that, two containers.
Environment Variables:
IMPORT_STATUS_DB_HOST: host (default: localhost)
IMPORT_STATUS_DB_PORT: port (default: 5432)
IMPORT_STATUS_DB_NAME: database (default: sientia-core-mlops-bff)
IMPORT_STATUS_DB_USER: user (default: mlops_bff_user)
IMPORT_STATUS_DB_PASSWORD: password (default: empty)
POSTGRES_HOST: server host, unless IMPORT_STATUS_DB_HOST overrides it (default: localhost)
POSTGRES_PORT: server port, unless IMPORT_STATUS_DB_PORT overrides it (default: 5432)
POSTGRES_USER: user, unless IMPORT_STATUS_DB_USER overrides it (default: sientia)
POSTGRES_PASSWORD: password, unless IMPORT_STATUS_DB_PASSWORD overrides it (default: empty)
Returns:
dict: import status database configuration dictionary
"""
# Written as `override or fallback or default` rather than
# `getenv(override) or getenv(fallback, default)`: inside an `or`, the two-argument `getenv`
# widens to `str | None` for mypy, so the `int(...)` below fails to type-check. Putting the
# default last as its own term keeps every value a plain `str` and reads in precedence order.
return {
'host': getenv('IMPORT_STATUS_DB_HOST', 'localhost'),
'port': int(getenv('IMPORT_STATUS_DB_PORT', '5432')),
'dbname': getenv('IMPORT_STATUS_DB_NAME', 'sientia-core-mlops-bff'),
'user': getenv('IMPORT_STATUS_DB_USER', 'mlops_bff_user'),
'password': getenv('IMPORT_STATUS_DB_PASSWORD', ''),
'host': getenv('IMPORT_STATUS_DB_HOST') or getenv('POSTGRES_HOST') or 'localhost',
'port': int(getenv('IMPORT_STATUS_DB_PORT') or getenv('POSTGRES_PORT') or '5432'),
'dbname': getenv('IMPORT_STATUS_DB_NAME') or 'sientia-core-mlops-bff',
'user': getenv('IMPORT_STATUS_DB_USER') or getenv('POSTGRES_USER') or 'sientia',
'password': getenv('IMPORT_STATUS_DB_PASSWORD') or getenv('POSTGRES_PASSWORD') or '',
'schema': 'public',
}