gitea_admin/sientia-dataops-laborious-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

QTZPOC-20

Browse Source 
QTZPOC-20: Refactor model import workflow, introduce atomic model ID generation, and clarify MinIO expiry.

Fuses bundle download, decryption, MLflow experiment creation, and artifact upload into a single activity to prevent cross-worker issues. Implements an atomic counter for MongoDB model IDs, replacing the max(id) + 1 scan to avoid collisions during concurrent imports. Adds comprehensive documentation for MinIO object expiry mechanisms and refines lifecycle rule application. Improves error handling with clearer responsibility classification and centralizes MLflow repository injection.
This commit is contained in:
Vitor Pimentel dos Santos
2026-09-08 17:06:25 -03:00
parent e4fc12e529
commit 192834bde3
34 changed files with 1485 additions and 531 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
laborious/utils/connectors_config.py
View File

@@ -112,6 +112,13 @@ def build_import_config() -> dict[str, Any]:
IMPORT_MAX_UNCOMPRESSED_BYTES: gate 4 uncompressed size ceiling (default: 4 GiB)
IMPORT_MAX_COMPRESSION_RATIO: gate 4 compression ratio ceiling (default: 200)
IMPORT_MODELS_COLLECTION: MongoDB collection holding the model listing (default: models)
IMPORT_MODEL_ID_COUNTERS_COLLECTION: MongoDB collection holding the atomic id counters that
assign `models` document ids (default: model_id_counters). **External dependency**: this
collection and its seed value (`seq = max(int(id) for id in models)`) are not created by
this repository — a migration must add them to
`sientia-dataops-database-migrations/mongodb/migrations/`, following the `models`
baseline's `ensureCollection`/`ensureIndexes` convention, before any environment's counter
is first incremented. See `openspec/changes/model-import-fix-bugs/design.md`, Decision 3.
IMPORT_BUNDLE_RETENTION_DAYS: lifecycle expiry for the uploaded bundle (default: 7)
Returns:
@@ -130,6 +137,7 @@ def build_import_config() -> dict[str, Any]:
),
'max_compression_ratio': float(getenv('IMPORT_MAX_COMPRESSION_RATIO', '200')),
'models_collection': getenv('IMPORT_MODELS_COLLECTION', 'models'),
'counters_collection': getenv('IMPORT_MODEL_ID_COUNTERS_COLLECTION', 'model_id_counters'),
'retention_days': int(getenv('IMPORT_BUNDLE_RETENTION_DAYS', '7')),
}