QTZPOC-20

QTZPOC-20: Implement model import workflow for .sientia bundles

Introduces the import_model workflow and its associated activities to enable importing .sientia bundles into MLflow and MongoDB. This includes:

*   New core workflow and activity logic for handling bundle download, decryption, validation, MLflow provisioning (experiments, runs, registered versions), and MongoDB model document creation.
*   Comprehensive E2E and unit test suites, including a dedicated test harness and a manual run script for development.
*   Updated .env.example and README.md with new configurations and usage instructions.
*   Ensures correct MLflow model source resolution for legacy loader compatibility and robust MongoDB document management.
*   Refactors existing utility scripts into a scripts/ directory.
This commit is contained in:
2026-08-25 17:07:42 -03:00
parent 0e445de386
commit dac4f1718f
41 changed files with 8168 additions and 1045 deletions

View File

@@ -181,8 +181,11 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API, ModelImport):
import_config=import_config or build_import_config(),
minio_repository=minio_repository,
# The MLflow repository is shared rather than rebuilt: setting the tracking URI is a
# process-wide side effect, so two instances would be two chances to disagree.
mlflow_repository=self.model_monitoring_repository,
# process-wide side effect, so two instances would be two chances to disagree. Read
# through `getattr` because `MLFlow.__init__` is what creates it — if that ever stops
# running before this line, the import activities refuse with "MLflow repository not
# initialized" at the point of use rather than silently provisioning nothing.
mlflow_repository=getattr(self, 'model_monitoring_repository', None),
mongo_repository=mongo_repository,
logger=logger,
notification_handler=notification_handler,

View File

@@ -853,8 +853,15 @@ class ModelImport(SientiaMonitoring):
async def register_import_model_version(self, input_data: dict[str, Any]) -> dict[str, Any]:
"""Register a version whose source the legacy loader can parse. No stage change.
The source is `runs:/<run_id>/prediction_model`, because `get_model_run_id` extracts the run
id as `source.split('/')[2]` — a `models:/m-*` source, what the new stack emits, breaks it.
The source is the run's **resolved artifact URI** with `prediction_model` appended —
`mlflow-artifacts:/<experiment_id>/<run_id>/artifacts/prediction_model` on this deployment —
because `get_model_run_id` extracts the run id as `source.split('/')[2]`. That index is the
run id only in the resolved form: a literal `runs:/<run_id>/prediction_model` has just two
slashes, so index 2 is the string `prediction_model`, and a `models:/m-*` source (what the
new stack emits) has none of the parts at all. This is the same value the training flow
stores, because `mlflow.register_model` resolves the `runs:/` URI before writing it
(`model_repository.py`, `register_model`) — `create_model_version` does not.
The stage is left untouched: a `COMPLETED` import is a registered model, not a servable one,
and promoting it to `Production` is a manual operator step outside this workflow.
@@ -880,7 +887,7 @@ class ModelImport(SientiaMonitoring):
# lands as a new version of the same registered model, never a differently named one.
client.create_registered_model(model_name)
source = f'runs:/{run_id}/prediction_model'
source = f'{str(client.get_run(run_id).info.artifact_uri).rstrip("/")}/prediction_model'
version = client.create_model_version(name=model_name, source=source, run_id=run_id)
self.info(
f'import registered {model_name} version {version.version}',