QTZPOC-20
QTZPOC-20: Implement model import workflow for .sientia bundles Introduces the import_model workflow and its associated activities to enable importing .sientia bundles into MLflow and MongoDB. This includes: * New core workflow and activity logic for handling bundle download, decryption, validation, MLflow provisioning (experiments, runs, registered versions), and MongoDB model document creation. * Comprehensive E2E and unit test suites, including a dedicated test harness and a manual run script for development. * Updated .env.example and README.md with new configurations and usage instructions. * Ensures correct MLflow model source resolution for legacy loader compatibility and robust MongoDB document management. * Refactors existing utility scripts into a scripts/ directory.
This commit is contained in:
@@ -181,8 +181,11 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API, ModelImport):
|
||||
import_config=import_config or build_import_config(),
|
||||
minio_repository=minio_repository,
|
||||
# The MLflow repository is shared rather than rebuilt: setting the tracking URI is a
|
||||
# process-wide side effect, so two instances would be two chances to disagree.
|
||||
mlflow_repository=self.model_monitoring_repository,
|
||||
# process-wide side effect, so two instances would be two chances to disagree. Read
|
||||
# through `getattr` because `MLFlow.__init__` is what creates it — if that ever stops
|
||||
# running before this line, the import activities refuse with "MLflow repository not
|
||||
# initialized" at the point of use rather than silently provisioning nothing.
|
||||
mlflow_repository=getattr(self, 'model_monitoring_repository', None),
|
||||
mongo_repository=mongo_repository,
|
||||
logger=logger,
|
||||
notification_handler=notification_handler,
|
||||
|
||||
@@ -853,8 +853,15 @@ class ModelImport(SientiaMonitoring):
|
||||
async def register_import_model_version(self, input_data: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Register a version whose source the legacy loader can parse. No stage change.
|
||||
|
||||
The source is `runs:/<run_id>/prediction_model`, because `get_model_run_id` extracts the run
|
||||
id as `source.split('/')[2]` — a `models:/m-*` source, what the new stack emits, breaks it.
|
||||
The source is the run's **resolved artifact URI** with `prediction_model` appended —
|
||||
`mlflow-artifacts:/<experiment_id>/<run_id>/artifacts/prediction_model` on this deployment —
|
||||
because `get_model_run_id` extracts the run id as `source.split('/')[2]`. That index is the
|
||||
run id only in the resolved form: a literal `runs:/<run_id>/prediction_model` has just two
|
||||
slashes, so index 2 is the string `prediction_model`, and a `models:/m-*` source (what the
|
||||
new stack emits) has none of the parts at all. This is the same value the training flow
|
||||
stores, because `mlflow.register_model` resolves the `runs:/` URI before writing it
|
||||
(`model_repository.py`, `register_model`) — `create_model_version` does not.
|
||||
|
||||
The stage is left untouched: a `COMPLETED` import is a registered model, not a servable one,
|
||||
and promoting it to `Production` is a manual operator step outside this workflow.
|
||||
|
||||
@@ -880,7 +887,7 @@ class ModelImport(SientiaMonitoring):
|
||||
# lands as a new version of the same registered model, never a differently named one.
|
||||
client.create_registered_model(model_name)
|
||||
|
||||
source = f'runs:/{run_id}/prediction_model'
|
||||
source = f'{str(client.get_run(run_id).info.artifact_uri).rstrip("/")}/prediction_model'
|
||||
version = client.create_model_version(name=model_name, source=source, run_id=run_id)
|
||||
self.info(
|
||||
f'import registered {model_name} version {version.version}',
|
||||
|
||||
Reference in New Issue
Block a user