sientia-dataops-laborious_temporal/laborious-temporal-plugin-store-migration-plan.md

tags, created, modified, created_by, modified_by, status

tags	created	modified	created_by	modified_by	status
engineering sientia runtime-system laborious-temporal plugin-store migration-plan	2026-03-02	2026-03-02	Vitor Pimentel	Vitor Pimentel	draft

Sientia Laborious Temporal — PluginStore & Wrapper Migration Plan

Migration plan for evolving sientia-dataops-laborious_temporal from direct MLflow model loading to a runtime-aware architecture that uses Sientia model wrappers (SientiaModel) via their public methods, aligned with the runtime strategy.

Summary

1. #Objectives and Scope — What this migration must achieve
2. #Existing State Overview (laborious_temporal) — Current responsibilities and coupling points
3. #Requirements Mapping — Functional and non-functional requirements
4. #Target Architecture — Desired runtime and model interaction architecture
5. #Implementation Plan — Phased, detailed changes to apply
6. #Testing Strategy — How to validate the new behavior
7. #Rollout and Migration Strategy — How to safely roll out the changes
8. #Related Documents — Cross-links to supporting documents

---

Objectives and Scope

This migration focuses on the sientia-dataops-laborious_temporal application and aims to:

• Keep the runtime-aware deployment model consistent with the rest of the runtime system (Helm + RUNTIME env var, runtime installation via PluginStore).
• Ensure that all interactions with models use the public methods of the Sientia wrapper (SientiaModel):
  • Use SientiaModel.train(...) and retrain(...) for training and retraining flows.
  • Use SientiaModel.predict(...) and SientiaModel.transform(...) for inference and preprocessing.
• Use the shared MLflow repository (SientiaMLflowRepository) for all MLflow operations (load, runs, artifacts, promotion, production lookup, metadata logging); do not implement these in Laborious.

Out of scope:

• Replacing MLflow as the tracking and registry backend.
• Redesigning Temporal workflows (queues, retry policies) beyond what is required for the new model interaction style.

---

Existing State Overview (laborious_temporal)

Key components in sientia-dataops-laborious_temporal:

• MLflow activities (laborious/activities/mlflow.py)

  • MLFlow class exposes Temporal activities for:
    • request_transform — loads transformation models from MLflow and applies them to input data.
    • request_predict — loads predictive models from MLflow and generates predictions.
    • retrain_model — orchestrates retraining using historical data stored in MinIO and MLflow registry.
    • update_production_model — promotes new versions to production.
    • get_reference_data — fetches evaluation/reference datasets from model artifacts.
  • These activities delegate ML-specific work to MLFlowRepository.

• MLflow repository (laborious/utils/repository/model_repository.py)

  • MLFlowRepository encapsulates the interaction with MLflow:
    • Model discovery and run resolution (get_model_run_id, get_model_uri, get_experiment, etc.).
    • Artifact download and loading for both transformer and prediction models.
    • Model caching and retention (get_model, get_cached_operation).
    • Transformation and prediction entry points:
      • transform(...) wraps get_cached_operation(..., operation='transform').
      • predict(...) wraps get_cached_operation(..., operation='predict').
    • Retraining orchestration (fit_models, create_new_experiment, retrain_model, update_production_model).
  • Today:
    • Models are loaded via MLflow flavors: sklearn, pyfunc, pytorch.
    • When flavor == 'pyfunc' and load_wrapper=True, the repository loads a wrapper via:
      • raw_model = mlflow.pyfunc.load_model(artifact_path)
      • model = raw_model._model_impl.python_model
    • Production models are resolved using stages in the Model Registry (for example, selecting the latest version in stage Production); aliases such as @production are not used yet, and models are registered explicitly as part of the current retrain/promotion flows.
    • The wrapper’s _model_impl class does not extend SientiaModel.
    • All this MLflow-specific logic is local to laborious_temporal and partially duplicated in sientia-dataops-model-manager, which motivates the extraction of a shared MLflow repository in sientia-dataops-library (see mlflow-shared-repository-migration-plan).

MLflow: All MLflow ops → mlflow-shared-repository-migration-plan. Laborious uses the interface; SientiaModel lifecycle is in sientia-model-library.

Current vs Target — High-level Flow

flowchart LR
    subgraph current [Current State — Laborious Temporal]
        direction TB
        TemporalWorker["Temporal Worker"]
        MlflowActivities["MLFlow Activities\nrequest_transform / request_predict / retrain_model"]
        MLFlowRepositoryNode["MLFlowRepository"]
        MLflowRegistry["MLflow Tracking + Registry"]
        RawModel["Loaded Model\n(sklearn / pyfunc / pytorch)"]

        TemporalWorker -->|"start workflow\n(Temporal)"| MlflowActivities
        MlflowActivities -->|"call transform()/predict()/retrain_model()"| MLFlowRepositoryNode
        MLFlowRepositoryNode -->|"search_model_versions()\ncurrent_stage == 'Production'"| MLflowRegistry
        MLFlowRepositoryNode -->|"mlflow.*.load_model(model_uri)"| RawModel
        MLFlowRepositoryNode -->|"raw_model.predict(data)\nraw_model.fit(data)"| RawModel
    end

    subgraph target [Target State — Laborious Temporal]
        direction TB
        TemporalWorker2["Temporal Worker"]
        MlflowActivities2["MLFlow Activities\n(same APIs)"]
        MLFlowRepositoryNode2["MLFlowRepository\n(wrapper-aware)"]
        MLflowRegistry2["MLflow Tracking + Registry\n(aliases enabled)"]
        SientiaWrapperNode["Wrapper Instance\n(extends SientiaModel)"]

        TemporalWorker2 -->|"start workflow\n(Temporal)"| MlflowActivities2
        MlflowActivities2 -->|"call transform()/predict()/retrain_model()"| MLFlowRepositoryNode2
        MLFlowRepositoryNode2 -->|"get_model_version_by_alias('production')\n& models:/name@production"| MLflowRegistry2
        MLFlowRepositoryNode2 -->|"mlflow.pyfunc.load_model(...)"| SientiaWrapperNode
        MLFlowRepositoryNode2 -->|"wrapper.transform(...)\nwrapper.predict(...)\nwrapper.train()/retrain()"| SientiaWrapperNode
        SientiaWrapperNode -->|"store_model(...)\n(auto-register + update alias)"| MLflowRegistry2
    end

Current vs Target — Retrain Hot Path (Code Sketch)

Current retrain flow inside MLFlowRepository.fit_models / retrain_model (simplified):

data_model, _ = await self.download_model(
    model_name=model_name,
    metadata=metadata,
    model_type="transform",
    flavor=transform_flavor,
    load_wrapper=(transform_flavor == "pyfunc"),
)

prediction_model, _ = await self.download_model(
    model_name=model_name,
    metadata=metadata,
    model_type="predict",
    flavor=predict_flavor,
    load_wrapper=(predict_flavor == "pyfunc"),
)

treated_data_candidate = data_model.fit(data)          # or data_model.predict(data)
...
prediction_model.fit(retrain_dataset)                  # direct fit on underlying model

Target retrain flow when using wrappers that extend SientiaModel:

wrapper, _ = await self.download_model(
    model_name=model_name,
    metadata=metadata,
    # model_type="predict",              # wrapper owns both transformer + model stack
    # flavor="pyfunc", flavor will always be pyfunc, this parameter will be removed
    # load_wrapper=True, wrapper will always be loaded from _model_impl
)

# First-time training or full retrain using public API
wrapper.train(
    train_data=train_df,               # features + target
    val_data=val_df,
    target=target_name,
)

# Incremental retrain (when appropriate)
wrapper.retrain(full_retrain_df)

transformed_df, trans_meta = wrapper.transform(raw_df)
pred_df, pred_meta = wrapper.predict({}, transformed_df, params={})

---

Requirements Mapping

Functional Requirements

ID	Requirement	Description	Impacted Areas
FR-01	Runtime detection and installation	Align worker startup with RUNTIME and runtime installation via PluginStore	Worker
FR-02	Wrapper‑based training using public API	Retraining must call the public retrain(...) method of SientiaModel	fit_models, retrain_model paths
FR-03	Wrapper‑based inference using public API	Inference must call predict(...) and transform(...) on the wrapper; obtain wrappers via SientiaMLflowRepository	Activities, shared repository
FR-04	Use shared MLflow repository	Delegate all MLflow operations (load, runs, artifacts, promotion, production lookup) to SientiaMLflowRepository; do not implement in Laborious	mlflow-shared-repository-migration-plan
FR-05	Model configuration for training/retraining	model_config must define target and retention_minutes; all models are pyfunc + wrapper (no flavor selection)	model_config structures

Non-Functional Requirements

ID	Requirement	Description	Impacted Areas
NFR-01	Consistent public API usage	All wrapper interactions must go through public SientiaModel methods; metadata logging is handled by the shared repository	Activities
NFR-02	Fail fast when wrappers unavailable	Where wrappers are not yet available, raise an explicit error requesting model update	Shared repository, config
NFR-03	Testability	Enable unit tests to validate wrapper-based flows and integration with shared repository	tests/laborious
NFR-04	Operational safety	Retraining and promotion behavior must remain auditable and robust	Retrain & promotion flows

---

Target Architecture

Wrapper-centric model interactions

The target state for laborious_temporal is:

• All training and retraining logic goes through the wrapper’s public methods: train(...) and retrain(...).
• Inference and transformation use transform(df) and predict(context, df, params).
• All MLflow operations (load, runs, artifacts, promotion, production lookup) go through SientiaMLflowRepository (see mlflow-shared-repository-migration-plan).

Use of Shared MLflow Repository

Laborious obtains wrappers and performs all MLflow operations via SientiaMLflowRepository. The shared repository (see mlflow-shared-repository-migration-plan) owns: alias-based URIs, pyfunc loading, wrapper extraction, promotion, runs, artifacts, and metadata logging. Laborious activities call repo.load_wrapper(...) and then the wrapper’s public methods (transform, predict, train, retrain); they do not implement MLflow logic.

---

Implementation Plan

Phase 0 — Design and Configuration Alignment

• P0-01: Catalog model types and flavors used by laborious_temporal:
  • For each active model:
    • Flavor will be always pyfunc
    • Whether a _model_impl wrapper is already present and extends SientiaModel.
• P0-02: Define configuration fields in model_config for wrapper usage:
  • Example:
    • target field for training/retraining (already partially present).
    • retention_minutes field for model retention in minutes (unchanged).

Phase 1 — Inference via SientiaModel Public API (using shared repository)

• P1-01: Replace download_model with SientiaMLflowRepository.load_wrapper(...); remove local MLflow loading logic.
• P1-02: Update get_cached_operation to call wrapper.transform(data) and wrapper.predict({}, data); unpack returned metadata; metadata logging is handled by the shared repository.
• P1-03: Ensure activities (request_transform, request_predict) remain unchanged externally (inputs/outputs unchanged).

Phase 2 — Retraining via SientiaModel.retrain

• P2-01: Refactor fit_models to call wrapper.retrain(data) and wrapper.train(...); use SientiaMLflowRepository for runs, metrics, artifacts, and promotion (no local MLflow logic).

Phase 3 — MLflow Logging and Promotion (via shared repository)

• P3-01: In create_new_experiment, use the wrapper’s store_model(...) for model artifacts; use SientiaMLflowRepository for runs, metrics, and any additional MLflow operations.
• P3-02: Use SientiaMLflowRepository.promote_to_alias(...) for production promotion; model registration is auto-handled by wrappers.

Phase 4 — Runtime Alignment

• P4-01:laborious_temporal is also deployed via the runtime-aware Helm chart:
  • Read RUNTIME env var.
  • Install runtime via PluginStore before starting Temporal workers.
• P4-02: Standardize worker queue name as {project_name}-{runtime}-queue.
• P4-03: Fix quality pipelines to a single runtime (to be decided).

---

Testing Strategy

• T1 — Unit tests
  • Add tests in tests/laborious/utils/repository/test_model_repository.py to cover:
    • Wrapper-based get_cached_operation for both transform and predict (using shared repository).
    • Wrapper-based fit_models and retrain_model paths calling train and retrain respectively.
  • Code coverage must be 100%.
• T2 — Integration tests
  • Use existing end-to-end tests under e2e/:
    • Configure a model with SientiaModel wrapper (loaded via shared repository).
    • Run full prediction and retrain workflows; compare predictions, retrain outcomes, and MLflow artifacts.

---

Rollout and Migration Strategy

• R1 — Create models with the new architecture

  • Update or create models in the modeling pipeline so they:
    • Use wrappers that extend SientiaModel.
    • Correctly implement train, retrain, predict, transform, and store_model.
    • Produce structured metadata in transform_meta and pred_meta.
  • Publish these models to a test store (or dedicated branch/experiment) for initial validation.

• R2 — Provision runtimes for the new models

  • Configure and install dedicated runtimes for the new models:
    • Ensure runtime dependencies (Python and system libraries) are available via PluginStore/runtime installer.
    • Validate that each runtime can:
      • Load the wrapper through MLflow.
      • Execute transform and predict end-to-end on sample data.

• R3 — Run new models in real workflows

  • Integrate the new wrapper-based models into real Laborious workflows, initially in non-critical environments:
    • Route only a subset of flows or entities to the new models.
    • Monitor logs (including metadata), business metrics, and retraining/promotion behavior.
  • Promote these models to production using aliases (@production) in MLflow 3+.

• R4 — Migrate remaining models progressively

  • Define migration waves by model family:
    • For each existing model:
      • Create or adapt a SientiaModel wrapper.
      • Provision the corresponding runtime.
      • Execute the test cycle (T1–T2) from the previous section.
      • Update aliases so production traffic uses the new wrapper.
  • After all models are migrated:
    • Remove legacy stage-based paths (Production) and non-wrapper models.
    • Simplify the codebase to assume wrappers + aliases only.

---

Related Documents

• mlflow-shared-repository-migration-plan — Concepts implemented in the common interface (production lookup, wrapper loading, promotion, metadata logging, etc.)
• model-manager-plugin-store-migration-plan
• analytics-implementation-plan
• analytics
• ../model-plugin-system/06-end-to-end-flow