sientia-dataops-laborious_temporal/e2e/scenarios.md

E2E Scenario Documentation - Predictions Batch

This document describes the end-to-end scenarios for predictions_batch and its child workflows: prediction_process and format_and_export_prediction.

It is a functional reference of scenario behavior, inputs, and expected outcomes.

Execution Context

• Tests run under e2e/ and are marked with @pytest.mark.integration.
• PostgreSQL and MinIO are provisioned with testcontainers.
• test_minio_offload.py uses real MinIO I/O; other scenario suites may use stubs/mocks for optional outputs.

---

1. Main Workflow Scenarios

Source: e2e/test_predictions_batch_main_workflow.py

1.1.1 Happy Path - Complete Success

Summary: Full workflow succeeds with valid query and default gate behavior.

Description:

• Query returns rows for a model.
• prediction_process runs transform and predict paths.
• Final prediction and transformed data are persisted.

Expected Outcome:

• Exactly one prediction row is created.
• Transform rows are created.
• Confidence/status/comments are success values.

1.2.1 SQL Query Execution Error

Summary: Invalid SQL leads to no persisted prediction.

Description:

• Input query is invalid.
• Load step fails and workflow follows error/short-circuit path.

Expected Outcome:

• No prediction rows for the model.
• Workflow does not require retry-loop assumptions in assertions.

1.2.2 Missing Required Parameters

Summary: Missing required fields prevent workflow completion path.

Description:

• Required input key (e.g. query) is omitted.
• Workflow fails to produce actionable input for child flow.

Expected Outcome:

• No prediction rows are persisted.
• Workflow handle may require explicit terminate in E2E harness.

1.2.3 Invalid Datetime Column Specification (de-prioritized)

Summary: Legacy invalid datetime-column case is retained only as low-priority legacy coverage.

Description:

• datetime_columns references non-existing columns.
• Behavior may vary by query shape and parser fallback.

Expected Outcome:

• No predictions persisted in the covered legacy assertion path.
• Scenario is not considered primary behavior coverage.

---

2. Prediction Process Scenarios

Source: e2e/test_predictions_batch_prediction_process.py

2.1 Input Gate Path Decisions

2.1.1 CONTINUE

Summary: Input filter flags quality issue but allows continuation via default path.

Description:

• Input gate returns CONTINUE.
• MLFlow transform/predict are skipped.
• Export path persists default-style prediction with warning context.

2.1.2 STOP

Summary: Input filter blocks processing.

Description:

• Input gate returns STOP.
• Workflow exits without export.

2.1.3 REPEAT with history

Summary: Prior prediction is reused.

Description:

• Input gate returns REPEAT.
• repeat_last_prediction path is executed using existing historical row.

2.1.4 REPEAT without history

Summary: Repeat requested but no previous prediction exists.

Description:

• Input gate returns REPEAT.
• No prior row is available to duplicate.

Expected Outcome:

• No new prediction rows are created for the model.

2.2 Transform Gate Decisions

2.2.1 CONTINUE on transform response error

Summary: Transform response is degraded, but workflow continues.

2.2.2 STOP on transform response error

Summary: Transform response error blocks downstream processing.

2.2.3 REPEAT on transform response error

Summary: Transform response error triggers repeat-last-prediction path.

2.2.4 STOP on transform content NaN

Summary: Content gate (NAN_VALUES) blocks on all-NaN transform payload.

2.3 Predict Gate Decisions

2.3.1 CONTINUE on predict response error

Summary: Predict response degraded; workflow exports with degraded metadata.

2.3.2 STOP on predict response error

Summary: Predict response error blocks export.

2.3.3 REPEAT on predict response error

Summary: Predict response error routes to repeat-last-prediction.

2.4.1 Priority Conflict Resolution

Summary: Deterministic selection when multiple filters produce different flags.

Description:

• Multiple filters may produce STOP, CONTINUE, and/or REPEAT.
• path_priority defines precedence.

Expected Outcome:

• Highest-priority flag is applied consistently.
• Executed branch matches configured priority ordering.

---

3. Format and Export Scenarios

Source: e2e/test_predictions_batch_format_export.py

3.1 Output Combination Scenarios

3.1.1 Default prediction export

Summary: Non-None path flag uses format_default_prediction.

Description:

• Default prediction is generated.
• Transform export is skipped.
• Optional outputs (PI/OPC) still execute when configured.

3.1.2 OPC only

Summary: Postgres + OPC writes, PI Web API disabled.

3.1.3 PI Web API only

Summary: Postgres + PI writes, OPC disabled.

3.1.4 Postgres only

Summary: Both optional outputs disabled; only Postgres persistence and metrics.

3.1.5 No transformed data export

Summary: Prediction is persisted; transformed table is not written.

3.2 Degraded-but-successful Completion

3.2.1 PI Web API write error

Summary: PI write failure does not fail workflow.

Expected Outcome:

• Workflow completes.
• Prediction persisted with degraded confidence/comments (PI error semantics).

3.2.2 OPC write error

Summary: OPC write failure does not fail workflow.

Expected Outcome:

• Workflow completes.
• Prediction persisted with OPC degraded confidence/comments.

3.2.3 PI Web API partial write error

Summary: Partial PI acknowledgement is treated as degraded success.

Expected Outcome:

• Workflow completes.
• Prediction persisted with PI error confidence and descriptive comment.

3.3.1 Combined Optional Outputs (PI + OPC)

Summary: Both external output channels are enabled together.

Description:

• PI Web API and OPC configs are both present.
• Output mutation order matters for final persisted payload.

Expected Outcome:

• PI write executes before OPC write in workflow sequence.
• Final Postgres payload reflects any confidence/comment updates.
• OPC metrics are emitted when tag writes return response times.

---

4. MinIO Offload Scenarios

Source: e2e/test_minio_offload.py

4.1.1 Forced offload to MinIO

Summary: Very low threshold forces parquet upload.

Description:

• Payload is offloaded (object_key present, inline data absent/empty).
• Object is present in MinIO under prediction_datasets/....
• Retrieval reconstructs the dataframe.

4.1.2 Full workflow with offloaded load payload

Summary: Offload path works during full predictions_batch execution.

Expected Outcome:

• Workflow completes.
• Prediction row is persisted.

4.2.1 Inline payload below threshold

Summary: Data remains inline when threshold is not exceeded.

Expected Outcome:

• Payload stores inline data.
• object_key is None.
• Downstream persistence behavior matches offload scenario semantics.

---

5. Drift Workflow Scenarios

Source: e2e/test_drift.py

The drift suite mocks sientia.ModelAnalysis.ModelAnalysis (not installed; see CODE_ISSUES.md issue #1) through the controllable _FakeModelAnalysis stub exposed by the model_analysis_stub fixture. The mlflow_repository_stub provides the reference-data CSV via download_artifacts. Every scenario asserts postgres rows in predictions_schema.drift against this canonical schema:

id, model_id, feature, method, value, drift, chunk, timestamp, timestamp_end, accurate, created_at, updated_at.

5.1 Happy paths

D.1.1 Full pipeline persists all columns with reference data

Summary: ModelAnalysis returns a deterministic drift dataframe; the reference CSV is downloaded from the MLflow stub.

Expected Outcome:

• One row per (chunk, feature, method) plus a multivariate block per chunk.
• Every drift column is populated and accurate=True.
• timestamp_end preserves the high-precision string (HH:MM:59.999999999).
• p_value is dropped before persistence.
• drift flags propagate per (feature, method) configuration.

D.1.2 30% fallback when reference data is unavailable

Summary: get_reference_data fails alias resolution and returns None; calculate_drift uses the first 30% of target rows as reference.

Expected Outcome:

• Persisted rows carry accurate=False.
• A MODEL_METRICS_REFERENCE_DATA_WARNING notification is emitted to MongoDB.

5.2 Filtering / dedup invariants

D.2.1 Deduplication and p_value removal

Summary: ModelAnalysis returns duplicate (timestamp, method, feature) rows plus a p_value column.

Expected Outcome:

• Duplicates are collapsed keeping the first occurrence.
• p_value is absent from the persisted rows.

D.2.2 Out-of-range timestamps filtered

Summary: Drift rows whose timestamps are not present in the target window must be discarded before persistence.

5.3 Failure paths

D.3.1 Empty target data short-circuits the workflow

Summary: load_custom_query returns no rows; ModelAnalysis is never instantiated and no drift rows are written.

D.3.2 ModelAnalysis raises during dataframe assembly

Summary: get_drift_metrics_dataframe raises. The activity catches the error, sends a MODEL_METRICS_GET_DRIFT_METRICS_ERROR notification, and the workflow completes without persisting drift rows.

5.4 Configuration paths

D.4.1 Default drift metrics propagated to analyzer

Summary: Omitting drift_metrics defaults to ['kolmogorov_smirnov', 'jensen_shannon', 'wasserstein'] and forwards the exact list to detect_univariate_drift.

D.4.2 Invalid chunk_period raises ValueError

Summary: Anything other than min / s is rejected by calculate_drift.

D.4.3 chunk_period='s' keeps seconds in timestamp filtering

Summary: Truncated YYYY-MM-DD HH:MM rows are filtered out when chunking runs at second granularity.

---

6. Simple Metrics Workflow Scenarios

Source: e2e/test_simple_metrics.py

Validates predictions_schema.simple_metrics_data columns: id, model_id, metric, value, timestamp, data_size, interval_minutes, created_at.

6.1 Happy paths

S.1.1 rmse/mse/mae/r2 happy path

Summary: Prediction/target pairs are inserted; the activity computes all four metrics with closed-form expected values.

Expected Outcome:

• One row per metric is persisted; all columns populated.
• data_size matches the joined row count and interval_minutes=60.

S.1.2 Subset metrics

Summary: Requesting metrics=['rmse'] writes only the rmse row.

6.2 Edge cases

S.2.1 Zero-variance target returns r2=0

Summary: When all targets are equal, ss_tot=0; the activity must guard against division by zero and return r2=0.

6.3 Failure paths

S.3.1 No overlapping data short-circuits persistence

Summary: With no laborious_data rows for the configured target variable the workflow exits before calculate_simple_metrics and writes nothing.

---

7. Minimal Retrain Workflow Scenarios

Source: e2e/test_minimal_retrain.py

The MLflow registry is fully mocked (no real artifacts in test container). Validates predictions_schema.retrain_reports columns: id, model_id, model_name, timestamp, status, version, mlflow_run_id, mlflow_experiment_id, created_at.

7.1 Happy path

MR.1.1 Successful retrain + promotion

Summary: Training data loads via MinIO offload, wrapper.retrain succeeds, the new version is promoted to the production alias.

Expected Outcome:

• Report row has success status, version='7', mlflow_run_id='retrain-run-id', mlflow_experiment_id='experiment-id'.
• mlflow.log_artifact is called with the input CSV.
• promote_to_alias is called once with the resolved version and alias.

7.2 Failure paths

MR.2.1 Wrapper retrain raises

Summary: wrapper.retrain raises RuntimeError. The activity returns success=False, update_production_model is NOT invoked.

Expected Outcome:

• Report row carries the error message and version/mlflow_* columns are NULL.

MR.2.2 Missing model_config.target

Summary: Empty model config short-circuits before any MLflow call.

Expected Outcome:

• Report row carries the explicit guard message.
• get_cached_model is never invoked.

MR.3.1 No training data

Summary: The training query returns no rows; the workflow does not persist any report row. The current code raises plain ValueError from the workflow function, which Temporal treats as a workflow-task failure (see CODE_ISSUES.md issue MR-1).

---

Input Contract Reference

Common scenario input fields:

• schedule_name
• model_name
• model_id
• query
• schema
• table_name
• transform_table_name
• input_filters
• mlflow_transform_filters
• mlflow_predict_filters
• path_priority (default order: STOP, CONTINUE, REPEAT)
• save_transform
• prediction_store_policy
• model_config.target
• datetime_columns (when query returns temporal fields)

Optional outputs:

• opc_output_config
• pi_web_api_output_config