sientia-dataops-laborious_temporal/e2e/scenarios.md

E2E Scenario Documentation - Predictions Batch

This document describes the end-to-end scenarios for predictions_batch and its child workflows: prediction_process and format_and_export_prediction.

It is a functional reference of scenario behavior, inputs, and expected outcomes.

Execution Context

• Tests run under e2e/ and are marked with @pytest.mark.integration.
• PostgreSQL and MinIO are provisioned with testcontainers.
• test_minio_offload.py uses real MinIO I/O; other scenario suites may use stubs/mocks for optional outputs.

---

1. Main Workflow Scenarios

Source: e2e/test_predictions_batch_main_workflow.py

1.1.1 Happy Path - Complete Success

Summary: Full workflow succeeds with valid query and default gate behavior.

Description:

• Query returns rows for a model.
• prediction_process runs transform and predict paths.
• Final prediction and transformed data are persisted.

Expected Outcome:

• Exactly one prediction row is created.
• Transform rows are created.
• Confidence/status/comments are success values.

1.2.1 SQL Query Execution Error

Summary: Invalid SQL leads to no persisted prediction.

Description:

• Input query is invalid.
• Load step fails and workflow follows error/short-circuit path.

Expected Outcome:

• No prediction rows for the model.
• Workflow does not require retry-loop assumptions in assertions.

1.2.2 Missing Required Parameters

Summary: Missing required fields prevent workflow completion path.

Description:

• Required input key (e.g. query) is omitted.
• Workflow fails to produce actionable input for child flow.

Expected Outcome:

• No prediction rows are persisted.
• Workflow handle may require explicit terminate in E2E harness.

1.2.3 Invalid Datetime Column Specification (de-prioritized)

Summary: Legacy invalid datetime-column case is retained only as low-priority legacy coverage.

Description:

• datetime_columns references non-existing columns.
• Behavior may vary by query shape and parser fallback.

Expected Outcome:

• No predictions persisted in the covered legacy assertion path.
• Scenario is not considered primary behavior coverage.

---

2. Prediction Process Scenarios

Source: e2e/test_predictions_batch_prediction_process.py

2.1 Input Gate Path Decisions

2.1.1 CONTINUE

Summary: Input filter flags quality issue but allows continuation via default path.

Description:

• Input gate returns CONTINUE.
• MLFlow transform/predict are skipped.
• Export path persists default-style prediction with warning context.

2.1.2 STOP

Summary: Input filter blocks processing.

Description:

• Input gate returns STOP.
• Workflow exits without export.

2.1.3 REPEAT with history

Summary: Prior prediction is reused; outcome depends on batch vs history timestamp.

Description:

• Input gate returns REPEAT.
• repeat_last_prediction inserts a row using last_timestamp from the batch payload (max timestamp in laborious_data for the query), not the copied row’s timestamp.

Tests:

• Collision: batch last_timestamp equals the historical prediction row’s timestamp → Postgres unique_model_id_timestamp violation; workflow fails; still one row.
• Distinct batch time: laborious rows are stamped later than the historical prediction → second row inserted; same prediction fields as the first (see assert_repeat).

2.1.4 REPEAT without history

Summary: Repeat requested but no previous prediction exists.

Description:

• Input gate returns REPEAT.
• No prior row is available to duplicate.

Expected Outcome:

• No new prediction rows are created for the model.

2.2 Transform Gate Decisions

2.2.1 CONTINUE on transform response error

Summary: Transform response is degraded, but workflow continues.

2.2.2 STOP on transform response error

Summary: Transform response error blocks downstream processing.

2.2.3 REPEAT on transform response error

Summary: Transform response error triggers repeat-last-prediction path.

Tests: Same timestamp collision vs distinct batch timestamp as §2.1.3 (*_fails / *_inserts_second_row).

2.2.4 STOP on transform content NaN

Summary: Content gate (NAN_VALUES) blocks on all-NaN transform payload.

2.3 Predict Gate Decisions

2.3.1 CONTINUE on predict response error

Summary: Predict response degraded; workflow exports with degraded metadata.

2.3.2 STOP on predict response error

Summary: Predict response error blocks export.

2.3.3 REPEAT on predict response error

Summary: Predict response error routes to repeat-last-prediction.

Tests: Same timestamp collision vs distinct batch timestamp as §2.1.3 (*_fails / *_inserts_second_row).

2.4.1 Priority Conflict Resolution

Summary: Deterministic selection when multiple filters produce different flags.

Description:

• Multiple filters may produce STOP, CONTINUE, and/or REPEAT.
• path_priority defines precedence.

Expected Outcome:

• Highest-priority flag is applied consistently.
• Executed branch matches configured priority ordering.

---

3. Format and Export Scenarios

Source: e2e/test_predictions_batch_format_export.py

3.1 Output Combination Scenarios

3.1.1 Default prediction export

Summary: Non-None path flag uses format_default_prediction.

Description:

• Default prediction is generated.
• Transform export is skipped.
• Optional outputs (PI/OPC) still execute when configured.

3.1.2 OPC only

Summary: Postgres + OPC writes, PI Web API disabled.

3.1.3 PI Web API only

Summary: Postgres + PI writes, OPC disabled.

3.1.4 Postgres only

Summary: Both optional outputs disabled; only Postgres persistence and metrics.

3.1.5 No transformed data export

Summary: Prediction is persisted; transformed table is not written.

3.2 Degraded-but-successful Completion

3.2.1 PI Web API write error

Summary: PI write failure does not fail workflow.

Expected Outcome:

• Workflow completes.
• Prediction persisted with degraded confidence/comments (PI error semantics).

3.2.2 OPC write error

Summary: OPC write failure does not fail workflow.

Expected Outcome:

• Workflow completes.
• Prediction persisted with OPC degraded confidence/comments.

3.2.3 PI Web API partial write error

Summary: Partial PI acknowledgement is treated as degraded success.

Expected Outcome:

• Workflow completes.
• Prediction persisted with PI error confidence and descriptive comment.

3.3.1 Combined Optional Outputs (PI + OPC)

Summary: Both external output channels are enabled together.

Description:

• PI Web API and OPC configs are both present.
• Output mutation order matters for final persisted payload.

Expected Outcome:

• PI write executes before OPC write in workflow sequence.
• Final Postgres payload reflects any confidence/comment updates.
• OPC metrics are emitted when tag writes return response times.

---

4. MinIO Offload Scenarios

Source: e2e/test_minio_offload.py

4.1.1 Forced offload to MinIO

Summary: Very low threshold forces parquet upload.

Description:

• Payload is offloaded (object_key present, inline data absent/empty).
• Object is present in MinIO under prediction_datasets/....
• Retrieval reconstructs the dataframe.

4.1.2 Full workflow with offloaded load payload

Summary: Offload path works during full predictions_batch execution.

Expected Outcome:

• Workflow completes.
• Prediction row is persisted.

4.2.1 Inline payload below threshold

Summary: Data remains inline when threshold is not exceeded.

Expected Outcome:

• Payload stores inline data.
• object_key is None.
• Downstream persistence behavior matches offload scenario semantics.

---

5. Drift Workflow Scenarios

Source: e2e/test_drift.py

The drift suite drives the real sientia_model.analytics.drift_analysis.DriftAnalysis analyzer (no stubs / mocks). Each scenario exercises the full pipeline:

laborious_data (Postgres) -> load_custom_query
                          -> calculate_drift (DriftAnalysis univariate + multivariate)
                          -> export_data_to_postgres (sientia_data.drift_metrics)

The mlflow_repository_stub provides the reference-data CSV via download_artifacts, and tests assert postgres rows in sientia_data.drift_metrics against this canonical schema:

id, model_id, feature, method, value, alert, chunk_index, chunk_start_date, chunk_end_date, accurate, timestamp, created_at.

Tests assert behavioral / structural properties (column presence, NOT NULL constraints, business-key invariants like uniform timestamp and stamped model_id) rather than exact numeric drift scores, since those depend on the real analyzer implementation and the synthetic data fed in.

5.1 Happy paths

D.1.1 Full pipeline persists all columns with reference data

Summary: 10 minutes of target data are inserted; a 10-row reference CSV is configured via the MLflow stub. The DriftAnalysis runs end-to-end.

Expected Outcome:

• One row per (chunk_index, feature, method) plus a multivariate block per chunk is persisted.
• Every column in the DDL is populated; feature is the only nullable column per the new schema.
• accurate=True for every row (reference path).
• All three default univariate methods reach the analyzer.
• model_id is stamped as text and uniform across rows.
• timestamp equals max(target_data.timestamp) and is uniform across rows.
• chunk_start_date / chunk_end_date are persisted as ISO text and ordered.
• p_value is dropped before persistence.

D.1.2 30% fallback when reference data is unavailable

Summary: MLflow alias resolution is forced to fail so get_reference_data returns None; calculate_drift falls back to the first 30% of target rows as reference.

Expected Outcome:

• All persisted rows carry accurate=False.
• A MODEL_METRICS_REFERENCE_DATA_WARNING notification is emitted to MongoDB.

5.2 Failure paths

D.3.1 Empty target data short-circuits the workflow

Summary: load_custom_query returns no rows.

Expected Outcome:

• The workflow returns early and writes nothing to sientia_data.drift_metrics.

5.3 Configuration paths

D.4.2 Invalid chunk_period raises ValueError

Summary: Anything other than min / s is rejected by calculate_drift.

Expected Outcome:

• The workflow surfaces the ValueError ("Invalid chunk period: ...").
• No rows are persisted.

D.4.3a Insufficient drift metrics while target has rows

Summary: Laborious raises when the merged drift table is empty but the target window is non-empty (Insufficient drift data: + notification MODEL_METRICS_DRIFT_INSUFFICIENT_DATA).

Description:

• The e2e patches ModelMetrics.get_drift_metrics to return an empty DataFrame, simulating a sientia_model path that emits no rows.

Expected Outcome:

• Workflow fails; no rows in sientia_data.drift_metrics.

D.4.3b chunk_period='s' preserves seconds in chunk_start_date

Summary: Target data includes sub-minute spacing across several minutes; the activity uses chunk_period='s'.

Expected Outcome:

• At least one persisted chunk_start_date carries seconds=30, proving that the analyzer chunked at sub-minute granularity and the ISO-text serialization preserved the boundary.

Note: The e2e patches DriftAnalysis._chunk_dataframe to skip empty pd.Grouper(freq='s') buckets. The stock implementation iterates every second between min/max timestamps, producing empty chunks and NaT rows that calculate_drift filters away entirely. The durable fix belongs in sientia_model.

---

6. Simple Metrics Workflow Scenarios

Source: e2e/test_simple_metrics.py

Validates sientia_data.simple_metrics columns: id, model_id, metric, value, timestamp, data_size, interval_minutes, created_at. Note: timestamp is now nullable per the new DDL and model_id is text.

6.1 Happy paths

S.1.1 rmse/mse/mae/r2 happy path

Summary: Prediction/target pairs are inserted; the activity computes all four metrics with closed-form expected values.

Expected Outcome:

• One row per metric is persisted; all columns populated.
• data_size matches the joined row count and interval_minutes=60.

S.1.2 Subset metrics

Summary: Requesting metrics=['rmse'] writes only the rmse row.

6.2 Edge cases

S.2.1 Zero-variance target returns r2=0

Summary: When all targets are equal, ss_tot=0; the activity must guard against division by zero and return r2=0.

6.3 Failure paths

S.3.1 No overlapping data short-circuits persistence

Summary: With no laborious_data rows for the configured target variable the workflow exits before calculate_simple_metrics and writes nothing.

---

7. Minimal Retrain Workflow Scenarios

Source: e2e/test_minimal_retrain.py

The MLflow registry is fully mocked (no real artifacts in test container). Validates sientia_data.log_retrain columns: mlflow_experiment_id, mlflow_run_id, model_id, model_name, status, timestamp, version. Note: the new DDL drops the legacy id and created_at columns, mlflow_experiment_id is now int8 and model_id is text.

7.1 Happy path

MR.1.1 Successful retrain + promotion

Summary: Training data loads via MinIO offload, wrapper.retrain succeeds, the new version is promoted to the production alias.

Expected Outcome:

• Report row has success status, version='7', mlflow_run_id='retrain-run-id', mlflow_experiment_id=4242 (int8).
• mlflow.log_artifact is called with the input CSV.
• promote_to_alias is called once with the resolved version and alias.

7.2 Failure paths

MR.2.1 Wrapper retrain raises

Summary: wrapper.retrain raises RuntimeError. The activity returns success=False, update_production_model is NOT invoked.

Expected Outcome:

• Report row carries the error message and version/mlflow_* columns are NULL.

MR.2.2 Missing model_config.target

Summary: Empty model config short-circuits before any MLflow call.

Expected Outcome:

• Report row carries the explicit guard message.
• get_cached_model is never invoked.

MR.3.1 No training data

Summary: The training query returns no rows; the workflow does not persist any report row. The current code raises plain ValueError from the workflow function, which Temporal treats as a workflow-task failure (see CODE_ISSUES.md issue MR-1).

---

Input Contract Reference

Common scenario input fields:

• schedule_name
• model_name
• model_id
• query
• schema
• table_name
• transform_table_name
• input_filters
• mlflow_transform_filters
• mlflow_predict_filters
• path_priority (default order: STOP, CONTINUE, REPEAT)
• save_transform
• prediction_store_policy
• model_config.target
• datetime_columns (when query returns temporal fields)

Optional outputs:

• opc_output_config
• pi_web_api_output_config