sientia-dataops-laborious_temporal/e2e/scenarios.md

# E2E Scenario Documentation - Predictions Batch

This document describes the end-to-end scenarios for `predictions_batch` and its child workflows:
`prediction_process` and `format_and_export_prediction`.

It is a functional reference of scenario behavior, inputs, and expected outcomes.

## Execution Context

- Tests run under `e2e/` and are marked with `@pytest.mark.integration`.
- PostgreSQL and MinIO are provisioned with testcontainers.
- `test_minio_offload.py` uses real MinIO I/O; other scenario suites may use stubs/mocks for optional outputs.

---

## 1. Main Workflow Scenarios
Source: `e2e/test_predictions_batch_main_workflow.py`

### 1.1.1 Happy Path - Complete Success
**Summary**: Full workflow succeeds with valid query and default gate behavior.

**Description**:
- Query returns rows for a model.
- `prediction_process` runs transform and predict paths.
- Final prediction and transformed data are persisted.

**Expected Outcome**:
- Exactly one prediction row is created.
- Transform rows are created.
- Confidence/status/comments are success values.

### 1.2.1 SQL Query Execution Error
**Summary**: Invalid SQL leads to no persisted prediction.

**Description**:
- Input query is invalid.
- Load step fails and workflow follows error/short-circuit path.

**Expected Outcome**:
- No prediction rows for the model.
- Workflow does not require retry-loop assumptions in assertions.

### 1.2.2 Missing Required Parameters
**Summary**: Missing required fields prevent workflow completion path.

**Description**:
- Required input key (e.g. `query`) is omitted.
- Workflow fails to produce actionable input for child flow.

**Expected Outcome**:
- No prediction rows are persisted.
- Workflow handle may require explicit terminate in E2E harness.

### 1.2.3 Invalid Datetime Column Specification (de-prioritized)
**Summary**: Legacy invalid datetime-column case is retained only as low-priority legacy coverage.

**Description**:
- `datetime_columns` references non-existing columns.
- Behavior may vary by query shape and parser fallback.

**Expected Outcome**:
- No predictions persisted in the covered legacy assertion path.
- Scenario is not considered primary behavior coverage.

---

## 2. Prediction Process Scenarios
Source: `e2e/test_predictions_batch_prediction_process.py`

### 2.1 Input Gate Path Decisions

#### 2.1.1 CONTINUE
**Summary**: Input filter flags quality issue but allows continuation via default path.

**Description**:
- Input gate returns `CONTINUE`.
- MLFlow transform/predict are skipped.
- Export path persists default-style prediction with warning context.

#### 2.1.2 STOP
**Summary**: Input filter blocks processing.

**Description**:
- Input gate returns `STOP`.
- Workflow exits without export.

#### 2.1.3 REPEAT with history
**Summary**: Prior prediction is reused.

**Description**:
- Input gate returns `REPEAT`.
- `repeat_last_prediction` path is executed using existing historical row.

#### 2.1.4 REPEAT without history
**Summary**: Repeat requested but no previous prediction exists.

**Description**:
- Input gate returns `REPEAT`.
- No prior row is available to duplicate.

**Expected Outcome**:
- No new prediction rows are created for the model.

### 2.2 Transform Gate Decisions

#### 2.2.1 CONTINUE on transform response error
**Summary**: Transform response is degraded, but workflow continues.

#### 2.2.2 STOP on transform response error
**Summary**: Transform response error blocks downstream processing.

#### 2.2.3 REPEAT on transform response error
**Summary**: Transform response error triggers repeat-last-prediction path.

#### 2.2.4 STOP on transform content NaN
**Summary**: Content gate (`NAN_VALUES`) blocks on all-NaN transform payload.

### 2.3 Predict Gate Decisions

#### 2.3.1 CONTINUE on predict response error
**Summary**: Predict response degraded; workflow exports with degraded metadata.

#### 2.3.2 STOP on predict response error
**Summary**: Predict response error blocks export.

#### 2.3.3 REPEAT on predict response error
**Summary**: Predict response error routes to repeat-last-prediction.

### 2.4.1 Priority Conflict Resolution
**Summary**: Deterministic selection when multiple filters produce different flags.

**Description**:
- Multiple filters may produce `STOP`, `CONTINUE`, and/or `REPEAT`.
- `path_priority` defines precedence.

**Expected Outcome**:
- Highest-priority flag is applied consistently.
- Executed branch matches configured priority ordering.

---

## 3. Format and Export Scenarios
Source: `e2e/test_predictions_batch_format_export.py`

### 3.1 Output Combination Scenarios

#### 3.1.1 Default prediction export
**Summary**: Non-`None` path flag uses `format_default_prediction`.

**Description**:
- Default prediction is generated.
- Transform export is skipped.
- Optional outputs (PI/OPC) still execute when configured.

#### 3.1.2 OPC only
**Summary**: Postgres + OPC writes, PI Web API disabled.

#### 3.1.3 PI Web API only
**Summary**: Postgres + PI writes, OPC disabled.

#### 3.1.4 Postgres only
**Summary**: Both optional outputs disabled; only Postgres persistence and metrics.

#### 3.1.5 No transformed data export
**Summary**: Prediction is persisted; transformed table is not written.

### 3.2 Degraded-but-successful Completion

#### 3.2.1 PI Web API write error
**Summary**: PI write failure does not fail workflow.

**Expected Outcome**:
- Workflow completes.
- Prediction persisted with degraded confidence/comments (PI error semantics).

#### 3.2.2 OPC write error
**Summary**: OPC write failure does not fail workflow.

**Expected Outcome**:
- Workflow completes.
- Prediction persisted with OPC degraded confidence/comments.

#### 3.2.3 PI Web API partial write error
**Summary**: Partial PI acknowledgement is treated as degraded success.

**Expected Outcome**:
- Workflow completes.
- Prediction persisted with PI error confidence and descriptive comment.

### 3.3.1 Combined Optional Outputs (PI + OPC)
**Summary**: Both external output channels are enabled together.

**Description**:
- PI Web API and OPC configs are both present.
- Output mutation order matters for final persisted payload.

**Expected Outcome**:
- PI write executes before OPC write in workflow sequence.
- Final Postgres payload reflects any confidence/comment updates.
- OPC metrics are emitted when tag writes return response times.

---

## 4. MinIO Offload Scenarios
Source: `e2e/test_minio_offload.py`

### 4.1.1 Forced offload to MinIO
**Summary**: Very low threshold forces parquet upload.

**Description**:
- Payload is offloaded (`object_key` present, inline data absent/empty).
- Object is present in MinIO under `prediction_datasets/...`.
- Retrieval reconstructs the dataframe.

### 4.1.2 Full workflow with offloaded load payload
**Summary**: Offload path works during full `predictions_batch` execution.

**Expected Outcome**:
- Workflow completes.
- Prediction row is persisted.

### 4.2.1 Inline payload below threshold
**Summary**: Data remains inline when threshold is not exceeded.

**Expected Outcome**:
- Payload stores inline `data`.
- `object_key` is `None`.
- Downstream persistence behavior matches offload scenario semantics.

---

## 5. Drift Workflow Scenarios
Source: `e2e/test_drift.py`

The drift suite mocks `sientia.ModelAnalysis.ModelAnalysis` (not installed; see
`CODE_ISSUES.md` issue #1) through the controllable `_FakeModelAnalysis` stub
exposed by the `model_analysis_stub` fixture. The `mlflow_repository_stub`
provides the reference-data CSV via `download_artifacts`. Every scenario asserts
postgres rows in `predictions_schema.drift` against this canonical schema:

`id, model_id, feature, method, value, drift, chunk, timestamp, timestamp_end, accurate, created_at, updated_at`.

### 5.1 Happy paths

#### D.1.1 Full pipeline persists all columns with reference data
**Summary**: ModelAnalysis returns a deterministic drift dataframe; the
reference CSV is downloaded from the MLflow stub.

**Expected Outcome**:
- One row per `(chunk, feature, method)` plus a `multivariate` block per chunk.
- Every drift column is populated and `accurate=True`.
- `timestamp_end` preserves the high-precision string (`HH:MM:59.999999999`).
- `p_value` is dropped before persistence.
- `drift` flags propagate per `(feature, method)` configuration.

#### D.1.2 30% fallback when reference data is unavailable
**Summary**: `get_reference_data` fails alias resolution and returns `None`;
`calculate_drift` uses the first 30% of target rows as reference.

**Expected Outcome**:
- Persisted rows carry `accurate=False`.
- A `MODEL_METRICS_REFERENCE_DATA_WARNING` notification is emitted to MongoDB.

### 5.2 Filtering / dedup invariants

#### D.2.1 Deduplication and `p_value` removal
**Summary**: ModelAnalysis returns duplicate `(timestamp, method, feature)` rows
plus a `p_value` column.

**Expected Outcome**:
- Duplicates are collapsed keeping the first occurrence.
- `p_value` is absent from the persisted rows.

#### D.2.2 Out-of-range timestamps filtered
**Summary**: Drift rows whose timestamps are not present in the target window
must be discarded before persistence.

### 5.3 Failure paths

#### D.3.1 Empty target data short-circuits the workflow
**Summary**: `load_custom_query` returns no rows; ModelAnalysis is never
instantiated and no drift rows are written.

#### D.3.2 ModelAnalysis raises during dataframe assembly
**Summary**: `get_drift_metrics_dataframe` raises. The activity catches the
error, sends a `MODEL_METRICS_GET_DRIFT_METRICS_ERROR` notification, and the
workflow completes without persisting drift rows.

### 5.4 Configuration paths

#### D.4.1 Default drift metrics propagated to analyzer
**Summary**: Omitting `drift_metrics` defaults to
`['kolmogorov_smirnov', 'jensen_shannon', 'wasserstein']` and forwards the
exact list to `detect_univariate_drift`.

#### D.4.2 Invalid `chunk_period` raises ValueError
**Summary**: Anything other than `min` / `s` is rejected by `calculate_drift`.

#### D.4.3 `chunk_period='s'` keeps seconds in timestamp filtering
**Summary**: Truncated `YYYY-MM-DD HH:MM` rows are filtered out when chunking
runs at second granularity.

---

## 6. Simple Metrics Workflow Scenarios
Source: `e2e/test_simple_metrics.py`

Validates `predictions_schema.simple_metrics_data` columns:
`id, model_id, metric, value, timestamp, data_size, interval_minutes, created_at`.

### 6.1 Happy paths

#### S.1.1 rmse/mse/mae/r2 happy path
**Summary**: Prediction/target pairs are inserted; the activity computes all
four metrics with closed-form expected values.

**Expected Outcome**:
- One row per metric is persisted; all columns populated.
- `data_size` matches the joined row count and `interval_minutes=60`.

#### S.1.2 Subset metrics
**Summary**: Requesting `metrics=['rmse']` writes only the rmse row.

### 6.2 Edge cases

#### S.2.1 Zero-variance target returns r2=0
**Summary**: When all targets are equal, `ss_tot=0`; the activity must guard
against division by zero and return `r2=0`.

### 6.3 Failure paths

#### S.3.1 No overlapping data short-circuits persistence
**Summary**: With no `laborious_data` rows for the configured target variable
the workflow exits before `calculate_simple_metrics` and writes nothing.

---

## 7. Minimal Retrain Workflow Scenarios
Source: `e2e/test_minimal_retrain.py`

The MLflow registry is fully mocked (no real artifacts in test container).
Validates `predictions_schema.retrain_reports` columns:
`id, model_id, model_name, timestamp, status, version, mlflow_run_id, mlflow_experiment_id, created_at`.

### 7.1 Happy path

#### MR.1.1 Successful retrain + promotion
**Summary**: Training data loads via MinIO offload, `wrapper.retrain` succeeds,
the new version is promoted to the `production` alias.

**Expected Outcome**:
- Report row has success status, `version='7'`, `mlflow_run_id='retrain-run-id'`,
  `mlflow_experiment_id='experiment-id'`.
- `mlflow.log_artifact` is called with the input CSV.
- `promote_to_alias` is called once with the resolved version and alias.

### 7.2 Failure paths

#### MR.2.1 Wrapper retrain raises
**Summary**: `wrapper.retrain` raises `RuntimeError`. The activity returns
`success=False`, `update_production_model` is NOT invoked.

**Expected Outcome**:
- Report row carries the error message and `version`/`mlflow_*` columns are NULL.

#### MR.2.2 Missing `model_config.target`
**Summary**: Empty model config short-circuits before any MLflow call.

**Expected Outcome**:
- Report row carries the explicit guard message.
- `get_cached_model` is never invoked.

#### MR.3.1 No training data
**Summary**: The training query returns no rows; the workflow does not
persist any report row. The current code raises plain `ValueError` from the
workflow function, which Temporal treats as a workflow-task failure (see
`CODE_ISSUES.md` issue MR-1).

---

## Input Contract Reference

Common scenario input fields:
- `schedule_name`
- `model_name`
- `model_id`
- `query`
- `schema`
- `table_name`
- `transform_table_name`
- `input_filters`
- `mlflow_transform_filters`
- `mlflow_predict_filters`
- `path_priority` (default order: `STOP`, `CONTINUE`, `REPEAT`)
- `save_transform`
- `prediction_store_policy`
- `model_config.target`
- `datetime_columns` (when query returns temporal fields)

Optional outputs:
- `opc_output_config`
- `pi_web_api_output_config`