sientia-dataops-laborious_temporal/e2e/scenarios.md

# E2E Scenario Documentation - Predictions Batch

This document describes the end-to-end scenarios for `predictions_batch` and its child workflows:
`prediction_process` and `format_and_export_prediction`.

It is a functional reference of scenario behavior, inputs, and expected outcomes.

## Execution Context

- Tests run under `e2e/` and are marked with `@pytest.mark.integration`.
- PostgreSQL and MinIO are provisioned with testcontainers.
- `test_minio_offload.py` uses real MinIO I/O; other scenario suites may use stubs/mocks for optional outputs.

---

## 1. Main Workflow Scenarios
Source: `e2e/test_predictions_batch_main_workflow.py`

### 1.1.1 Happy Path - Complete Success
**Summary**: Full workflow succeeds with valid query and default gate behavior.

**Description**:
- Query returns rows for a model.
- `prediction_process` runs transform and predict paths.
- Final prediction and transformed data are persisted.

**Expected Outcome**:
- Exactly one prediction row is created.
- Transform rows are created.
- Confidence/status/comments are success values.

### 1.2.1 SQL Query Execution Error
**Summary**: Invalid SQL leads to no persisted prediction.

**Description**:
- Input query is invalid.
- Load step fails and workflow follows error/short-circuit path.

**Expected Outcome**:
- No prediction rows for the model.
- Workflow does not require retry-loop assumptions in assertions.

### 1.2.2 Missing Required Parameters
**Summary**: Missing required fields prevent workflow completion path.

**Description**:
- Required input key (e.g. `query`) is omitted.
- Workflow fails to produce actionable input for child flow.

**Expected Outcome**:
- No prediction rows are persisted.
- Workflow handle may require explicit terminate in E2E harness.

### 1.2.3 Invalid Datetime Column Specification (de-prioritized)
**Summary**: Legacy invalid datetime-column case is retained only as low-priority legacy coverage.

**Description**:
- `datetime_columns` references non-existing columns.
- Behavior may vary by query shape and parser fallback.

**Expected Outcome**:
- No predictions persisted in the covered legacy assertion path.
- Scenario is not considered primary behavior coverage.

---

## 2. Prediction Process Scenarios
Source: `e2e/test_predictions_batch_prediction_process.py`

### 2.1 Input Gate Path Decisions

#### 2.1.1 CONTINUE
**Summary**: Input filter flags quality issue but allows continuation via default path.

**Description**:
- Input gate returns `CONTINUE`.
- MLFlow transform/predict are skipped.
- Export path persists default-style prediction with warning context.

#### 2.1.2 STOP
**Summary**: Input filter blocks processing.

**Description**:
- Input gate returns `STOP`.
- Workflow exits without export.

#### 2.1.3 REPEAT with history
**Summary**: Prior prediction is reused; outcome depends on batch vs history timestamp.

**Description**:
- Input gate returns `REPEAT`.
- `repeat_last_prediction` inserts a row using ``last_timestamp`` from the batch payload (max timestamp in `laborious_data` for the query), not the copied row’s timestamp.

**Tests**:
- **Collision**: batch `last_timestamp` equals the historical prediction row’s `timestamp` → Postgres `unique_model_id_timestamp` violation; workflow fails; still one row.
- **Distinct batch time**: laborious rows are stamped later than the historical prediction → second row inserted; same prediction fields as the first (see `assert_repeat`).

#### 2.1.4 REPEAT without history
**Summary**: Repeat requested but no previous prediction exists.

**Description**:
- Input gate returns `REPEAT`.
- No prior row is available to duplicate.

**Expected Outcome**:
- No new prediction rows are created for the model.

### 2.2 Transform Gate Decisions

#### 2.2.1 CONTINUE on transform response error
**Summary**: Transform response is degraded, but workflow continues.

#### 2.2.2 STOP on transform response error
**Summary**: Transform response error blocks downstream processing.

#### 2.2.3 REPEAT on transform response error
**Summary**: Transform response error triggers repeat-last-prediction path.

**Tests**: Same timestamp collision vs distinct batch timestamp as §2.1.3 (`*_fails` / `*_inserts_second_row`).

#### 2.2.4 STOP on transform content NaN
**Summary**: Content gate (`NAN_VALUES`) blocks on all-NaN transform payload.

### 2.3 Predict Gate Decisions

#### 2.3.1 CONTINUE on predict response error
**Summary**: Predict response degraded; workflow exports with degraded metadata.

#### 2.3.2 STOP on predict response error
**Summary**: Predict response error blocks export.

#### 2.3.3 REPEAT on predict response error
**Summary**: Predict response error routes to repeat-last-prediction.

**Tests**: Same timestamp collision vs distinct batch timestamp as §2.1.3 (`*_fails` / `*_inserts_second_row`).

### 2.4.1 Priority Conflict Resolution
**Summary**: Deterministic selection when multiple filters produce different flags.

**Description**:
- Multiple filters may produce `STOP`, `CONTINUE`, and/or `REPEAT`.
- `path_priority` defines precedence.

**Expected Outcome**:
- Highest-priority flag is applied consistently.
- Executed branch matches configured priority ordering.

---

## 3. Format and Export Scenarios
Source: `e2e/test_predictions_batch_format_export.py`

### 3.1 Output Combination Scenarios

#### 3.1.1 Default prediction export
**Summary**: Non-`None` path flag uses `format_default_prediction`.

**Description**:
- Default prediction is generated.
- Transform export is skipped.
- Optional outputs (PI/OPC) still execute when configured.

#### 3.1.2 OPC only
**Summary**: Postgres + OPC writes, PI Web API disabled.

#### 3.1.3 PI Web API only
**Summary**: Postgres + PI writes, OPC disabled.

#### 3.1.4 Postgres only
**Summary**: Both optional outputs disabled; only Postgres persistence and metrics.

#### 3.1.5 No transformed data export
**Summary**: Prediction is persisted; transformed table is not written.

### 3.2 Degraded-but-successful Completion

#### 3.2.1 PI Web API write error
**Summary**: PI write failure does not fail workflow.

**Expected Outcome**:
- Workflow completes.
- Prediction persisted with degraded confidence/comments (PI error semantics).

#### 3.2.2 OPC write error
**Summary**: OPC write failure does not fail workflow.

**Expected Outcome**:
- Workflow completes.
- Prediction persisted with OPC degraded confidence/comments.

#### 3.2.3 PI Web API partial write error
**Summary**: Partial PI acknowledgement is treated as degraded success.

**Expected Outcome**:
- Workflow completes.
- Prediction persisted with PI error confidence and descriptive comment.

### 3.3.1 Combined Optional Outputs (PI + OPC)
**Summary**: Both external output channels are enabled together.

**Description**:
- PI Web API and OPC configs are both present.
- Output mutation order matters for final persisted payload.

**Expected Outcome**:
- PI write executes before OPC write in workflow sequence.
- Final Postgres payload reflects any confidence/comment updates.
- OPC metrics are emitted when tag writes return response times.

---

## 4. MinIO Offload Scenarios
Source: `e2e/test_minio_offload.py`

### 4.1.1 Forced offload to MinIO
**Summary**: Very low threshold forces parquet upload.

**Description**:
- Payload is offloaded (`object_key` present, inline data absent/empty).
- Object is present in MinIO under `prediction_datasets/...`.
- Retrieval reconstructs the dataframe.

### 4.1.2 Full workflow with offloaded load payload
**Summary**: Offload path works during full `predictions_batch` execution.

**Expected Outcome**:
- Workflow completes.
- Prediction row is persisted.

### 4.2.1 Inline payload below threshold
**Summary**: Data remains inline when threshold is not exceeded.

**Expected Outcome**:
- Payload stores inline `data`.
- `object_key` is `None`.
- Downstream persistence behavior matches offload scenario semantics.

---

## 5. Drift Workflow Scenarios
Source: `e2e/test_drift.py`

The drift suite drives the **real** `sientia_model.analytics.drift_analysis.DriftAnalysis`
analyzer (no stubs / mocks). Each scenario exercises the full pipeline:

```
laborious_data (Postgres) -> load_custom_query
                          -> calculate_drift (DriftAnalysis univariate + multivariate)
                          -> export_data_to_postgres (sientia_data.drift_metrics)
```

The `mlflow_repository_stub` provides the reference-data CSV via
`download_artifacts`, and tests assert postgres rows in
`sientia_data.drift_metrics` against this canonical schema:

`id, model_id, feature, method, value, alert, chunk_index, chunk_start_date, chunk_end_date, accurate, timestamp, created_at`.

Tests assert behavioral / structural properties (column presence, NOT NULL
constraints, business-key invariants like uniform `timestamp` and stamped
`model_id`) rather than exact numeric drift scores, since those depend on
the real analyzer implementation and the synthetic data fed in.

### 5.1 Happy paths

#### D.1.1 Full pipeline persists all columns with reference data
**Summary**: 10 minutes of target data are inserted; a 10-row reference CSV
is configured via the MLflow stub. The `DriftAnalysis` runs end-to-end.

**Expected Outcome**:
- One row per `(chunk_index, feature, method)` plus a `multivariate` block
  per chunk is persisted.
- Every column in the DDL is populated; `feature` is the only nullable column
  per the new schema.
- `accurate=True` for every row (reference path).
- All three default univariate methods reach the analyzer.
- `model_id` is stamped as `text` and uniform across rows.
- `timestamp` equals `max(target_data.timestamp)` and is uniform across rows.
- `chunk_start_date` / `chunk_end_date` are persisted as ISO text and ordered.
- `p_value` is dropped before persistence.

#### D.1.2 30% fallback when reference data is unavailable
**Summary**: MLflow alias resolution is forced to fail so
`get_reference_data` returns `None`; `calculate_drift` falls back to the
first 30% of target rows as reference.

**Expected Outcome**:
- All persisted rows carry `accurate=False`.
- A `MODEL_METRICS_REFERENCE_DATA_WARNING` notification is emitted to MongoDB.

### 5.2 Failure paths

#### D.3.1 Empty target data short-circuits the workflow
**Summary**: `load_custom_query` returns no rows.

**Expected Outcome**:
- The workflow returns early and writes nothing to `sientia_data.drift_metrics`.

### 5.3 Configuration paths

#### D.4.2 Invalid `chunk_period` raises ValueError
**Summary**: Anything other than `min` / `s` is rejected by `calculate_drift`.

**Expected Outcome**:
- The workflow surfaces the `ValueError` ("Invalid chunk period: ...").
- No rows are persisted.

#### D.4.3a Insufficient drift metrics while target has rows
**Summary**: Laborious raises when the merged drift table is empty but the
target window is non-empty (`Insufficient drift data:` + notification
`MODEL_METRICS_DRIFT_INSUFFICIENT_DATA`).

**Description**:
- The e2e patches `ModelMetrics.get_drift_metrics` to return an empty
  DataFrame, simulating a ``sientia_model`` path that emits no rows.

**Expected Outcome**:
- Workflow fails; no rows in `sientia_data.drift_metrics`.

#### D.4.3b `chunk_period='s'` preserves seconds in `chunk_start_date`
**Summary**: Target data includes sub-minute spacing across several minutes;
the activity uses `chunk_period='s'`.

**Expected Outcome**:
- At least one persisted `chunk_start_date` carries `seconds=30`, proving
  that the analyzer chunked at sub-minute granularity and the ISO-text
  serialization preserved the boundary.

**Note**: The e2e patches `DriftAnalysis._chunk_dataframe` to **skip empty**
`pd.Grouper(freq='s')` buckets. The stock implementation iterates every
second between min/max timestamps, producing empty chunks and NaT rows that
`calculate_drift` filters away entirely. The durable fix belongs in
`sientia_model`.

---

## 6. Simple Metrics Workflow Scenarios
Source: `e2e/test_simple_metrics.py`

Validates `sientia_data.simple_metrics` columns:
`id, model_id, metric, value, timestamp, data_size, interval_minutes, created_at`.
Note: ``timestamp`` is now nullable per the new DDL and ``model_id`` is ``text``.

### 6.1 Happy paths

#### S.1.1 rmse/mse/mae/r2 happy path
**Summary**: Prediction/target pairs are inserted; the activity computes all
four metrics with closed-form expected values.

**Expected Outcome**:
- One row per metric is persisted; all columns populated.
- `data_size` matches the joined row count and `interval_minutes=60`.

#### S.1.2 Subset metrics
**Summary**: Requesting `metrics=['rmse']` writes only the rmse row.

### 6.2 Edge cases

#### S.2.1 Zero-variance target returns r2=0
**Summary**: When all targets are equal, `ss_tot=0`; the activity must guard
against division by zero and return `r2=0`.

### 6.3 Failure paths

#### S.3.1 No overlapping data short-circuits persistence
**Summary**: With no `laborious_data` rows for the configured target variable
the workflow exits before `calculate_simple_metrics` and writes nothing.

---

## 7. Minimal Retrain Workflow Scenarios
Source: `e2e/test_minimal_retrain.py`

The MLflow registry is fully mocked (no real artifacts in test container).
Validates `sientia_data.log_retrain` columns:
`mlflow_experiment_id, mlflow_run_id, model_id, model_name, status, timestamp, version`.
Note: the new DDL drops the legacy ``id`` and ``created_at`` columns,
``mlflow_experiment_id`` is now ``int8`` and ``model_id`` is ``text``.

### 7.1 Happy path

#### MR.1.1 Successful retrain + promotion
**Summary**: Training data loads via MinIO offload, `wrapper.retrain` succeeds,
the new version is promoted to the `production` alias.

**Expected Outcome**:
- Report row has success status, `version='7'`, `mlflow_run_id='retrain-run-id'`,
  `mlflow_experiment_id=4242` (`int8`).
- `mlflow.log_artifact` is called with the input CSV.
- `promote_to_alias` is called once with the resolved version and alias.

### 7.2 Failure paths

#### MR.2.1 Wrapper retrain raises
**Summary**: `wrapper.retrain` raises `RuntimeError`. The activity returns
`success=False`, `update_production_model` is NOT invoked.

**Expected Outcome**:
- Report row carries the error message and `version`/`mlflow_*` columns are NULL.

#### MR.2.2 Missing `model_config.target`
**Summary**: Empty model config short-circuits before any MLflow call.

**Expected Outcome**:
- Report row carries the explicit guard message.
- `get_cached_model` is never invoked.

#### MR.3.1 No training data
**Summary**: The training query returns no rows; the workflow does not
persist any report row. The current code raises plain `ValueError` from the
workflow function, which Temporal treats as a workflow-task failure (see
`CODE_ISSUES.md` issue MR-1).

---

## Input Contract Reference

Common scenario input fields:
- `schedule_name`
- `model_name`
- `model_id`
- `query`
- `schema`
- `table_name`
- `transform_table_name`
- `input_filters`
- `mlflow_transform_filters`
- `mlflow_predict_filters`
- `path_priority` (default order: `STOP`, `CONTINUE`, `REPEAT`)
- `save_transform`
- `prediction_store_policy`
- `model_config.target`
- `datetime_columns` (when query returns temporal fields)

Optional outputs:
- `opc_output_config`
- `pi_web_api_output_config`