sientia-dataops-laborious_temporal/e2e/scenarios.md

# Test Scenarios for Predictions Batch Workflow

This document describes all possible test scenarios for the `predictions_batch` workflow and its child workflows `prediction_process` and `format_and_export_prediction`.

## Workflow Overview

The `predictions_batch` workflow:
1. Loads data using a custom SQL query
2. Prepares prediction configuration
3. Delegates to `prediction_process` child workflow which:
   - Retrieves last timestamp for incremental processing
   - Applies input data quality gates
   - Executes MLFlow transform operation
   - Validates transform response
   - Executes MLFlow predict operation
   - Validates predict response
   - Delegates to `format_and_export_prediction` child workflow
4. The `format_and_export_prediction` workflow:
   - Formats prediction data (normal or default)
   - Exports to PI Web API (optional)
   - Exports to OPC server (optional)
   - Exports to PostgreSQL
   - Writes metrics

---

## 1. Predictions Batch - Main Workflow Scenarios

### 1.1 Success Scenarios

#### Scenario 1.1.1: Happy Path - Complete Success
**Description**: Workflow completes successfully with valid SQL query and all activities succeed

**Input**:
- Valid `schedule_name`, `model_name`, `model_id`
- Valid `query` returning non-empty DataFrame
- Valid `schema`, `table_name`, `transform_table_name`
- Optional `datetime_columns` for timestamp parsing
- Optional `input_filters`, `mlflow_transform_filters`, `mlflow_predict_filters`
- Optional `path_priority`, `opc_output_config`, `pi_web_api_output_config`

**Expected Behavior**:
- `load_custom_query` returns DataFrame with data
- Workflow prepares prediction input with all configurations
- `prediction_process` child workflow executes successfully
- All gates pass with no issues
- Transform and predict operations succeed
- Data exported to PostgreSQL
- Metrics written

**Assertions**:
- SQL query executed once
- `prediction_process` workflow called with correct parameters
- Data exists in PostgreSQL (predictions table)
- Metrics recorded
- No errors raised

---

### 1.2 Error Scenarios

#### Scenario 1.2.1: SQL Query Execution Error
**Description**: SQL query fails due to syntax error or connection issue

**Input**:
- Invalid SQL query (syntax error)
- Or database connection unavailable

**Expected Behavior**:
- `load_custom_query` raises exception (caught by Temporal retry policy)
- Notification sent with SQL error details
- After retries, activity may return empty data or workflow may fail
- If empty data returned, workflow completes with early exit via input gate

**Assertions**:
- Error notification sent
- Workflow completes (either fails or exits early)
- No data in predictions table

---

#### Scenario 1.2.2: Missing Required Parameters
**Description**: Essential parameters missing from input

**Input**:
- Missing `query` or `model_id` or `schema` or `table_name`

**Expected Behavior**:
- Workflow or activity raises KeyError or validation error
- Workflow fails immediately

**Assertions**:
- Workflow fails with parameter error
- Error notification sent
- No child workflow called

---

#### Scenario 1.2.3: Invalid Datetime Column Specification
**Description**: Datetime column specified doesn't exist in query results

**Input**:
- `datetime_columns: ['nonexistent_column']`
- Query results don't have this column

**Expected Behavior**:
- `load_custom_query` may raise KeyError or warning
- Depending on implementation, workflow may fail or continue
- Error notification sent

**Assertions**:
- Error raised or warning logged
- Workflow behavior depends on error handling policy

---

## 2. Prediction Process - Child Workflow Scenarios

### 2.1 Input gate Early Exit Scenarios

#### Scenario 2.1.1: Input Gate Triggers CONTINUE
**Description**: Input gate determines data should use previous prediction

**Input**:
- Data that should continue with input data as prediction
- `input_filters` configured with `POLICY: 'CONTINUE'`
- `path_priority` includes CONTINUE

**Expected Behavior**:
- `input_gate` returns `path_flag='CONTINUE'`
- `path_flag_handler` calls export workflow with input data directly
- MLFlow transform and predict skipped
- Data exported as-is

**Assertions**:
- `input_gate` called
- MLFlow operations NOT called
- Export workflow called with original data
- Workflow completes


#### Scenario 2.1.2: Input Gate Triggers STOP
**Description**: Input data quality gate fails with STOP policy

**Input**:
- Data with EMPTY_DATA or other critical issues
- `input_filters` configured with `POLICY: 'STOP'`

**Expected Behavior**:
- `input_gate` returns `path_flag='STOP'`
- `path_flag_handler` detects STOP
- Workflow returns early without calling MLFlow
- No prediction exported

**Assertions**:
- `input_gate` called
- `path_flag_handler` returns True (early exit)
- MLFlow transform NOT called
- Export workflow NOT called
- Workflow completes without error


#### Scenario 2.1.3: Input Gate Triggers REPEAT
**Description**: Input gate determines data should repeat last prediction

**Input**:
- Data with quality issues that require using previous prediction
- `input_filters` configured with `POLICY: 'REPEAT'`
- `path_priority` includes REPEAT

**Expected Behavior**:
- `input_gate` returns `path_flag='REPEAT'`
- `path_flag_handler` calls `repeat_last_prediction` activity
- MLFlow transform and predict skipped
- Last prediction repeated and exported

**Assertions**:
- `input_gate` called
- MLFlow operations NOT called
- `repeat_last_prediction` activity called
- Workflow completes

---

### 2.2 Transform gate Early Exit Scenarios

#### Scenario 2.2.1: Transform Gate Triggers CONTINUE
**Description**: Transform response gate determines data should continue despite issues

**Input**:
- Valid input data
- Transform response has quality issues but policy is CONTINUE
- `mlflow_transform_filters` configured with `POLICY: 'CONTINUE'`
- `path_priority` includes CONTINUE

**Expected Behavior**:
- `request_transform` succeeds
- `mlflow_response_gate` for transform returns `path_flag='CONTINUE'`
- `path_flag_handler` calls export workflow with transform data
- MLFlow predict skipped
- Transform data exported as-is

**Assertions**:
- Transform completed
- `mlflow_response_gate` called for transform
- MLFlow predict NOT called
- Export workflow called with transform data
- Workflow completes

---

#### Scenario 2.2.2: Transform Gate Triggers STOP
**Description**: Transform response validation fails with STOP policy

**Input**:
- Valid input data
- Transform response has critical errors
- `mlflow_transform_filters` configured with `POLICY: 'STOP'`

**Expected Behavior**:
- `request_transform` succeeds but response invalid
- `mlflow_response_gate` for transform returns `path_flag='STOP'`
- Workflow exits without calling predict or export

**Assertions**:
- Transform completed but validation failed
- `mlflow_response_gate` called for transform
- MLFlow predict NOT called
- Export workflow NOT called
- Workflow completes without error

---

#### Scenario 2.2.3: Transform Gate Triggers REPEAT
**Description**: Transform response gate determines data should repeat last prediction

**Input**:
- Valid input data
- Transform response has quality issues that require using previous prediction
- `mlflow_transform_filters` configured with `POLICY: 'REPEAT'`
- `path_priority` includes REPEAT

**Expected Behavior**:
- `request_transform` succeeds but response has issues
- `mlflow_response_gate` for transform returns `path_flag='REPEAT'`
- `path_flag_handler` calls `repeat_last_prediction` activity
- MLFlow predict skipped
- Last prediction repeated and exported

**Assertions**:
- Transform completed but validation triggered REPEAT
- `mlflow_response_gate` called for transform
- MLFlow predict NOT called
- `repeat_last_prediction` activity called
- Workflow completes

---

### 2.3 Predict gate Early Exit Scenarios

#### Scenario 2.3.1: Predict Gate Triggers CONTINUE
**Description**: Predict response gate determines data should continue despite issues

**Input**:
- Valid input and transform data
- Predict response has quality issues but policy is CONTINUE
- `mlflow_predict_filters` configured with `POLICY: 'CONTINUE'`
- `path_priority` includes CONTINUE

**Expected Behavior**:
- `request_predict` succeeds
- `mlflow_response_gate` for predict returns `path_flag='CONTINUE'`
- `path_flag_handler` calls export workflow with predict data
- Prediction exported despite quality issues

**Assertions**:
- Transform and predict completed
- `mlflow_response_gate` called for predict
- Export workflow called with predict data
- Workflow completes

---

#### Scenario 2.3.2: Predict Gate Triggers STOP
**Description**: Prediction validation fails with STOP policy

**Input**:
- Valid input and transform
- Predict response has critical errors
- `mlflow_predict_filters` configured with `POLICY: 'STOP'`

**Expected Behavior**:
- `request_predict` succeeds but response invalid
- `mlflow_response_gate` for predict returns `path_flag='STOP'`
- Workflow exits without export

**Assertions**:
- Transform completed
- Predict completed but validation failed
- Export workflow NOT called
- Workflow completes without error

---

#### Scenario 2.3.3: Predict Gate Triggers REPEAT
**Description**: Predict response gate determines data should repeat last prediction

**Input**:
- Valid input and transform data
- Predict response has quality issues that require using previous prediction
- `mlflow_predict_filters` configured with `POLICY: 'REPEAT'`
- `path_priority` includes REPEAT

**Expected Behavior**:
- `request_predict` succeeds but response has issues
- `mlflow_response_gate` for predict returns `path_flag='REPEAT'`
- `path_flag_handler` calls `repeat_last_prediction` activity
- Last prediction repeated and exported

**Assertions**:
- Transform and predict completed but validation triggered REPEAT
- `mlflow_response_gate` called for predict
- `repeat_last_prediction` activity called
- Export workflow NOT called with current prediction
- Workflow completes

---

### 2.4 Error Scenarios

#### Scenario 2.4.1: MLFlow Transform API Error
**Description**: MLFlow transform request fails

**Input**:
- Valid input data
- MLFlow service unavailable or returns error

**Expected Behavior**:
- `request_transform` raises exception
- Notification sent with MLFlow error details
- Workflow fails after retry attempts

**Assertions**:
- Exception raised from transform activity
- Error notification sent
- Workflow fails
- Export NOT called

---

#### Scenario 2.4.2: MLFlow Predict API Error
**Description**: MLFlow predict request fails

**Input**:
- Valid input and transform data
- MLFlow predict service unavailable

**Expected Behavior**:
- `request_predict` raises exception
- Notification sent
- Workflow fails after retries

**Assertions**:
- Transform succeeded
- Predict raised exception
- Error notification sent
- Workflow fails

---

## 3. Format and Export Prediction - Child Workflow Scenarios

### 3.1 Success Scenarios

#### Scenario 3.1.1: Default Prediction Export
**Description**: Error prediction path creates default prediction

**Input**:
- `path_flag: 'STOP'` or other non-None value
- `comment` provided with error details

**Expected Behavior**:
- `format_default_prediction` called instead of `format_prediction`
- Default prediction created with error metadata
- Exported to PostgreSQL only
- Transformed data NOT processed
- Metrics written

**Assertions**:
- `format_default_prediction` called
- `format_prediction` NOT called
- `format_transformed_data` NOT called
- One PostgreSQL export only
- Default values in prediction data
- Comment included

---

#### Scenario 3.1.2: Export Without Optional Outputs
**Description**: Export only to PostgreSQL (no OPC or PI Web API)

**Input**:
- `path_flag: None`
- `opc_output_config: None` or `{}`
- `pi_web_api_output_config: None` or `{}`

**Expected Behavior**:
- Normal formatting
- Only PostgreSQL export executed
- OPC and PI Web API activities skipped
- Metrics written without OPC metrics

**Assertions**:
- PI Web API activity NOT called
- OPC activity NOT called
- PostgreSQL export called
- Metrics written with empty `opc_metrics`

---

#### Scenario 3.1.3: Export Without Transformed Data
**Description**: Only prediction exported, no transform table

**Input**:
- `path_flag: None`
- `transformed_data: None`

**Expected Behavior**:
- Only prediction formatted and exported
- Transform export skipped
- Single PostgreSQL write

**Assertions**:
- `format_transformed_data` NOT called
- One PostgreSQL export
- Transform table remains empty

---

### 3.2 Error Scenarios

#### Scenario 3.2.1: PostgreSQL Export Error - Predictions Table
**Description**: Failed to write predictions to database

**Input**:
- Valid formatted prediction
- PostgreSQL connection fails or table doesn't exist

**Expected Behavior**:
- `export_data_to_postgres` raises exception
- Notification sent with database error
- Workflow fails after retries

**Assertions**:
- Exception raised from export activity
- Error notification sent
- Workflow fails
- Metrics NOT written (activity doesn't execute)

---

#### Scenario 3.2.2: PI Web API Write Error
**Description**: PI Web API export fails

**Input**:
- Valid prediction
- PI Web API service unavailable or invalid config

**Expected Behavior**:
- `write_pi_web_api_data` raises exception
- Notification sent
- Workflow fails after retries
- PostgreSQL export may not execute (depends on execution order)

**Assertions**:
- PI Web API error notification sent
- Workflow fails
- May impact subsequent exports

---

#### Scenario 3.2.3: OPC Write Error
**Description**: OPC server write fails

**Input**:
- Valid prediction
- OPC server unavailable or invalid configuration

**Expected Behavior**:
- `write_opc_data` raises exception
- Notification sent
- Workflow fails after retries

**Assertions**:
- OPC error notification sent
- Workflow fails
- PostgreSQL export may not execute

---

## 4. End-to-End Integration Scenarios

### 4.1 Complete Success Path

#### Scenario 4.1.1: Full Pipeline Success with All Features
**Description**: Complete workflow execution with all optional features enabled

**Input**:
- Valid SQL query returning data
- All configurations provided (OPC, PI Web API, filters, policies)
- MLFlow services available
- All databases available

**Expected Behavior**:
- SQL query loads data
- Input gate passes
- MLFlow transform succeeds
- MLFlow predict succeeds
- All validations pass
- Prediction formatted
- Transformed data formatted
- Both exported to PostgreSQL
- PI Web API write succeeds
- OPC write succeeds
- Metrics written

**Assertions**:
- All activities executed in correct order
- All three workflows execute (batch, process, export)
- All exports succeed
- All tables have data
- All external systems updated
- Metrics recorded

---

### 4.2 Error Recovery Integration

#### Scenario 4.2.1: Transform Error with Repeat Fallback
**Description**: Transform fails, workflow repeats last prediction

**Input**:
- Valid input
- MLFlow transform fails
- REPEAT policy configured
- Previous prediction exists

**Expected Behavior**:
- Transform fails
- Filter detects error
- Path handler triggers REPEAT
- Last prediction retrieved and re-exported
- Workflow completes successfully

**Assertions**:
- Transform attempted
- Error handled gracefully
- Last prediction copied
- Workflow completes without exception

---