SIENTIAPDE-1478
Update coverage source in pyproject.toml, add testcontainers for PostgreSQL in requirements-dev.txt, increment image tag and adjust probe delays in values.yaml, and refine condition checks in format_and_export_prediction.py and mlflow.py. Additionally, enhance test coverage in test_gates.py.
This commit is contained in:
vitor-aignosi
561
e2e/scenarios.md
Normal file
561
e2e/scenarios.md
Normal file
@@ -0,0 +1,561 @@
|
||||
# Test Scenarios for Predictions Batch Workflow
|
||||
|
||||
This document describes all possible test scenarios for the `predictions_batch` workflow and its child workflows `prediction_process` and `format_and_export_prediction`.
|
||||
|
||||
## Workflow Overview
|
||||
|
||||
The `predictions_batch` workflow:
|
||||
1. Loads data using a custom SQL query
|
||||
2. Prepares prediction configuration
|
||||
3. Delegates to `prediction_process` child workflow which:
|
||||
- Retrieves last timestamp for incremental processing
|
||||
- Applies input data quality gates
|
||||
- Executes MLFlow transform operation
|
||||
- Validates transform response
|
||||
- Executes MLFlow predict operation
|
||||
- Validates predict response
|
||||
- Delegates to `format_and_export_prediction` child workflow
|
||||
4. The `format_and_export_prediction` workflow:
|
||||
- Formats prediction data (normal or default)
|
||||
- Exports to PI Web API (optional)
|
||||
- Exports to OPC server (optional)
|
||||
- Exports to PostgreSQL
|
||||
- Writes metrics
|
||||
|
||||
---
|
||||
|
||||
## 1. Predictions Batch - Main Workflow Scenarios
|
||||
|
||||
### 1.1 Success Scenarios
|
||||
|
||||
#### Scenario 1.1.1: Happy Path - Complete Success
|
||||
**Description**: Workflow completes successfully with valid SQL query and all activities succeed
|
||||
|
||||
**Input**:
|
||||
- Valid `schedule_name`, `model_name`, `model_id`
|
||||
- Valid `query` returning non-empty DataFrame
|
||||
- Valid `schema`, `table_name`, `transform_table_name`
|
||||
- Optional `datetime_columns` for timestamp parsing
|
||||
- Optional `input_filters`, `mlflow_transform_filters`, `mlflow_predict_filters`
|
||||
- Optional `path_priority`, `opc_output_config`, `pi_web_api_output_config`
|
||||
|
||||
**Expected Behavior**:
|
||||
- `load_custom_query` returns DataFrame with data
|
||||
- Workflow prepares prediction input with all configurations
|
||||
- `prediction_process` child workflow executes successfully
|
||||
- All gates pass with no issues
|
||||
- Transform and predict operations succeed
|
||||
- Data exported to PostgreSQL
|
||||
- Metrics written
|
||||
|
||||
**Assertions**:
|
||||
- SQL query executed once
|
||||
- `prediction_process` workflow called with correct parameters
|
||||
- Data exists in PostgreSQL (predictions table)
|
||||
- Metrics recorded
|
||||
- No errors raised
|
||||
|
||||
---
|
||||
|
||||
### 1.2 Error Scenarios
|
||||
|
||||
#### Scenario 1.2.1: SQL Query Execution Error
|
||||
**Description**: SQL query fails due to syntax error or connection issue
|
||||
|
||||
**Input**:
|
||||
- Invalid SQL query (syntax error)
|
||||
- Or database connection unavailable
|
||||
|
||||
**Expected Behavior**:
|
||||
- `load_custom_query` raises exception (caught by Temporal retry policy)
|
||||
- Notification sent with SQL error details
|
||||
- After retries, activity may return empty data or workflow may fail
|
||||
- If empty data returned, workflow completes with early exit via input gate
|
||||
|
||||
**Assertions**:
|
||||
- Error notification sent
|
||||
- Workflow completes (either fails or exits early)
|
||||
- No data in predictions table
|
||||
|
||||
---
|
||||
|
||||
#### Scenario 1.2.2: Missing Required Parameters
|
||||
**Description**: Essential parameters missing from input
|
||||
|
||||
**Input**:
|
||||
- Missing `query` or `model_id` or `schema` or `table_name`
|
||||
|
||||
**Expected Behavior**:
|
||||
- Workflow or activity raises KeyError or validation error
|
||||
- Workflow fails immediately
|
||||
|
||||
**Assertions**:
|
||||
- Workflow fails with parameter error
|
||||
- Error notification sent
|
||||
- No child workflow called
|
||||
|
||||
---
|
||||
|
||||
#### Scenario 1.2.3: Invalid Datetime Column Specification
|
||||
**Description**: Datetime column specified doesn't exist in query results
|
||||
|
||||
**Input**:
|
||||
- `datetime_columns: ['nonexistent_column']`
|
||||
- Query results don't have this column
|
||||
|
||||
**Expected Behavior**:
|
||||
- `load_custom_query` may raise KeyError or warning
|
||||
- Depending on implementation, workflow may fail or continue
|
||||
- Error notification sent
|
||||
|
||||
**Assertions**:
|
||||
- Error raised or warning logged
|
||||
- Workflow behavior depends on error handling policy
|
||||
|
||||
---
|
||||
|
||||
## 2. Prediction Process - Child Workflow Scenarios
|
||||
|
||||
### 2.1 Input gate Early Exit Scenarios
|
||||
|
||||
#### Scenario 2.1.1: Input Gate Triggers CONTINUE
|
||||
**Description**: Input gate determines data should use previous prediction
|
||||
|
||||
**Input**:
|
||||
- Data that should continue with input data as prediction
|
||||
- `input_filters` configured with `POLICY: 'CONTINUE'`
|
||||
- `path_priority` includes CONTINUE
|
||||
|
||||
**Expected Behavior**:
|
||||
- `input_gate` returns `path_flag='CONTINUE'`
|
||||
- `path_flag_handler` calls export workflow with input data directly
|
||||
- MLFlow transform and predict skipped
|
||||
- Data exported as-is
|
||||
|
||||
**Assertions**:
|
||||
- `input_gate` called
|
||||
- MLFlow operations NOT called
|
||||
- Export workflow called with original data
|
||||
- Workflow completes
|
||||
|
||||
|
||||
#### Scenario 2.1.2: Input Gate Triggers STOP
|
||||
**Description**: Input data quality gate fails with STOP policy
|
||||
|
||||
**Input**:
|
||||
- Data with EMPTY_DATA or other critical issues
|
||||
- `input_filters` configured with `POLICY: 'STOP'`
|
||||
|
||||
**Expected Behavior**:
|
||||
- `input_gate` returns `path_flag='STOP'`
|
||||
- `path_flag_handler` detects STOP
|
||||
- Workflow returns early without calling MLFlow
|
||||
- No prediction exported
|
||||
|
||||
**Assertions**:
|
||||
- `input_gate` called
|
||||
- `path_flag_handler` returns True (early exit)
|
||||
- MLFlow transform NOT called
|
||||
- Export workflow NOT called
|
||||
- Workflow completes without error
|
||||
|
||||
|
||||
#### Scenario 2.1.3: Input Gate Triggers REPEAT
|
||||
**Description**: Input gate determines data should repeat last prediction
|
||||
|
||||
**Input**:
|
||||
- Data with quality issues that require using previous prediction
|
||||
- `input_filters` configured with `POLICY: 'REPEAT'`
|
||||
- `path_priority` includes REPEAT
|
||||
|
||||
**Expected Behavior**:
|
||||
- `input_gate` returns `path_flag='REPEAT'`
|
||||
- `path_flag_handler` calls `repeat_last_prediction` activity
|
||||
- MLFlow transform and predict skipped
|
||||
- Last prediction repeated and exported
|
||||
|
||||
**Assertions**:
|
||||
- `input_gate` called
|
||||
- MLFlow operations NOT called
|
||||
- `repeat_last_prediction` activity called
|
||||
- Workflow completes
|
||||
|
||||
---
|
||||
|
||||
### 2.2 Transform gate Early Exit Scenarios
|
||||
|
||||
#### Scenario 2.2.1: Transform Gate Triggers CONTINUE
|
||||
**Description**: Transform response gate determines data should continue despite issues
|
||||
|
||||
**Input**:
|
||||
- Valid input data
|
||||
- Transform response has quality issues but policy is CONTINUE
|
||||
- `mlflow_transform_filters` configured with `POLICY: 'CONTINUE'`
|
||||
- `path_priority` includes CONTINUE
|
||||
|
||||
**Expected Behavior**:
|
||||
- `request_transform` succeeds
|
||||
- `mlflow_response_gate` for transform returns `path_flag='CONTINUE'`
|
||||
- `path_flag_handler` calls export workflow with transform data
|
||||
- MLFlow predict skipped
|
||||
- Transform data exported as-is
|
||||
|
||||
**Assertions**:
|
||||
- Transform completed
|
||||
- `mlflow_response_gate` called for transform
|
||||
- MLFlow predict NOT called
|
||||
- Export workflow called with transform data
|
||||
- Workflow completes
|
||||
|
||||
---
|
||||
|
||||
#### Scenario 2.2.2: Transform Gate Triggers STOP
|
||||
**Description**: Transform response validation fails with STOP policy
|
||||
|
||||
**Input**:
|
||||
- Valid input data
|
||||
- Transform response has critical errors
|
||||
- `mlflow_transform_filters` configured with `POLICY: 'STOP'`
|
||||
|
||||
**Expected Behavior**:
|
||||
- `request_transform` succeeds but response invalid
|
||||
- `mlflow_response_gate` for transform returns `path_flag='STOP'`
|
||||
- Workflow exits without calling predict or export
|
||||
|
||||
**Assertions**:
|
||||
- Transform completed but validation failed
|
||||
- `mlflow_response_gate` called for transform
|
||||
- MLFlow predict NOT called
|
||||
- Export workflow NOT called
|
||||
- Workflow completes without error
|
||||
|
||||
---
|
||||
|
||||
#### Scenario 2.2.3: Transform Gate Triggers REPEAT
|
||||
**Description**: Transform response gate determines data should repeat last prediction
|
||||
|
||||
**Input**:
|
||||
- Valid input data
|
||||
- Transform response has quality issues that require using previous prediction
|
||||
- `mlflow_transform_filters` configured with `POLICY: 'REPEAT'`
|
||||
- `path_priority` includes REPEAT
|
||||
|
||||
**Expected Behavior**:
|
||||
- `request_transform` succeeds but response has issues
|
||||
- `mlflow_response_gate` for transform returns `path_flag='REPEAT'`
|
||||
- `path_flag_handler` calls `repeat_last_prediction` activity
|
||||
- MLFlow predict skipped
|
||||
- Last prediction repeated and exported
|
||||
|
||||
**Assertions**:
|
||||
- Transform completed but validation triggered REPEAT
|
||||
- `mlflow_response_gate` called for transform
|
||||
- MLFlow predict NOT called
|
||||
- `repeat_last_prediction` activity called
|
||||
- Workflow completes
|
||||
|
||||
---
|
||||
|
||||
### 2.3 Predict gate Early Exit Scenarios
|
||||
|
||||
#### Scenario 2.3.1: Predict Gate Triggers CONTINUE
|
||||
**Description**: Predict response gate determines data should continue despite issues
|
||||
|
||||
**Input**:
|
||||
- Valid input and transform data
|
||||
- Predict response has quality issues but policy is CONTINUE
|
||||
- `mlflow_predict_filters` configured with `POLICY: 'CONTINUE'`
|
||||
- `path_priority` includes CONTINUE
|
||||
|
||||
**Expected Behavior**:
|
||||
- `request_predict` succeeds
|
||||
- `mlflow_response_gate` for predict returns `path_flag='CONTINUE'`
|
||||
- `path_flag_handler` calls export workflow with predict data
|
||||
- Prediction exported despite quality issues
|
||||
|
||||
**Assertions**:
|
||||
- Transform and predict completed
|
||||
- `mlflow_response_gate` called for predict
|
||||
- Export workflow called with predict data
|
||||
- Workflow completes
|
||||
|
||||
---
|
||||
|
||||
#### Scenario 2.3.2: Predict Gate Triggers STOP
|
||||
**Description**: Prediction validation fails with STOP policy
|
||||
|
||||
**Input**:
|
||||
- Valid input and transform
|
||||
- Predict response has critical errors
|
||||
- `mlflow_predict_filters` configured with `POLICY: 'STOP'`
|
||||
|
||||
**Expected Behavior**:
|
||||
- `request_predict` succeeds but response invalid
|
||||
- `mlflow_response_gate` for predict returns `path_flag='STOP'`
|
||||
- Workflow exits without export
|
||||
|
||||
**Assertions**:
|
||||
- Transform completed
|
||||
- Predict completed but validation failed
|
||||
- Export workflow NOT called
|
||||
- Workflow completes without error
|
||||
|
||||
---
|
||||
|
||||
#### Scenario 2.3.3: Predict Gate Triggers REPEAT
|
||||
**Description**: Predict response gate determines data should repeat last prediction
|
||||
|
||||
**Input**:
|
||||
- Valid input and transform data
|
||||
- Predict response has quality issues that require using previous prediction
|
||||
- `mlflow_predict_filters` configured with `POLICY: 'REPEAT'`
|
||||
- `path_priority` includes REPEAT
|
||||
|
||||
**Expected Behavior**:
|
||||
- `request_predict` succeeds but response has issues
|
||||
- `mlflow_response_gate` for predict returns `path_flag='REPEAT'`
|
||||
- `path_flag_handler` calls `repeat_last_prediction` activity
|
||||
- Last prediction repeated and exported
|
||||
|
||||
**Assertions**:
|
||||
- Transform and predict completed but validation triggered REPEAT
|
||||
- `mlflow_response_gate` called for predict
|
||||
- `repeat_last_prediction` activity called
|
||||
- Export workflow NOT called with current prediction
|
||||
- Workflow completes
|
||||
|
||||
---
|
||||
|
||||
### 2.4 Error Scenarios
|
||||
|
||||
#### Scenario 2.4.1: MLFlow Transform API Error
|
||||
**Description**: MLFlow transform request fails
|
||||
|
||||
**Input**:
|
||||
- Valid input data
|
||||
- MLFlow service unavailable or returns error
|
||||
|
||||
**Expected Behavior**:
|
||||
- `request_transform` raises exception
|
||||
- Notification sent with MLFlow error details
|
||||
- Workflow fails after retry attempts
|
||||
|
||||
**Assertions**:
|
||||
- Exception raised from transform activity
|
||||
- Error notification sent
|
||||
- Workflow fails
|
||||
- Export NOT called
|
||||
|
||||
---
|
||||
|
||||
#### Scenario 2.4.2: MLFlow Predict API Error
|
||||
**Description**: MLFlow predict request fails
|
||||
|
||||
**Input**:
|
||||
- Valid input and transform data
|
||||
- MLFlow predict service unavailable
|
||||
|
||||
**Expected Behavior**:
|
||||
- `request_predict` raises exception
|
||||
- Notification sent
|
||||
- Workflow fails after retries
|
||||
|
||||
**Assertions**:
|
||||
- Transform succeeded
|
||||
- Predict raised exception
|
||||
- Error notification sent
|
||||
- Workflow fails
|
||||
|
||||
---
|
||||
|
||||
## 3. Format and Export Prediction - Child Workflow Scenarios
|
||||
|
||||
### 3.1 Success Scenarios
|
||||
|
||||
#### Scenario 3.1.1: Default Prediction Export
|
||||
**Description**: Error prediction path creates default prediction
|
||||
|
||||
**Input**:
|
||||
- `path_flag: 'STOP'` or other non-None value
|
||||
- `comment` provided with error details
|
||||
|
||||
**Expected Behavior**:
|
||||
- `format_default_prediction` called instead of `format_prediction`
|
||||
- Default prediction created with error metadata
|
||||
- Exported to PostgreSQL only
|
||||
- Transformed data NOT processed
|
||||
- Metrics written
|
||||
|
||||
**Assertions**:
|
||||
- `format_default_prediction` called
|
||||
- `format_prediction` NOT called
|
||||
- `format_transformed_data` NOT called
|
||||
- One PostgreSQL export only
|
||||
- Default values in prediction data
|
||||
- Comment included
|
||||
|
||||
---
|
||||
|
||||
#### Scenario 3.1.2: Export Without Optional Outputs
|
||||
**Description**: Export only to PostgreSQL (no OPC or PI Web API)
|
||||
|
||||
**Input**:
|
||||
- `path_flag: None`
|
||||
- `opc_output_config: None` or `{}`
|
||||
- `pi_web_api_output_config: None` or `{}`
|
||||
|
||||
**Expected Behavior**:
|
||||
- Normal formatting
|
||||
- Only PostgreSQL export executed
|
||||
- OPC and PI Web API activities skipped
|
||||
- Metrics written without OPC metrics
|
||||
|
||||
**Assertions**:
|
||||
- PI Web API activity NOT called
|
||||
- OPC activity NOT called
|
||||
- PostgreSQL export called
|
||||
- Metrics written with empty `opc_metrics`
|
||||
|
||||
---
|
||||
|
||||
#### Scenario 3.1.3: Export Without Transformed Data
|
||||
**Description**: Only prediction exported, no transform table
|
||||
|
||||
**Input**:
|
||||
- `path_flag: None`
|
||||
- `transformed_data: None`
|
||||
|
||||
**Expected Behavior**:
|
||||
- Only prediction formatted and exported
|
||||
- Transform export skipped
|
||||
- Single PostgreSQL write
|
||||
|
||||
**Assertions**:
|
||||
- `format_transformed_data` NOT called
|
||||
- One PostgreSQL export
|
||||
- Transform table remains empty
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Error Scenarios
|
||||
|
||||
#### Scenario 3.2.1: PostgreSQL Export Error - Predictions Table
|
||||
**Description**: Failed to write predictions to database
|
||||
|
||||
**Input**:
|
||||
- Valid formatted prediction
|
||||
- PostgreSQL connection fails or table doesn't exist
|
||||
|
||||
**Expected Behavior**:
|
||||
- `export_data_to_postgres` raises exception
|
||||
- Notification sent with database error
|
||||
- Workflow fails after retries
|
||||
|
||||
**Assertions**:
|
||||
- Exception raised from export activity
|
||||
- Error notification sent
|
||||
- Workflow fails
|
||||
- Metrics NOT written (activity doesn't execute)
|
||||
|
||||
---
|
||||
|
||||
#### Scenario 3.2.2: PI Web API Write Error
|
||||
**Description**: PI Web API export fails
|
||||
|
||||
**Input**:
|
||||
- Valid prediction
|
||||
- PI Web API service unavailable or invalid config
|
||||
|
||||
**Expected Behavior**:
|
||||
- `write_pi_web_api_data` raises exception
|
||||
- Notification sent
|
||||
- Workflow fails after retries
|
||||
- PostgreSQL export may not execute (depends on execution order)
|
||||
|
||||
**Assertions**:
|
||||
- PI Web API error notification sent
|
||||
- Workflow fails
|
||||
- May impact subsequent exports
|
||||
|
||||
---
|
||||
|
||||
#### Scenario 3.2.3: OPC Write Error
|
||||
**Description**: OPC server write fails
|
||||
|
||||
**Input**:
|
||||
- Valid prediction
|
||||
- OPC server unavailable or invalid configuration
|
||||
|
||||
**Expected Behavior**:
|
||||
- `write_opc_data` raises exception
|
||||
- Notification sent
|
||||
- Workflow fails after retries
|
||||
|
||||
**Assertions**:
|
||||
- OPC error notification sent
|
||||
- Workflow fails
|
||||
- PostgreSQL export may not execute
|
||||
|
||||
---
|
||||
|
||||
## 4. End-to-End Integration Scenarios
|
||||
|
||||
### 4.1 Complete Success Path
|
||||
|
||||
#### Scenario 4.1.1: Full Pipeline Success with All Features
|
||||
**Description**: Complete workflow execution with all optional features enabled
|
||||
|
||||
**Input**:
|
||||
- Valid SQL query returning data
|
||||
- All configurations provided (OPC, PI Web API, filters, policies)
|
||||
- MLFlow services available
|
||||
- All databases available
|
||||
|
||||
**Expected Behavior**:
|
||||
- SQL query loads data
|
||||
- Input gate passes
|
||||
- MLFlow transform succeeds
|
||||
- MLFlow predict succeeds
|
||||
- All validations pass
|
||||
- Prediction formatted
|
||||
- Transformed data formatted
|
||||
- Both exported to PostgreSQL
|
||||
- PI Web API write succeeds
|
||||
- OPC write succeeds
|
||||
- Metrics written
|
||||
|
||||
**Assertions**:
|
||||
- All activities executed in correct order
|
||||
- All three workflows execute (batch, process, export)
|
||||
- All exports succeed
|
||||
- All tables have data
|
||||
- All external systems updated
|
||||
- Metrics recorded
|
||||
|
||||
---
|
||||
|
||||
### 4.2 Error Recovery Integration
|
||||
|
||||
#### Scenario 4.2.1: Transform Error with Repeat Fallback
|
||||
**Description**: Transform fails, workflow repeats last prediction
|
||||
|
||||
**Input**:
|
||||
- Valid input
|
||||
- MLFlow transform fails
|
||||
- REPEAT policy configured
|
||||
- Previous prediction exists
|
||||
|
||||
**Expected Behavior**:
|
||||
- Transform fails
|
||||
- Filter detects error
|
||||
- Path handler triggers REPEAT
|
||||
- Last prediction retrieved and re-exported
|
||||
- Workflow completes successfully
|
||||
|
||||
**Assertions**:
|
||||
- Transform attempted
|
||||
- Error handled gracefully
|
||||
- Last prediction copied
|
||||
- Workflow completes without exception
|
||||
|
||||
---
|
||||
Reference in New Issue
Block a user