gitea_admin/sientia-dataops-laborious-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6b9bb389683f9f3284ae158ef98aa824e95e53e9
sientia-dataops-laborious-l…/e2e/scenarios.md
vitor-aignosi 6b9bb38968 SIENTIAPDE-1478 
Enhance end-to-end tests for PredictionsBatch workflow scenarios

- Updated scenario descriptions and assertions for error handling in prediction exports.
- Introduced new test cases for handling exports with only OPC or PI Web API.
- Refactored existing tests to improve clarity and maintainability, including dynamic data insertion.
- Adjusted workflow input configurations to better reflect expected behaviors for various error scenarios.
2026-01-16 10:38:08 -03:00

15 KiB
Raw Blame History

Test Scenarios for Predictions Batch Workflow

This document describes all possible test scenarios for the predictions_batch workflow and its child workflows prediction_process and format_and_export_prediction.

Workflow Overview

The predictions_batch workflow:

  1. Loads data using a custom SQL query
  2. Prepares prediction configuration
  3. Delegates to prediction_process child workflow which:
    • Retrieves last timestamp for incremental processing
    • Applies input data quality gates
    • Executes MLFlow transform operation
    • Validates transform response
    • Executes MLFlow predict operation
    • Validates predict response
    • Delegates to format_and_export_prediction child workflow
  4. The format_and_export_prediction workflow:
    • Formats prediction data (normal or default)
    • Exports to PI Web API (optional)
    • Exports to OPC server (optional)
    • Exports to PostgreSQL
    • Writes metrics

1. Predictions Batch - Main Workflow Scenarios

1.1 Success Scenarios

Scenario 1.1.1: Happy Path - Complete Success

Description: Workflow completes successfully with valid SQL query and all activities succeed

Input:

  • Valid schedule_name, model_name, model_id
  • Valid query returning non-empty DataFrame
  • Valid schema, table_name, transform_table_name
  • Optional datetime_columns for timestamp parsing
  • Optional input_filters, mlflow_transform_filters, mlflow_predict_filters
  • Optional path_priority, opc_output_config, pi_web_api_output_config

Expected Behavior:

  • load_custom_query returns DataFrame with data
  • Workflow prepares prediction input with all configurations
  • prediction_process child workflow executes successfully
  • All gates pass with no issues
  • Transform and predict operations succeed
  • Data exported to PostgreSQL
  • Metrics written

Assertions:

  • SQL query executed once
  • prediction_process workflow called with correct parameters
  • Data exists in PostgreSQL (predictions table)
  • Metrics recorded
  • No errors raised

1.2 Error Scenarios

Scenario 1.2.1: SQL Query Execution Error

Description: SQL query fails due to syntax error or connection issue

Input:

  • Invalid SQL query (syntax error)
  • Or database connection unavailable

Expected Behavior:

  • load_custom_query raises exception (caught by Temporal retry policy)
  • Notification sent with SQL error details
  • After retries, activity may return empty data or workflow may fail
  • If empty data returned, workflow completes with early exit via input gate

Assertions:

  • Error notification sent
  • Workflow completes (either fails or exits early)
  • No data in predictions table

Scenario 1.2.2: Missing Required Parameters

Description: Essential parameters missing from input

Input:

  • Missing query or model_id or schema or table_name

Expected Behavior:

  • Workflow or activity raises KeyError or validation error
  • Workflow fails immediately

Assertions:

  • Workflow fails with parameter error
  • Error notification sent
  • No child workflow called

Scenario 1.2.3: Invalid Datetime Column Specification

Description: Datetime column specified doesn't exist in query results

Input:

  • datetime_columns: ['nonexistent_column']
  • Query results don't have this column

Expected Behavior:

  • load_custom_query may raise KeyError or warning
  • Depending on implementation, workflow may fail or continue
  • Error notification sent

Assertions:

  • Error raised or warning logged
  • Workflow behavior depends on error handling policy

2. Prediction Process - Child Workflow Scenarios

2.1 Input gate Early Exit Scenarios

Scenario 2.1.1: Input Gate Triggers CONTINUE

Description: Input gate determines data should use previous prediction

Input:

  • Data that should continue with input data as prediction
  • input_filters configured with POLICY: 'CONTINUE'
  • path_priority includes CONTINUE

Expected Behavior:

  • input_gate returns path_flag='CONTINUE'
  • path_flag_handler calls export workflow with input data directly
  • MLFlow transform and predict skipped
  • Data exported as-is

Assertions:

  • input_gate called
  • MLFlow operations NOT called
  • Export workflow called with original data
  • Workflow completes

Scenario 2.1.2: Input Gate Triggers STOP

Description: Input data quality gate fails with STOP policy

Input:

  • Data with EMPTY_DATA or other critical issues
  • input_filters configured with POLICY: 'STOP'

Expected Behavior:

  • input_gate returns path_flag='STOP'
  • path_flag_handler detects STOP
  • Workflow returns early without calling MLFlow
  • No prediction exported

Assertions:

  • input_gate called
  • path_flag_handler returns True (early exit)
  • MLFlow transform NOT called
  • Export workflow NOT called
  • Workflow completes without error

Scenario 2.1.3: Input Gate Triggers REPEAT

Description: Input gate determines data should repeat last prediction

Input:

  • Data with quality issues that require using previous prediction
  • input_filters configured with POLICY: 'REPEAT'
  • path_priority includes REPEAT

Expected Behavior:

  • input_gate returns path_flag='REPEAT'
  • path_flag_handler calls repeat_last_prediction activity
  • MLFlow transform and predict skipped
  • Last prediction repeated and exported

Assertions:

  • input_gate called
  • MLFlow operations NOT called
  • repeat_last_prediction activity called
  • Workflow completes

2.2 Transform gate Early Exit Scenarios

Scenario 2.2.1: Transform Gate Triggers CONTINUE

Description: Transform response gate determines data should continue despite issues

Input:

  • Valid input data
  • Transform response has quality issues but policy is CONTINUE
  • mlflow_transform_filters configured with POLICY: 'CONTINUE'
  • path_priority includes CONTINUE

Expected Behavior:

  • request_transform succeeds
  • mlflow_response_gate for transform returns path_flag='CONTINUE'
  • path_flag_handler calls export workflow with transform data
  • MLFlow predict skipped
  • Transform data exported as-is

Assertions:

  • Transform completed
  • mlflow_response_gate called for transform
  • MLFlow predict NOT called
  • Export workflow called with transform data
  • Workflow completes

Scenario 2.2.2: Transform Gate Triggers STOP

Description: Transform response validation fails with STOP policy

Input:

  • Valid input data
  • Transform response has critical errors
  • mlflow_transform_filters configured with POLICY: 'STOP'

Expected Behavior:

  • request_transform succeeds but response invalid
  • mlflow_response_gate for transform returns path_flag='STOP'
  • Workflow exits without calling predict or export

Assertions:

  • Transform completed but validation failed
  • mlflow_response_gate called for transform
  • MLFlow predict NOT called
  • Export workflow NOT called
  • Workflow completes without error

Scenario 2.2.3: Transform Gate Triggers REPEAT

Description: Transform response gate determines data should repeat last prediction

Input:

  • Valid input data
  • Transform response has quality issues that require using previous prediction
  • mlflow_transform_filters configured with POLICY: 'REPEAT'
  • path_priority includes REPEAT

Expected Behavior:

  • request_transform succeeds but response has issues
  • mlflow_response_gate for transform returns path_flag='REPEAT'
  • path_flag_handler calls repeat_last_prediction activity
  • MLFlow predict skipped
  • Last prediction repeated and exported

Assertions:

  • Transform completed but validation triggered REPEAT
  • mlflow_response_gate called for transform
  • MLFlow predict NOT called
  • repeat_last_prediction activity called
  • Workflow completes

2.3 Predict gate Early Exit Scenarios

Scenario 2.3.1: Predict Gate Triggers CONTINUE

Description: Predict response gate determines data should continue despite issues

Input:

  • Valid input and transform data
  • Predict response has quality issues but policy is CONTINUE
  • mlflow_predict_filters configured with POLICY: 'CONTINUE'
  • path_priority includes CONTINUE

Expected Behavior:

  • request_predict succeeds
  • mlflow_response_gate for predict returns path_flag='CONTINUE'
  • path_flag_handler calls export workflow with predict data
  • Prediction exported despite quality issues

Assertions:

  • Transform and predict completed
  • mlflow_response_gate called for predict
  • Export workflow called with predict data
  • Workflow completes

Scenario 2.3.2: Predict Gate Triggers STOP

Description: Prediction validation fails with STOP policy

Input:

  • Valid input and transform
  • Predict response has critical errors
  • mlflow_predict_filters configured with POLICY: 'STOP'

Expected Behavior:

  • request_predict succeeds but response invalid
  • mlflow_response_gate for predict returns path_flag='STOP'
  • Workflow exits without export

Assertions:

  • Transform completed
  • Predict completed but validation failed
  • Export workflow NOT called
  • Workflow completes without error

Scenario 2.3.3: Predict Gate Triggers REPEAT

Description: Predict response gate determines data should repeat last prediction

Input:

  • Valid input and transform data
  • Predict response has quality issues that require using previous prediction
  • mlflow_predict_filters configured with POLICY: 'REPEAT'
  • path_priority includes REPEAT

Expected Behavior:

  • request_predict succeeds but response has issues
  • mlflow_response_gate for predict returns path_flag='REPEAT'
  • path_flag_handler calls repeat_last_prediction activity
  • Last prediction repeated and exported

Assertions:

  • Transform and predict completed but validation triggered REPEAT
  • mlflow_response_gate called for predict
  • repeat_last_prediction activity called
  • Export workflow NOT called with current prediction
  • Workflow completes

3. Format and Export Prediction - Child Workflow Scenarios

3.1 Success Scenarios

Scenario 3.1.1: Default Prediction Export

Description: Error prediction path creates default prediction

Input:

  • path_flag: 'ERROR' or other non-None value (not STOP/CONTINUE/REPEAT)
  • comment provided with error details

Expected Behavior:

  • format_default_prediction called instead of format_prediction
  • Default prediction created with error metadata
  • Exported to PostgreSQL only
  • Transformed data NOT processed
  • Metrics written

Assertions:

  • format_default_prediction called
  • format_prediction NOT called
  • format_transformed_data NOT called
  • One PostgreSQL export only
  • Default values in prediction data
  • Comment included

Scenario 3.1.2: Export with OPC only

Description: Export to PostgreSQL and OPC server only (no PI Web API)

Input:

  • path_flag: None
  • opc_output_config configured with valid OPC settings
  • pi_web_api_output_config: None or {}

Expected Behavior:

  • Normal formatting
  • PostgreSQL export executed
  • OPC export executed
  • PI Web API activity skipped
  • Metrics written with OPC metrics

Assertions:

  • PI Web API activity NOT called
  • OPC activity called
  • PostgreSQL export called
  • Metrics written with opc_metrics populated

Scenario 3.1.3: Export with PI Web API only

Description: Export to PostgreSQL and PI Web API only (no OPC)

Input:

  • path_flag: None
  • pi_web_api_output_config configured with valid PI Web API settings
  • opc_output_config: None or {}

Expected Behavior:

  • Normal formatting
  • PostgreSQL export executed
  • PI Web API export executed
  • OPC activity skipped
  • Metrics written without OPC metrics

Assertions:

  • OPC activity NOT called
  • PI Web API activity called
  • PostgreSQL export called
  • Metrics written with empty opc_metrics

Scenario 3.1.4: Export Without Optional Outputs

Description: Export only to PostgreSQL (no OPC or PI Web API)

Input:

  • path_flag: None
  • opc_output_config: None or {}
  • pi_web_api_output_config: None or {}

Expected Behavior:

  • Normal formatting
  • Only PostgreSQL export executed
  • OPC and PI Web API activities skipped
  • Metrics written without OPC metrics

Assertions:

  • PI Web API activity NOT called
  • OPC activity NOT called
  • PostgreSQL export called
  • Metrics written with empty opc_metrics

Scenario 3.1.5: Export Without Transformed Data

Description: Only prediction exported, no transform table

Input:

  • path_flag: None
  • transformed_data: None or save_transform: False
  • opc_output_config: None or {}
  • pi_web_api_output_config: None or {}

Expected Behavior:

  • Only prediction formatted and exported
  • Transform export skipped
  • Single PostgreSQL write

Assertions:

  • format_transformed_data NOT called
  • One PostgreSQL export
  • Transform table remains empty

3.2 Error Scenarios

Scenario 3.2.1: PostgreSQL Export Error - Predictions Table

Description: Failed to write predictions to database

Input:

  • Valid formatted prediction
  • PostgreSQL connection fails or table doesn't exist

Expected Behavior:

  • export_data_to_postgres raises exception
  • Notification sent with database error
  • Workflow fails after retries

Assertions:

  • Exception raised from export activity
  • Error notification sent
  • Workflow fails
  • Metrics NOT written (activity doesn't execute)

Scenario 3.2.2: PI Web API Write Error

Description: PI Web API export fails

Input:

  • Valid prediction
  • PI Web API service unavailable or invalid config

Expected Behavior:

  • write_pi_web_api_data raises exception
  • Notification sent
  • Workflow fails after retries
  • PostgreSQL export may not execute (depends on execution order)

Assertions:

  • PI Web API error notification sent
  • Workflow fails
  • May impact subsequent exports

Scenario 3.2.3: OPC Write Error

Description: OPC server write fails

Input:

  • Valid prediction
  • OPC server unavailable or invalid configuration

Expected Behavior:

  • write_opc_data raises exception
  • Notification sent
  • Workflow fails after retries

Assertions:

  • OPC error notification sent
  • Workflow fails
  • PostgreSQL export may not execute

4. End-to-End Integration Scenarios

4.1 Complete Success Path

Scenario 4.1.1: Full Pipeline Success with All Features

Description: Complete workflow execution with all optional features enabled

Input:

  • Valid SQL query returning data
  • All configurations provided (OPC, PI Web API, filters, policies)
  • MLFlow services available
  • All databases available

Expected Behavior:

  • SQL query loads data
  • Input gate passes
  • MLFlow transform succeeds
  • MLFlow predict succeeds
  • All validations pass
  • Prediction formatted
  • Transformed data formatted
  • Both exported to PostgreSQL
  • PI Web API write succeeds
  • OPC write succeeds
  • Metrics written

Assertions:

  • All activities executed in correct order
  • All three workflows execute (batch, process, export)
  • All exports succeed
  • All tables have data
  • All external systems updated
  • Metrics recorded

4.2 Error Recovery Integration

Scenario 4.2.1: Transform Error with Repeat Fallback

Description: Transform fails, workflow repeats last prediction

Input:

  • Valid input
  • MLFlow transform fails
  • REPEAT policy configured
  • Previous prediction exists

Expected Behavior:

  • Transform fails
  • Filter detects error
  • Path handler triggers REPEAT
  • Last prediction retrieved and re-exported
  • Workflow completes successfully

Assertions:

  • Transform attempted
  • Error handled gracefully
  • Last prediction copied
  • Workflow completes without exception
Reference in New Issue View Git Blame Copy Permalink