gitea_admin/sientia-dataops-laborious_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
bda479af5f912f51967265b2e51d7471b04df059
sientia-dataops-laborious_t…/e2e/scenarios.md
vitor-aignosi 241f283724 SIENTIAPDE-1478 
Enhance end-to-end tests for PredictionsBatch workflow scenarios

- Introduced mock repositories for PI Web API and OPC operations to improve test coverage.
- Updated test scenarios to handle partial write errors for PI Web API and OPC.
- Refactored existing tests to assert correct behavior under various error conditions.
- Enhanced logging and error handling in API and OPC activities to provide clearer feedback on failures.
- Removed outdated integration test file to streamline test suite.
2026-01-16 16:41:36 -03:00

14 KiB
Raw Blame History

Test Scenarios for Predictions Batch Workflow

This document describes all possible test scenarios for the predictions_batch workflow and its child workflows prediction_process and format_and_export_prediction.

Workflow Overview

The predictions_batch workflow:

  1. Loads data using a custom SQL query
  2. Prepares prediction configuration
  3. Delegates to prediction_process child workflow which:
    • Retrieves last timestamp for incremental processing
    • Applies input data quality gates
    • Executes MLFlow transform operation
    • Validates transform response
    • Executes MLFlow predict operation
    • Validates predict response
    • Delegates to format_and_export_prediction child workflow
  4. The format_and_export_prediction workflow:
    • Formats prediction data (normal or default)
    • Exports to PI Web API (optional)
    • Exports to OPC server (optional)
    • Exports to PostgreSQL
    • Writes metrics

1. Predictions Batch - Main Workflow Scenarios

1.1 Success Scenarios

Scenario 1.1.1: Happy Path - Complete Success

Description: Workflow completes successfully with valid SQL query and all activities succeed

Input:

  • Valid schedule_name, model_name, model_id
  • Valid query returning non-empty DataFrame
  • Valid schema, table_name, transform_table_name
  • Optional datetime_columns for timestamp parsing
  • Optional input_filters, mlflow_transform_filters, mlflow_predict_filters
  • Optional path_priority, opc_output_config, pi_web_api_output_config

Expected Behavior:

  • load_custom_query returns DataFrame with data
  • Workflow prepares prediction input with all configurations
  • prediction_process child workflow executes successfully
  • All gates pass with no issues
  • Transform and predict operations succeed
  • Data exported to PostgreSQL
  • Metrics written

Assertions:

  • SQL query executed once
  • prediction_process workflow called with correct parameters
  • Data exists in PostgreSQL (predictions table)
  • Metrics recorded
  • No errors raised

1.2 Error Scenarios

Scenario 1.2.1: SQL Query Execution Error

Description: SQL query fails due to syntax error or connection issue

Input:

  • Invalid SQL query (syntax error)
  • Or database connection unavailable

Expected Behavior:

  • load_custom_query raises exception (caught by Temporal retry policy)
  • Notification sent with SQL error details
  • After retries, activity may return empty data or workflow may fail
  • If empty data returned, workflow completes with early exit via input gate

Assertions:

  • Error notification sent
  • Workflow completes (either fails or exits early)
  • No data in predictions table

Scenario 1.2.2: Missing Required Parameters

Description: Essential parameters missing from input

Input:

  • Missing query or model_id or schema or table_name

Expected Behavior:

  • Workflow or activity raises KeyError or validation error
  • Workflow fails immediately

Assertions:

  • Workflow fails with parameter error
  • Error notification sent
  • No child workflow called

Scenario 1.2.3: Invalid Datetime Column Specification

Description: Datetime column specified doesn't exist in query results

Input:

  • datetime_columns: ['nonexistent_column']
  • Query results don't have this column

Expected Behavior:

  • load_custom_query may raise KeyError or warning
  • Depending on implementation, workflow may fail or continue
  • Error notification sent

Assertions:

  • Error raised or warning logged
  • Workflow behavior depends on error handling policy

2. Prediction Process - Child Workflow Scenarios

2.1 Input gate Early Exit Scenarios

Scenario 2.1.1: Input Gate Triggers CONTINUE

Description: Input gate determines data should use previous prediction

Input:

  • Data that should continue with input data as prediction
  • input_filters configured with POLICY: 'CONTINUE'
  • path_priority includes CONTINUE

Expected Behavior:

  • input_gate returns path_flag='CONTINUE'
  • path_flag_handler calls export workflow with input data directly
  • MLFlow transform and predict skipped
  • Data exported as-is

Assertions:

  • input_gate called
  • MLFlow operations NOT called
  • Export workflow called with original data
  • Workflow completes

Scenario 2.1.2: Input Gate Triggers STOP

Description: Input data quality gate fails with STOP policy

Input:

  • Data with EMPTY_DATA or other critical issues
  • input_filters configured with POLICY: 'STOP'

Expected Behavior:

  • input_gate returns path_flag='STOP'
  • path_flag_handler detects STOP
  • Workflow returns early without calling MLFlow
  • No prediction exported

Assertions:

  • input_gate called
  • path_flag_handler returns True (early exit)
  • MLFlow transform NOT called
  • Export workflow NOT called
  • Workflow completes without error

Scenario 2.1.3: Input Gate Triggers REPEAT

Description: Input gate determines data should repeat last prediction

Input:

  • Data with quality issues that require using previous prediction
  • input_filters configured with POLICY: 'REPEAT'
  • path_priority includes REPEAT

Expected Behavior:

  • input_gate returns path_flag='REPEAT'
  • path_flag_handler calls repeat_last_prediction activity
  • MLFlow transform and predict skipped
  • Last prediction repeated and exported

Assertions:

  • input_gate called
  • MLFlow operations NOT called
  • repeat_last_prediction activity called
  • Workflow completes

2.2 Transform gate Early Exit Scenarios

Scenario 2.2.1: Transform Gate Triggers CONTINUE

Description: Transform response gate determines data should continue despite issues

Input:

  • Valid input data
  • Transform response has quality issues but policy is CONTINUE
  • mlflow_transform_filters configured with POLICY: 'CONTINUE'
  • path_priority includes CONTINUE

Expected Behavior:

  • request_transform succeeds
  • mlflow_response_gate for transform returns path_flag='CONTINUE'
  • path_flag_handler calls export workflow with transform data
  • MLFlow predict skipped
  • Transform data exported as-is

Assertions:

  • Transform completed
  • mlflow_response_gate called for transform
  • MLFlow predict NOT called
  • Export workflow called with transform data
  • Workflow completes

Scenario 2.2.2: Transform Gate Triggers STOP

Description: Transform response validation fails with STOP policy

Input:

  • Valid input data
  • Transform response has critical errors
  • mlflow_transform_filters configured with POLICY: 'STOP'

Expected Behavior:

  • request_transform succeeds but response invalid
  • mlflow_response_gate for transform returns path_flag='STOP'
  • Workflow exits without calling predict or export

Assertions:

  • Transform completed but validation failed
  • mlflow_response_gate called for transform
  • MLFlow predict NOT called
  • Export workflow NOT called
  • Workflow completes without error

Scenario 2.2.3: Transform Gate Triggers REPEAT

Description: Transform response gate determines data should repeat last prediction

Input:

  • Valid input data
  • Transform response has quality issues that require using previous prediction
  • mlflow_transform_filters configured with POLICY: 'REPEAT'
  • path_priority includes REPEAT

Expected Behavior:

  • request_transform succeeds but response has issues
  • mlflow_response_gate for transform returns path_flag='REPEAT'
  • path_flag_handler calls repeat_last_prediction activity
  • MLFlow predict skipped
  • Last prediction repeated and exported

Assertions:

  • Transform completed but validation triggered REPEAT
  • mlflow_response_gate called for transform
  • MLFlow predict NOT called
  • repeat_last_prediction activity called
  • Workflow completes

2.3 Predict gate Early Exit Scenarios

Scenario 2.3.1: Predict Gate Triggers CONTINUE

Description: Predict response gate determines data should continue despite issues

Input:

  • Valid input and transform data
  • Predict response has quality issues but policy is CONTINUE
  • mlflow_predict_filters configured with POLICY: 'CONTINUE'
  • path_priority includes CONTINUE

Expected Behavior:

  • request_predict succeeds
  • mlflow_response_gate for predict returns path_flag='CONTINUE'
  • path_flag_handler calls export workflow with predict data
  • Prediction exported despite quality issues

Assertions:

  • Transform and predict completed
  • mlflow_response_gate called for predict
  • Export workflow called with predict data
  • Workflow completes

Scenario 2.3.2: Predict Gate Triggers STOP

Description: Prediction validation fails with STOP policy

Input:

  • Valid input and transform
  • Predict response has critical errors
  • mlflow_predict_filters configured with POLICY: 'STOP'

Expected Behavior:

  • request_predict succeeds but response invalid
  • mlflow_response_gate for predict returns path_flag='STOP'
  • Workflow exits without export

Assertions:

  • Transform completed
  • Predict completed but validation failed
  • Export workflow NOT called
  • Workflow completes without error

Scenario 2.3.3: Predict Gate Triggers REPEAT

Description: Predict response gate determines data should repeat last prediction

Input:

  • Valid input and transform data
  • Predict response has quality issues that require using previous prediction
  • mlflow_predict_filters configured with POLICY: 'REPEAT'
  • path_priority includes REPEAT

Expected Behavior:

  • request_predict succeeds but response has issues
  • mlflow_response_gate for predict returns path_flag='REPEAT'
  • path_flag_handler calls repeat_last_prediction activity
  • Last prediction repeated and exported

Assertions:

  • Transform and predict completed but validation triggered REPEAT
  • mlflow_response_gate called for predict
  • repeat_last_prediction activity called
  • Export workflow NOT called with current prediction
  • Workflow completes

3. Format and Export Prediction - Child Workflow Scenarios

3.1 Success Scenarios

Scenario 3.1.1: Default Prediction Export

Description: Error prediction path creates default prediction

Input:

  • path_flag: 'ERROR' or other non-None value (not STOP/CONTINUE/REPEAT)
  • comment provided with error details

Expected Behavior:

  • format_default_prediction called instead of format_prediction
  • Default prediction created with error metadata
  • Exported to PostgreSQL only
  • Transformed data NOT processed
  • Metrics written

Assertions:

  • format_default_prediction called
  • format_prediction NOT called
  • format_transformed_data NOT called
  • One PostgreSQL export only
  • Default values in prediction data
  • Comment included

Scenario 3.1.2: Export with OPC only

Description: Export to PostgreSQL and OPC server only (no PI Web API)

Input:

  • path_flag: None
  • opc_output_config configured with valid OPC settings
  • pi_web_api_output_config: None or {}

Expected Behavior:

  • Normal formatting
  • PostgreSQL export executed
  • OPC export executed
  • PI Web API activity skipped
  • Metrics written with OPC metrics

Assertions:

  • PI Web API activity NOT called
  • OPC activity called
  • PostgreSQL export called
  • Metrics written with opc_metrics populated

Scenario 3.1.3: Export with PI Web API only

Description: Export to PostgreSQL and PI Web API only (no OPC)

Input:

  • path_flag: None
  • pi_web_api_output_config configured with valid PI Web API settings
  • opc_output_config: None or {}

Expected Behavior:

  • Normal formatting
  • PostgreSQL export executed
  • PI Web API export executed
  • OPC activity skipped
  • Metrics written without OPC metrics

Assertions:

  • OPC activity NOT called
  • PI Web API activity called
  • PostgreSQL export called
  • Metrics written with empty opc_metrics

Scenario 3.1.4: Export Without Optional Outputs

Description: Export only to PostgreSQL (no OPC or PI Web API)

Input:

  • path_flag: None
  • opc_output_config: None or {}
  • pi_web_api_output_config: None or {}

Expected Behavior:

  • Normal formatting
  • Only PostgreSQL export executed
  • OPC and PI Web API activities skipped
  • Metrics written without OPC metrics

Assertions:

  • PI Web API activity NOT called
  • OPC activity NOT called
  • PostgreSQL export called
  • Metrics written with empty opc_metrics

Scenario 3.1.5: Export Without Transformed Data

Description: Only prediction exported, no transform table

Input:

  • path_flag: None
  • transformed_data: None or save_transform: False
  • opc_output_config: None or {}
  • pi_web_api_output_config: None or {}

Expected Behavior:

  • Only prediction formatted and exported
  • Transform export skipped
  • Single PostgreSQL write

Assertions:

  • format_transformed_data NOT called
  • One PostgreSQL export
  • Transform table remains empty

3.2 Error Scenarios

Scenario 3.2.1: PI Web API Write Error

Description: PI Web API export fails

Input:

  • Valid prediction
  • PI Web API service unavailable or invalid config

Expected Behavior:

  • write_pi_web_api_data raises exception
  • Notification sent
  • Workflow fails after retries
  • PostgreSQL export may not execute (depends on execution order)

Assertions:

  • PI Web API error notification sent
  • Workflow fails
  • May impact subsequent exports

Scenario 3.2.2: OPC Write Error

Description: OPC server write fails

Input:

  • Valid prediction
  • OPC server unavailable or invalid configuration

Expected Behavior:

  • write_opc_data raises exception
  • Notification sent
  • Workflow fails after retries

Assertions:

  • OPC error notification sent
  • Workflow fails
  • PostgreSQL export may not execute

Scenario 3.2.3: PI Web API Partial Write Error

Description: Two prediction tags attempt to be written to PI Web API, but only one succeeds

Input:

  • Valid prediction
  • Two prediction tags configured
  • PI Web API returns partial success (one tag succeeds, one fails)

Expected Behavior:

  • write_pi_web_api_data processes response
  • process_pi_web_api_response detects partial failure
  • Error confidence set (13)
  • Notification sent for failed tag
  • Workflow completes with error confidence

Assertions:

  • One tag written successfully
  • One tag failed
  • Error confidence set in prediction
  • Error notification sent
  • Workflow completes
Reference in New Issue View Git Blame Copy Permalink