gitea_admin/sientia-dataops-laborious_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7c1dae8ef66719c4a264386cedd0b3afc5b3a5e1
sientia-dataops-laborious_t…/e2e/scenarios.md
vitor-aignosi e3636d4b88 SIENTIAPDE-1811 
Enhance OPC UA testing framework and documentation

- Added a new marker in pyproject.toml for tests using the in-process OPC UA server.
- Updated opc-communication.md to clarify E2E test scenarios involving the real OPC server and mock server.
- Introduced an in-process asyncua OPC UA server fixture in conftest.py for E2E tests.
- Created a new fixture for activities using the real OpcRepository connected to the in-process server.
- Updated scenarios.md to include instructions for running OPC real-server tests.
2026-05-15 15:46:39 -03:00

18 KiB
Raw Blame History

Test Scenarios for Predictions Batch Workflow

This document describes all possible test scenarios for the predictions_batch workflow and its child workflows prediction_process and format_and_export_prediction.

Running automated E2E tests (e2e/)

  • Runtime: Docker (or a Docker-compatible daemon) must be available so testcontainers can start PostgreSQL and MinIO containers.
  • Dependencies: install dev requirements (includes testcontainers[postgres,minio]).
  • Invocation: run only integration-marked tests, for example: pytest e2e/ -m integration.
  • MinIO tests: e2e/test_minio_offload.py exercises real S3 uploads; other E2E modules continue to mock MinIO on the worker used by most scenarios.
  • OPC tests (real server): e2e/test_opc_real_server.py uses an in-process asyncua server and real OpcRepository (test_activities_real_opc). Scenarios 3.1.2, 3.2.2, 3.2.4, and 3.2.5 are covered there. Other E2E modules keep the OPC mock.
  • Run only OPC real-server tests: pytest e2e/test_opc_real_server.py -m "integration and opc".

Workflow Overview

The predictions_batch workflow:

  1. Loads data using a custom SQL query
  2. Prepares prediction configuration
  3. Delegates to prediction_process child workflow which:
    • Retrieves last timestamp for incremental processing
    • Applies input data quality gates
    • Executes MLFlow transform operation
    • Validates transform response
    • Executes MLFlow predict operation
    • Validates predict response
    • Delegates to format_and_export_prediction child workflow
  4. The format_and_export_prediction workflow:
    • Formats prediction data (normal or default)
    • Exports to PI Web API (optional)
    • Exports to OPC server (optional)
    • Exports to PostgreSQL
    • Writes metrics

1. Predictions Batch - Main Workflow Scenarios

1.1 Success Scenarios

Scenario 1.1.1: Happy Path - Complete Success

Description: Workflow completes successfully with valid SQL query and all activities succeed

Input:

  • Valid schedule_name, model_name, model_id
  • Valid query returning non-empty DataFrame
  • Valid schema, table_name, transform_table_name
  • Optional datetime_columns for timestamp parsing
  • Optional input_filters, mlflow_transform_filters, mlflow_predict_filters
  • Optional path_priority, opc_output_config, pi_web_api_output_config

Expected Behavior:

  • load_custom_query returns DataFrame with data
  • Workflow prepares prediction input with all configurations
  • prediction_process child workflow executes successfully
  • All gates pass with no issues
  • Transform and predict operations succeed
  • Data exported to PostgreSQL
  • Metrics written

Assertions:

  • SQL query executed once
  • prediction_process workflow called with correct parameters
  • Data exists in PostgreSQL (predictions table)
  • Metrics recorded
  • No errors raised

1.2 Error Scenarios

Scenario 1.2.1: SQL Query Execution Error

Description: SQL query fails due to syntax error or connection issue

Input:

  • Invalid SQL query (syntax error)
  • Or database connection unavailable

Expected Behavior:

  • load_custom_query raises exception (caught by Temporal retry policy)
  • Notification sent with SQL error details
  • After retries, activity may return empty data or workflow may fail
  • If empty data returned, workflow completes with early exit via input gate

Assertions:

  • Error notification sent
  • Workflow completes (either fails or exits early)
  • No data in predictions table

Scenario 1.2.2: Missing Required Parameters

Description: Essential parameters missing from input

Input:

  • Missing query or model_id or schema or table_name

Expected Behavior:

  • Workflow or activity raises KeyError or validation error
  • Workflow fails immediately

Assertions:

  • Workflow fails with parameter error
  • Error notification sent
  • No child workflow called

Scenario 1.2.3: Invalid Datetime Column Specification

Description: Datetime column specified doesn't exist in query results

Input:

  • datetime_columns: ['nonexistent_column']
  • Query results don't have this column

Expected Behavior:

  • load_custom_query may raise KeyError or warning
  • Depending on implementation, workflow may fail or continue
  • Error notification sent

Assertions:

  • Error raised or warning logged
  • Workflow behavior depends on error handling policy

2. Prediction Process - Child Workflow Scenarios

2.1 Input gate Early Exit Scenarios

Scenario 2.1.1: Input Gate Triggers CONTINUE

Description: Input gate determines data should use previous prediction

Input:

  • Data that should continue with input data as prediction
  • input_filters configured with POLICY: 'CONTINUE'
  • path_priority includes CONTINUE

Expected Behavior:

  • input_gate returns path_flag='CONTINUE'
  • path_flag_handler calls export workflow with input data directly
  • MLFlow transform and predict skipped
  • Data exported as-is

Assertions:

  • input_gate called
  • MLFlow operations NOT called
  • Export workflow called with original data
  • Workflow completes

Scenario 2.1.2: Input Gate Triggers STOP

Description: Input data quality gate fails with STOP policy

Input:

  • Data with EMPTY_DATA or other critical issues
  • input_filters configured with POLICY: 'STOP'

Expected Behavior:

  • input_gate returns path_flag='STOP'
  • path_flag_handler detects STOP
  • Workflow returns early without calling MLFlow
  • No prediction exported

Assertions:

  • input_gate called
  • path_flag_handler returns True (early exit)
  • MLFlow transform NOT called
  • Export workflow NOT called
  • Workflow completes without error

Scenario 2.1.3: Input Gate Triggers REPEAT

Description: Input gate determines data should repeat last prediction

Input:

  • Data with quality issues that require using previous prediction
  • input_filters configured with POLICY: 'REPEAT'
  • path_priority includes REPEAT

Expected Behavior:

  • input_gate returns path_flag='REPEAT'
  • path_flag_handler calls repeat_last_prediction activity
  • MLFlow transform and predict skipped
  • Last prediction repeated and exported

Assertions:

  • input_gate called
  • MLFlow operations NOT called
  • repeat_last_prediction activity called
  • Workflow completes

2.2 Transform gate Early Exit Scenarios

Scenario 2.2.1: Transform Gate Triggers CONTINUE

Description: Transform response gate determines data should continue despite issues

Input:

  • Valid input data
  • Transform response has quality issues but policy is CONTINUE
  • mlflow_transform_filters configured with POLICY: 'CONTINUE'
  • path_priority includes CONTINUE

Expected Behavior:

  • request_transform succeeds
  • mlflow_response_gate for transform returns path_flag='CONTINUE'
  • path_flag_handler calls export workflow with transform data
  • MLFlow predict skipped
  • Transform data exported as-is

Assertions:

  • Transform completed
  • mlflow_response_gate called for transform
  • MLFlow predict NOT called
  • Export workflow called with transform data
  • Workflow completes

Scenario 2.2.2: Transform Gate Triggers STOP

Description: Transform response validation fails with STOP policy

Input:

  • Valid input data
  • Transform response has critical errors
  • mlflow_transform_filters configured with POLICY: 'STOP'

Expected Behavior:

  • request_transform succeeds but response invalid
  • mlflow_response_gate for transform returns path_flag='STOP'
  • Workflow exits without calling predict or export

Assertions:

  • Transform completed but validation failed
  • mlflow_response_gate called for transform
  • MLFlow predict NOT called
  • Export workflow NOT called
  • Workflow completes without error

Scenario 2.2.3: Transform Gate Triggers REPEAT

Description: Transform response gate determines data should repeat last prediction

Input:

  • Valid input data
  • Transform response has quality issues that require using previous prediction
  • mlflow_transform_filters configured with POLICY: 'REPEAT'
  • path_priority includes REPEAT

Expected Behavior:

  • request_transform succeeds but response has issues
  • mlflow_response_gate for transform returns path_flag='REPEAT'
  • path_flag_handler calls repeat_last_prediction activity
  • MLFlow predict skipped
  • Last prediction repeated and exported

Assertions:

  • Transform completed but validation triggered REPEAT
  • mlflow_response_gate called for transform
  • MLFlow predict NOT called
  • repeat_last_prediction activity called
  • Workflow completes

2.3 Predict gate Early Exit Scenarios

Scenario 2.3.1: Predict Gate Triggers CONTINUE

Description: Predict response gate determines data should continue despite issues

Input:

  • Valid input and transform data
  • Predict response has quality issues but policy is CONTINUE
  • mlflow_predict_filters configured with POLICY: 'CONTINUE'
  • path_priority includes CONTINUE

Expected Behavior:

  • request_predict succeeds
  • mlflow_response_gate for predict returns path_flag='CONTINUE'
  • path_flag_handler calls export workflow with predict data
  • Prediction exported despite quality issues

Assertions:

  • Transform and predict completed
  • mlflow_response_gate called for predict
  • Export workflow called with predict data
  • Workflow completes

Scenario 2.3.2: Predict Gate Triggers STOP

Description: Prediction validation fails with STOP policy

Input:

  • Valid input and transform
  • Predict response has critical errors
  • mlflow_predict_filters configured with POLICY: 'STOP'

Expected Behavior:

  • request_predict succeeds but response invalid
  • mlflow_response_gate for predict returns path_flag='STOP'
  • Workflow exits without export

Assertions:

  • Transform completed
  • Predict completed but validation failed
  • Export workflow NOT called
  • Workflow completes without error

Scenario 2.3.3: Predict Gate Triggers REPEAT

Description: Predict response gate determines data should repeat last prediction

Input:

  • Valid input and transform data
  • Predict response has quality issues that require using previous prediction
  • mlflow_predict_filters configured with POLICY: 'REPEAT'
  • path_priority includes REPEAT

Expected Behavior:

  • request_predict succeeds but response has issues
  • mlflow_response_gate for predict returns path_flag='REPEAT'
  • path_flag_handler calls repeat_last_prediction activity
  • Last prediction repeated and exported

Assertions:

  • Transform and predict completed but validation triggered REPEAT
  • mlflow_response_gate called for predict
  • repeat_last_prediction activity called
  • Export workflow NOT called with current prediction
  • Workflow completes

3. Format and Export Prediction - Child Workflow Scenarios

3.1 Success Scenarios

Scenario 3.1.1: Default Prediction Export

Description: Error prediction path creates default prediction

Input:

  • path_flag: 'ERROR' or other non-None value (not STOP/CONTINUE/REPEAT)
  • comment provided with error details

Expected Behavior:

  • format_default_prediction called instead of format_prediction
  • Default prediction created with error metadata
  • Exported to PostgreSQL only
  • Transformed data NOT processed
  • Metrics written

Assertions:

  • format_default_prediction called
  • format_prediction NOT called
  • format_transformed_data NOT called
  • One PostgreSQL export only
  • Default values in prediction data
  • Comment included

Scenario 3.1.2: Export with OPC only

Description: Export to PostgreSQL and OPC server only (no PI Web API)

Input:

  • path_flag: None
  • opc_output_config configured with valid OPC settings
  • pi_web_api_output_config: None or {}

Expected Behavior:

  • Normal formatting
  • PostgreSQL export executed
  • OPC export executed
  • PI Web API activity skipped
  • Metrics written with OPC metrics

Assertions:

  • PI Web API activity NOT called
  • OPC activity called
  • PostgreSQL export called
  • Metrics written with opc_metrics populated

Scenario 3.1.3: Export with PI Web API only

Description: Export to PostgreSQL and PI Web API only (no OPC)

Input:

  • path_flag: None
  • pi_web_api_output_config configured with valid PI Web API settings
  • opc_output_config: None or {}

Expected Behavior:

  • Normal formatting
  • PostgreSQL export executed
  • PI Web API export executed
  • OPC activity skipped
  • Metrics written without OPC metrics

Assertions:

  • OPC activity NOT called
  • PI Web API activity called
  • PostgreSQL export called
  • Metrics written with empty opc_metrics

Scenario 3.1.4: Export Without Optional Outputs

Description: Export only to PostgreSQL (no OPC or PI Web API)

Input:

  • path_flag: None
  • opc_output_config: None or {}
  • pi_web_api_output_config: None or {}

Expected Behavior:

  • Normal formatting
  • Only PostgreSQL export executed
  • OPC and PI Web API activities skipped
  • Metrics written without OPC metrics

Assertions:

  • PI Web API activity NOT called
  • OPC activity NOT called
  • PostgreSQL export called
  • Metrics written with empty opc_metrics

Scenario 3.1.5: Export Without Transformed Data

Description: Only prediction exported, no transform table

Input:

  • path_flag: None
  • transformed_data: None or save_transform: False
  • opc_output_config: None or {}
  • pi_web_api_output_config: None or {}

Expected Behavior:

  • Only prediction formatted and exported
  • Transform export skipped
  • Single PostgreSQL write

Assertions:

  • format_transformed_data NOT called
  • One PostgreSQL export
  • Transform table remains empty

3.2 Error Scenarios

These paths do not rely on Temporal activity retries for export failures: the write activities run once, errors are handled inside the activity, and the workflow completes successfully with degraded metadata on the persisted prediction (prediction_confidence and comments).

Scenario 3.2.1: PI Web API Write Error

Description: PI Web API export fails

Input:

  • Valid prediction
  • PI Web API service unavailable or invalid config

Expected Behavior:

  • write_pi_web_api_data surfaces the failure (exception handled in the activity layer)
  • Notification may be sent
  • Workflow completes (does not fail)
  • Prediction row is still written to PostgreSQL with error confidence 13 and a comment describing the PI error
  • Subsequent steps (e.g. OPC, Postgres) still run per workflow order with the updated prediction payload

Assertions:

  • PI Web API error notification sent (when applicable)
  • Workflow completes
  • PostgreSQL contains the prediction with prediction_confidence 13 and expected comments

Scenario 3.2.2: OPC Write Error

Description: OPC server write fails

Input:

  • Valid prediction
  • OPC server unavailable or invalid configuration

Expected Behavior:

  • write_opc_data reports failure without aborting the workflow
  • Notification may be sent
  • Workflow completes (does not fail)
  • Prediction row is written to PostgreSQL with OPC error confidence 12 and a comment indicating OPC write issues

Assertions:

  • OPC error notification sent (when applicable)
  • Workflow completes
  • PostgreSQL contains the prediction with prediction_confidence 12 and expected comments

Scenario 3.2.4: OPC Session / Channel Bad* (Tier-1)

Description: OPC write fails with a Tier-1 session or channel status (e.g. BadSessionIdInvalid) while transport may still appear open on the client

Input:

  • Valid prediction and OPC output config
  • Mock or server returning Tier-1 UaStatusCodeError on write (no write retry in the same activity)

Expected Behavior:

  • write_opc_data fails forward for affected tags; background reconnect may be scheduled if OPC_RECONNECTION_INTERVAL allows
  • Workflow completes
  • PostgreSQL row uses prediction_confidence 14 and comment prefix OPC UA session/channel error: (including OPC status name)
  • opc_write_attempts_total records result=BadSessionIdInvalid (or matching status); no second write attempt in the same activity

Assertions:

  • Workflow completes
  • prediction_confidence = 14
  • comments matches OPC UA session/channel error:%
  • Generic OPC error confidence 12 is not used for this case

Reference: docs/opc-communication.md, plan .cursor/plans/opc_bad_reconnect_ac4c6045.plan.md

Scenario 3.2.5: OPC Write Blocked During Reconnect

Description: A write is attempted while the repository is reconnecting (session not ready)

Input:

  • Valid prediction
  • Simulated slow reconnect (e.g. delayed connect) or concurrent writes where the first triggers reconnect

Expected Behavior:

  • Second write (or parallel write) is rejected immediately when reconnect is in progress or _session_ready is cleared — without calling write_value
  • No wait/sleep on the write path; no duplicate connect from parallel writers (connection lock)
  • prediction_confidence = 14, comments = OPC UA reconnect in progress (distinguish from Tier-1 Bad* via comment prefix in SQL)

Assertions:

  • At most one reconnect sequence (disconnect + connect) for the overlapping window
  • No write retry after failure
  • Tests in test_opc_repository (unit) and optional e2e in test_predictions_batch_format_export.py

Scenario 3.2.3: PI Web API Partial Write Error

Description: Two prediction tags attempt to be written to PI Web API, but only one succeeds

Input:

  • Valid prediction
  • Two prediction tags configured
  • PI Web API returns partial success (one tag succeeds, one fails)

Expected Behavior:

  • write_pi_web_api_data processes response
  • process_pi_web_api_response detects partial failure
  • Error confidence set (13)
  • Notification sent for failed tag
  • Workflow completes with error confidence (single activity attempt; no retry loop)

Assertions:

  • One tag written successfully
  • One tag failed
  • Error confidence set in prediction
  • Error notification sent
  • Workflow completes
Reference in New Issue View Git Blame Copy Permalink