sientia-dataops-scouter_temporal/e2e/scenarios.md

Test Scenarios for PI Web API Scouter Workflow

This document describes all possible test scenarios for the pi_web_api_scouter workflow and its child workflow core_scouter.

Workflow Overview

The pi_web_api_scouter workflow:

1. Retrieves tag values from PI Web API
2. Delegates processing to core_scouter child workflow which:
   • Applies data quality gates
   • Aggregates data
   • Groups and holds data in Redis
   • Exports to PostgreSQL
   • Writes metrics
   • Optionally stores debug data package

---

1. PI Web API Scouter - Main Workflow Scenarios

1.1 Success Scenarios

Scenario 1.1.1: Happy Path - Complete Success

Description: Workflow completes successfully with valid data from PI Web API

Input:

• Valid model_name, model_id, schedule_name
• Valid pi_web_api_query with endpoint, period, max_count, api_timeout
• Valid model_tags with webids and configurations
• Valid filters, schema, table_name, retention_time

Expected Behavior:

• get_tag_values returns non-empty list of records
• Workflow proceeds to core_scouter
• All activities execute successfully
• Data is stored in PostgreSQL
• Metrics are written
• Workflow completes without errors

Assertions:

• PI Web API client called once with correct parameters
• Data exists in SQLite (PostgreSQL substitute)
• Data cached in Redis
• Metrics written
• No errors raised

---

Scenario 1.1.2: Success with Multiple Tags

Description: Workflow processes multiple tags successfully

Input:

• Multiple tags in model_tags (3+ tags)
• Each tag has valid webid, aggr_function, data_range, frequency

Expected Behavior:

• All tags retrieved from PI Web API
• All tags processed through quality gates
• All tags aggregated correctly
• All tags stored in database

Assertions:

• Number of records matches number of tags
• All tags present in final data
• Aggregation applied per tag configuration

---

Scenario 1.1.3: Success with Debug Data Package Enabled

Description: Workflow completes with debug_data_package=True

Input:

• All standard input
• debug_data_package: True

Expected Behavior:

• Normal workflow execution
• store_data_package activity called
• Data package stored in Redis

Assertions:

• store_data_package called once
• Data package key exists in Redis
• Package contains both data and held_data

---

1.2 Early Exit Scenarios

Scenario 1.2.1: Empty Data from PI Web API

Description: PI Web API returns empty data

Input:

• Valid configuration
• PI Web API returns empty DataFrame or empty list

Expected Behavior:

• get_tag_values returns empty list []
• Workflow checks if not data: and returns early
• core_scouter is NOT called
• Workflow completes without error

Assertions:

• PI Web API called once
• core_scouter NOT called
• No data in PostgreSQL
• No data in Redis (except possibly from previous runs)

---

Scenario 1.2.2: None Returned from PI Web API

Description: PI Web API returns None

Input:

• Valid configuration
• PI Web API returns None

Expected Behavior:

• get_tag_values returns None
• Workflow checks if not data: and returns early
• core_scouter is NOT called

Assertions:

• PI Web API called once
• core_scouter NOT called
• Workflow completes without error

---

1.3 Error Scenarios

Scenario 1.3.1: PI Web API Connection Error

Description: PI Web API client raises connection error

Input:

• Valid configuration
• PI Web API client raises PIMSRequestError or connection exception

Expected Behavior:

• get_tag_values catches exception
• Sends notification with PI_WEB_API_REQUEST_ERROR
• Raises exception (workflow fails after retries)

Assertions:

• Notification sent with correct error details
• Exception propagated to workflow
• Workflow fails (after retry policy exhausted)
• core_scouter NOT called

---

Scenario 1.3.2: PI Web API Timeout

Description: PI Web API request times out

Input:

• Valid configuration
• api_timeout set to low value
• PI Web API takes longer than timeout

Expected Behavior:

• Request times out
• Exception raised
• Notification sent
• Workflow fails after retries

Assertions:

• Timeout exception caught
• Notification sent
• Workflow fails

---

Scenario 1.3.3: Invalid Endpoint

Description: Invalid PI Web API endpoint provided

Input:

• Invalid endpoint path in pi_web_api_query

Expected Behavior:

• PI Web API client raises error
• Notification sent
• Workflow fails

Assertions:

• Error notification sent
• Workflow fails

---

2. CoreScouter - Child Workflow Scenarios

2.1 Success Scenarios

Scenario 2.1.1: Complete Processing Success

Description: All stages complete successfully

Input:

• Valid data from parent workflow
• Valid filters, model_tags, schema, table_name
• fill_missing_tags: False
• debug_data_package: False

Expected Behavior:

• data_quality_gate filters data
• aggregate_data aggregates by tag
• group_and_hold_data stores in Redis
• export_data_to_postgres writes to database
• write_metrics records metrics
• Workflow completes

Assertions:

• All activities called in correct order
• Data in PostgreSQL
• Data in Redis
• Metrics written
• store_data_package NOT called

---

Scenario 2.1.2: Success with Data Quality Filters

Description: Data quality filters applied successfully

Input:

• Data with some quality issues
• Filters configured with NULL_VALUES_FILTER or OUT_OF_BOUNDS_FILTER
• Policy set to DISCARD or WARN

Expected Behavior:

• Quality gate identifies issues
• Notification sent (WARNING level)
• If policy is DISCARD, bad rows removed
• Remaining data processed normally

Assertions:

• Quality issues detected
• Notification sent
• Bad data discarded if policy is DISCARD
• Good data processed

---

Scenario 2.1.3: Success with Different Aggregation Functions

Description: Different aggregation functions applied correctly

Input:

• Multiple tags with different aggr_function: avg, mdn, max, min, lts
• Time-series data with multiple points per tag

Expected Behavior:

• Each tag aggregated with its configured function
• Aggregated values correct for each function type

Assertions:

• avg calculates mean correctly
• mdn calculates median correctly
• max returns maximum value
• min returns minimum value
• lts returns latest value

---

Scenario 2.1.4: Success with Fill Missing Tags

Description: Missing tags filled with None

Input:

• fill_missing_tags: True
• Some tags missing from data

Expected Behavior:

• Missing tags added to data_hold with value None
• All expected tags present in final data

Assertions:

• Missing tags present with None value
• All model_tags represented in output

---

2.2 Early Exit Scenarios

Scenario 2.2.1: Empty Data After Grouping

Description: group_and_hold_data returns empty dict

Input:

• Data that results in empty held_data after grouping

Expected Behavior:

• group_and_hold_data returns {}
• Workflow checks if held_data == {}: and returns early
• export_data_to_postgres NOT called
• write_metrics NOT called
• store_data_package NOT called

Assertions:

• Early return after grouping
• No database export
• No metrics written
• Workflow completes without error

---

Scenario 2.2.2: Zero Affected Rows After Export

Description: PostgreSQL export returns zero affected rows

Input:

• Data that results in affected_rows: 0 from export

Expected Behavior:

• export_data_to_postgres returns {'affected_rows': 0}
• Workflow checks if data_exported.get('affected_rows', 0) <= 0: and returns early
• write_metrics NOT called
• store_data_package NOT called

Assertions:

• Early return after export
• No metrics written
• Workflow completes without error

---

2.3 Error Scenarios

Scenario 2.3.1: Redis Connection Error

Description: Redis unavailable during group_and_hold_data

Input:

• Valid data
• Redis connection fails

Expected Behavior:

• redis_repository.get() or redis_repository.set() raises exception
• Notification sent with REDIS_GET_ERROR or REDIS_SET_ERROR
• Exception propagated (workflow fails after retries)

Assertions:

• Error notification sent
• Workflow fails

---

3. Activity-Specific Scenarios

Note

: Activity-specific scenarios are better suited for unit tests rather than e2e tests. These scenarios are covered indirectly through workflow e2e tests. For detailed activity testing, refer to the unit test suite in tests/activities/.

---

5. Test Data Requirements

5.1 Valid Test Data Structure

{
    'model_name': 'test_model',
    'model_id': 'test_model_id',
    'schedule_name': 'test_schedule',
    'model_tags': {
        'tag1': {
            'webid': 'webid1',
            'aggr_function': 'avg',
            'data_range': [0, 100],
            'frequency': 60000,
        },
    },
    'trigger_laborious': False,
    'filters': {},
    'schema': 'test_schema',
    'table_name': 'test_table',
    'retention_time': 3600,
    'fill_missing_tags': False,
    'debug_data_package': False,
    'pi_web_api_query': {
        'endpoint': '/streamsets/recorded',
        'period': '*-1d',
        'max_count': 10,
        'api_timeout': 30,
    },
}

5.2 Mock PI Web API Response

DataFrame({
    'timestamp': ['2024-01-01 12:00:00+0000', ...],
    'name': ['tag1', 'tag2', ...],
    'value': [10.5, 20.3, ...],
    'tag': ['webid1', 'webid2', ...],
})

---

6. Test Implementation Notes

6.1 Test Organization

• Group tests by scenario category
• Use descriptive test names matching scenario IDs
• Share fixtures for common setup
• Use parametrized tests for similar scenarios

6.2 Assertions Checklist

For each scenario, verify:

• Correct activities called
• Correct parameters passed
• Expected data in storage (SQLite/Redis)
• Expected notifications sent
• Expected metrics written
• No unexpected errors
• Workflow state correct

6.3 Mock Configuration

• Mock PI Web API client responses
• Use fake Redis (fakeredis)
• Use fake MongoDB (mongomock)
• Use SQLite for PostgreSQL
• Mock notification handler
• Mock metrics controller

---

7. Priority Scenarios

High Priority (Must Test)

1. Scenario 1.1.1: Happy Path
2. Scenario 1.2.1: Empty Data
3. Scenario 1.3.1: API Connection Error
4. Scenario 2.1.1: Complete Processing
5. Scenario 2.2.1: Empty After Grouping
6. Scenario 2.3.4: PostgreSQL Error

Medium Priority (Should Test)

1. Scenario 1.1.3: Debug Package
2. Scenario 2.1.2: Quality Filters
3. Scenario 2.1.3: Different Aggregations
4. Scenario 3.3.5: Invalid Aggregation

Low Priority (Nice to Have)

1. Scenario 4.1.2: Retry Success
2. Scenario 5.1.1: Large Dataset
3. Scenario 5.3.1: Concurrent Execution