gitea_admin/sientia-dataops-scouter_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SIENTIAPDE-1646

Browse Source 
Update project configuration and dependencies

- Added .mypy_cache and .cursor to .gitignore.
- Changed asyncio_default_fixture_loop_scope and asyncio_default_test_loop_scope to "session" in pyproject.toml.
- Updated e2e testing dependencies in requirements-dev.txt, replacing fakeredis and mongomock with pytest-httpserver.
- Updated requirements.txt to use sientia_do instead of a specific git commit.
- Modified sonar-project.properties to remove a file from coverage exclusions.
- Enhanced E2E test fixtures in e2e/conftest.py for better container management.
- Cleaned up e2e test files related to CoreScouter and PIWebAPIScouter workflows.
This commit is contained in:
vitor-aignosi
2026-05-25 12:58:03 -03:00
parent 909ad25b63
commit 34dbc886f3
65 changed files with 2591 additions and 2849 deletions
Download Patch File Download Diff File Expand all files Collapse all files

459
e2e/scenarios.md
View File

@@ -1,427 +1,138 @@
# Test Scenarios for PI Web API Scouter Workflow
# Scouter E2E scenario catalog
This document describes all possible test scenarios for the `pi_web_api_scouter` workflow and its child workflow `core_scouter`.
## Execution context
## Workflow Overview
The `pi_web_api_scouter` workflow:
1. Retrieves tag values from PI Web API
2. Delegates processing to `core_scouter` child workflow which:
- Applies data quality gates
- Aggregates data
- Groups and holds data in Redis
- Exports to PostgreSQL
- Writes metrics
- Optionally stores debug data package
- **MongoDB**, **Redis**, and **PostgreSQL** run as session-scoped testcontainers with autouse cleanup between tests.
- **PI Web API** is an in-process HTTP server (`e2e/pi_web_api_test_server.py`) speaking the wire format consumed by `PIWebAPIClient`.
- **Temporal** uses `WorkflowEnvironment.start_local()` and a single worker on `scouter-test-queue`.
- **Production code is not mocked** (except `Logger` and optional notification insert spy).
---
## 1. PI Web API Scouter - Main Workflow Scenarios
## 0. Harness smoke tests
### 1.1 Success Scenarios
Diagnostic-only checks under `e2e/test_harness_smoke.py`. They are not business scenarios; they exist to fail fast when the harness itself (Docker / containers / Temporal worker wiring) is broken, before the numbered suite runs.
#### Scenario 1.1.1: Happy Path - Complete Success
**Description**: Workflow completes successfully with valid data from PI Web API
### 0.0.1 Postgres schema ready (`test_postgres_schema_ready`)
**Input**:
- Valid `model_name`, `model_id`, `schedule_name`
- Valid `pi_web_api_query` with endpoint, period, max_count, api_timeout
- Valid `model_tags` with webids and configurations
- Valid filters, schema, table_name, retention_time
Confirms the autouse fixture executed `db_schema.sql` and `sientia_data.laborious_data` exists in the Postgres testcontainer.
**Expected Behavior**:
- `get_tag_values` returns non-empty list of records
- Workflow proceeds to `core_scouter`
- All activities execute successfully
- Data is stored in PostgreSQL
- Metrics are written
- Workflow completes without errors
### 0.0.2 Activities construct (`test_activities_construct`)
**Assertions**:
- PI Web API client called once with correct parameters
- Data exists in SQLite (PostgreSQL substitute)
- Data cached in Redis
- Metrics written
- No errors raised
Confirms the production `Activities` instance initializes against the Mongo/Redis/Postgres testcontainers without hanging (no `patch(...)` involved).
### 0.0.3 Temporal PI happy path (`test_temporal_pi_happy_path`)
End-to-end liveness check: `WorkflowEnvironment.start_local()` + worker + in-process PI server + `PIWebAPIScouter` complete without raising. Functional assertions for this flow live in scenario **2.1.1**.
---
#### Scenario 1.1.2: Success with Multiple Tags
**Description**: Workflow processes multiple tags successfully
## 1. Scouter workflow
**Input**:
- Multiple tags in `model_tags` (3+ tags)
- Each tag has valid webid, aggr_function, data_range, frequency
### 1.1.1 Happy path
**Expected Behavior**:
- All tags retrieved from PI Web API
- All tags processed through quality gates
- All tags aggregated correctly
- All tags stored in database
Seed `raw_<schedule>` with multiple documents, run `Scouter`, assert Postgres rows and Redis `last_data_timestamp:scouter:<schedule>`.
**Assertions**:
- Number of records matches number of tags
- All tags present in final data
- Aggregation applied per tag configuration
### 1.2.1 Incremental load
Pre-seed Redis timestamp; seed older and newer Mongo docs; assert only newer rows export and timestamp advances.
### 1.3.1 Empty Mongo early exit
Empty `raw_<schedule>`; workflow exits without Postgres rows or Redis timestamp key.
### 1.3.2 No Redis timestamp first run
No prior Redis key; all seeded Mongo docs load and timestamp is written after success.
---
#### Scenario 1.1.3: Success with Debug Data Package Enabled
**Description**: Workflow completes with `debug_data_package=True`
## 2. PIWebAPIScouter workflow
**Input**:
- All standard input
- `debug_data_package: True`
### 2.1.1 Happy path
**Expected Behavior**:
- Normal workflow execution
- `store_data_package` activity called
- Data package stored in Redis
PI server `success` mode with two tags; assert Postgres rows, Redis hold key, one HTTP request recorded.
**Assertions**:
- `store_data_package` called once
- Data package key exists in Redis
- Package contains both `data` and `held_data`
### 2.1.2 Multiple tags
Five tags with `avg` / `mdn` / `max` / `min` / `lts`; assert five distinct `variable` values and exact aggregated numbers in Postgres.
### 2.1.3 Debug data package
`debug_data_package=True`; assert `data_package_pi_web_api_scouter_*` Redis key with `data` and `held_data`.
### 2.2.1 Empty response early exit
Server `empty` mode; zero Postgres rows for `model_id`, one request recorded.
### 2.3.1 PI Web API connection error
Server `error` mode (HTTP 5xx); workflow fails; `PI_WEB_API_REQUEST_ERROR` notification in Mongo.
### 2.3.2 PI Web API timeout
Server `timeout` mode; workflow fails; `PI_WEB_API_REQUEST_ERROR` notification sent.
### 2.3.3 Invalid endpoint
Workflow uses `/invalid/endpoint` (404); workflow fails; `PI_WEB_API_REQUEST_ERROR` notification sent.
---
### 1.2 Early Exit Scenarios
## 3. CoreScouter subworkflow
#### Scenario 1.2.1: Empty Data from PI Web API
**Description**: PI Web API returns empty data
### 3.1.1 Complete processing success
**Input**:
- Valid configuration
- PI Web API returns empty DataFrame or empty list
Single tag, no filters; Postgres row and `held_data_*` Redis key; no `data_package_*` key.
**Expected Behavior**:
- `get_tag_values` returns empty list `[]`
- Workflow checks `if not data:` and returns early
- `core_scouter` is NOT called
- Workflow completes without error
### 3.1.2 Null values filter discard
**Assertions**:
- PI Web API called once
- `core_scouter` NOT called
- No data in PostgreSQL
- No data in Redis (except possibly from previous runs)
`NULL_VALUES_FILTER` DISCARD; one WARNING notification; only valid row in Postgres.
---
### 3.1.3 Null values filter warn
#### Scenario 1.2.2: None Returned from PI Web API
**Description**: PI Web API returns None
`NULL_VALUES_FILTER` WARN; notification sent; both rows in Postgres.
**Input**:
- Valid configuration
- PI Web API returns None
### 3.1.4 Out of bounds filter discard
**Expected Behavior**:
- `get_tag_values` returns None
- Workflow checks `if not data:` and returns early
- `core_scouter` is NOT called
`OUT_OF_BOUNDS_FILTER` DISCARD; in-range row only; WARNING notification.
**Assertions**:
- PI Web API called once
- `core_scouter` NOT called
- Workflow completes without error
### 3.1.5 Aggregation avg
---
Three points; Postgres `value` equals arithmetic mean (20.0).
### 1.3 Error Scenarios
### 3.1.6 Aggregation mdn
#### Scenario 1.3.1: PI Web API Connection Error
**Description**: PI Web API client raises connection error
Median equals 5.0 in Postgres.
**Input**:
- Valid configuration
- PI Web API client raises `PIMSRequestError` or connection exception
### 3.1.7 Aggregation max
**Expected Behavior**:
- `get_tag_values` catches exception
- Sends notification with `PI_WEB_API_REQUEST_ERROR`
- Raises exception (workflow fails after retries)
Maximum equals 15.0 in Postgres.
**Assertions**:
- Notification sent with correct error details
- Exception propagated to workflow
- Workflow fails (after retry policy exhausted)
- `core_scouter` NOT called
### 3.1.8 Aggregation min
---
Minimum equals 30.0 in Postgres.
#### Scenario 1.3.2: PI Web API Timeout
**Description**: PI Web API request times out
### 3.1.9 Aggregation lts
**Input**:
- Valid configuration
- `api_timeout` set to low value
- PI Web API takes longer than timeout
Last-by-timestamp value equals 300.0 in Postgres.
**Expected Behavior**:
- Request times out
- Exception raised
- Notification sent
- Workflow fails after retries
### 3.1.10 Fill missing tags
**Assertions**:
- Timeout exception caught
- Notification sent
- Workflow fails
`fill_missing_tags=True`; `held_data_*` contains missing tag keys with `None`.
---
### 3.1.11 Debug data package
#### Scenario 1.3.3: Invalid Endpoint
**Description**: Invalid PI Web API endpoint provided
`debug_data_package=True`; `data_package_*` Redis key decodes to dict with `data` and `held_data`.
**Input**:
- Invalid endpoint path in `pi_web_api_query`
### 3.2.1 Empty after grouping early exit
**Expected Behavior**:
- PI Web API client raises error
- Notification sent
- Workflow fails
Empty column-oriented `data`; no Postgres rows; no populated `held_data_*`.
**Assertions**:
- Error notification sent
- Workflow fails
### 3.3.1 Invalid aggregation function
---
`aggr_func= bogus`; `AGGREGATION_ISSUES` ERROR notification; bogus tag absent from Postgres.
## 2. CoreScouter - Child Workflow Scenarios
### 2.1 Success Scenarios
#### Scenario 2.1.1: Complete Processing Success
**Description**: All stages complete successfully
**Input**:
- Valid data from parent workflow
- Valid filters, model_tags, schema, table_name
- `fill_missing_tags: False`
- `debug_data_package: False`
**Expected Behavior**:
- `data_quality_gate` filters data
- `aggregate_data` aggregates by tag
- `group_and_hold_data` stores in Redis
- `export_data_to_postgres` writes to database
- `write_metrics` records metrics
- Workflow completes
**Assertions**:
- All activities called in correct order
- Data in PostgreSQL
- Data in Redis
- Metrics written
- `store_data_package` NOT called
---
#### Scenario 2.1.2: Success with Data Quality Filters
**Description**: Data quality filters applied successfully
**Input**:
- Data with some quality issues
- Filters configured with `NULL_VALUES_FILTER` or `OUT_OF_BOUNDS_FILTER`
- Policy set to `DISCARD` or `WARN`
**Expected Behavior**:
- Quality gate identifies issues
- Notification sent (WARNING level)
- If policy is `DISCARD`, bad rows removed
- Remaining data processed normally
**Assertions**:
- Quality issues detected
- Notification sent
- Bad data discarded if policy is `DISCARD`
- Good data processed
---
#### Scenario 2.1.3: Success with Different Aggregation Functions
**Description**: Different aggregation functions applied correctly
**Input**:
- Multiple tags with different `aggr_function`: `avg`, `mdn`, `max`, `min`, `lts`
- Time-series data with multiple points per tag
**Expected Behavior**:
- Each tag aggregated with its configured function
- Aggregated values correct for each function type
**Assertions**:
- `avg` calculates mean correctly
- `mdn` calculates median correctly
- `max` returns maximum value
- `min` returns minimum value
- `lts` returns latest value
---
#### Scenario 2.1.4: Success with Fill Missing Tags
**Description**: Missing tags filled with None
**Input**:
- `fill_missing_tags: True`
- Some tags missing from data
**Expected Behavior**:
- Missing tags added to `data_hold` with value `None`
- All expected tags present in final data
**Assertions**:
- Missing tags present with `None` value
- All model_tags represented in output
---
### 2.2 Early Exit Scenarios
#### Scenario 2.2.1: Empty Data After Grouping
**Description**: `group_and_hold_data` returns empty dict
**Input**:
- Data that results in empty `held_data` after grouping
**Expected Behavior**:
- `group_and_hold_data` returns `{}`
- Workflow checks `if held_data == {}:` and returns early
- `export_data_to_postgres` NOT called
- `write_metrics` NOT called
- `store_data_package` NOT called
**Assertions**:
- Early return after grouping
- No database export
- No metrics written
- Workflow completes without error
---
### 2.3 Error Scenarios
#### Scenario 2.3.1: Redis Connection Error
**Description**: Redis unavailable during `group_and_hold_data`
**Input**:
- Valid data
- Redis connection fails
**Expected Behavior**:
- `redis_repository.get()` or `redis_repository.set()` raises exception
- Notification sent with `REDIS_GET_ERROR` or `REDIS_SET_ERROR`
- Exception propagated (workflow fails after retries)
**Assertions**:
- Error notification sent
- Workflow fails
---
## 3. Activity-Specific Scenarios
> **Note**: Activity-specific scenarios are better suited for unit tests rather than e2e tests.
> These scenarios are covered indirectly through workflow e2e tests. For detailed activity testing,
> refer to the unit test suite in `tests/activities/`.
---
## 5. Test Data Requirements
### 5.1 Valid Test Data Structure
```python
{
'model_name': 'test_model',
'model_id': 'test_model_id',
'schedule_name': 'test_schedule',
'model_tags': {
'tag1': {
'webid': 'webid1',
'aggr_function': 'avg',
'data_range': [0, 100],
'frequency': 60000,
},
},
'trigger_laborious': False,
'filters': {},
'schema': 'test_schema',
'table_name': 'test_table',
'retention_time': 3600,
'fill_missing_tags': False,
'debug_data_package': False,
'pi_web_api_query': {
'endpoint': '/streamsets/recorded',
'period': '*-1d',
'max_count': 10,
'api_timeout': 30,
},
}
```
### 5.2 Mock PI Web API Response
```python
DataFrame({
'timestamp': ['2024-01-01 12:00:00+0000', ...],
'name': ['tag1', 'tag2', ...],
'value': [10.5, 20.3, ...],
'tag': ['webid1', 'webid2', ...],
})
```
---
## 6. Test Implementation Notes
### 6.1 Test Organization
- Group tests by scenario category
- Use descriptive test names matching scenario IDs
- Share fixtures for common setup
- Use parametrized tests for similar scenarios
### 6.2 Assertions Checklist
For each scenario, verify:
- [ ] Correct activities called
- [ ] Correct parameters passed
- [ ] Expected data in storage (SQLite/Redis)
- [ ] Expected notifications sent
- [ ] Expected metrics written
- [ ] No unexpected errors
- [ ] Workflow state correct
### 6.3 Mock Configuration
- Mock PI Web API client responses
- Use fake Redis (fakeredis)
- Use fake MongoDB (mongomock)
- Use SQLite for PostgreSQL
- Mock notification handler
- Mock metrics controller
---
## 7. Priority Scenarios
### High Priority (Must Test)
1. Scenario 1.1.1: Happy Path
2. Scenario 1.2.1: Empty Data
3. Scenario 1.3.1: API Connection Error
4. Scenario 2.1.1: Complete Processing
5. Scenario 2.2.1: Empty After Grouping
6. Scenario 2.3.4: PostgreSQL Error
### Medium Priority (Should Test)
1. Scenario 1.1.3: Debug Package
2. Scenario 2.1.2: Quality Filters
3. Scenario 2.1.3: Different Aggregations
4. Scenario 3.3.5: Invalid Aggregation
### Low Priority (Nice to Have)
1. Scenario 4.1.2: Retry Success
2. Scenario 5.1.1: Large Dataset
3. Scenario 5.3.1: Concurrent Execution
### 3.3.2 Postgres export failure surfaces
Drop `value` column before run; workflow fails; ERROR notification in Mongo.