455 lines
15 KiB
Markdown
455 lines
15 KiB
Markdown
# E2E Scenario Documentation - Predictions Batch
|
|
|
|
This document describes the end-to-end scenarios for `predictions_batch` and its child workflows:
|
|
`prediction_process` and `format_and_export_prediction`.
|
|
|
|
It is a functional reference of scenario behavior, inputs, and expected outcomes.
|
|
|
|
## Execution Context
|
|
|
|
- Tests run under `e2e/` and are marked with `@pytest.mark.integration`.
|
|
- PostgreSQL and MinIO are provisioned with testcontainers.
|
|
- `test_minio_offload.py` uses real MinIO I/O; other scenario suites may use stubs/mocks for optional outputs.
|
|
- Real OPC UA scenarios use `@pytest.mark.opc` and an in-process asyncua server (`e2e/test_opc_real_server.py`).
|
|
|
|
### Local validation
|
|
|
|
Use the existing project virtualenv and the shared `validate` script for unit/quality gates; run E2E separately (Docker required).
|
|
|
|
```bash
|
|
source ./venv/bin/activate
|
|
|
|
# Auto-fix + static checks (no pytest)
|
|
validate --fix --project-name=laborious
|
|
|
|
# Full unit + quality gate
|
|
validate --project-name=laborious
|
|
|
|
# E2E (integration)
|
|
pytest e2e/ --override-ini testpaths=e2e -m integration
|
|
|
|
# E2E (real OPC server only)
|
|
pytest e2e/test_opc_real_server.py --override-ini testpaths=e2e -m opc
|
|
```
|
|
|
|
---
|
|
|
|
## 1. Main Workflow Scenarios
|
|
Source: `e2e/test_predictions_batch_main_workflow.py`
|
|
|
|
### 1.1.1 Happy Path - Complete Success
|
|
**Summary**: Full workflow succeeds with valid query and default gate behavior.
|
|
|
|
**Description**:
|
|
- Query returns rows for a model.
|
|
- `prediction_process` runs transform and predict paths.
|
|
- Final prediction and transformed data are persisted.
|
|
|
|
**Expected Outcome**:
|
|
- Exactly one prediction row is created.
|
|
- Transform rows are created.
|
|
- Confidence/status/comments are success values.
|
|
|
|
### 1.2.1 SQL Query Execution Error
|
|
**Summary**: Invalid SQL leads to no persisted prediction.
|
|
|
|
**Description**:
|
|
- Input query is invalid.
|
|
- Load step fails and workflow follows error/short-circuit path.
|
|
|
|
**Expected Outcome**:
|
|
- No prediction rows for the model.
|
|
- Workflow does not require retry-loop assumptions in assertions.
|
|
|
|
### 1.2.2 Missing Required Parameters
|
|
**Summary**: Missing required fields prevent workflow completion path.
|
|
|
|
**Description**:
|
|
- Required input key (e.g. `query`) is omitted.
|
|
- Workflow fails to produce actionable input for child flow.
|
|
|
|
**Expected Outcome**:
|
|
- No prediction rows are persisted.
|
|
- Workflow handle may require explicit terminate in E2E harness.
|
|
|
|
### 1.2.3 Invalid Datetime Column Specification (de-prioritized)
|
|
**Summary**: Legacy invalid datetime-column case is retained only as low-priority legacy coverage.
|
|
|
|
**Description**:
|
|
- `datetime_columns` references non-existing columns.
|
|
- Behavior may vary by query shape and parser fallback.
|
|
|
|
**Expected Outcome**:
|
|
- No predictions persisted in the covered legacy assertion path.
|
|
- Scenario is not considered primary behavior coverage.
|
|
|
|
---
|
|
|
|
## 2. Prediction Process Scenarios
|
|
Source: `e2e/test_predictions_batch_prediction_process.py`
|
|
|
|
### 2.1 Input Gate Path Decisions
|
|
|
|
#### 2.1.1 CONTINUE
|
|
**Summary**: Input filter flags quality issue but allows continuation via default path.
|
|
|
|
**Description**:
|
|
- Input gate returns `CONTINUE`.
|
|
- MLFlow transform/predict are skipped.
|
|
- Export path persists default-style prediction with warning context.
|
|
|
|
#### 2.1.2 STOP
|
|
**Summary**: Input filter blocks processing.
|
|
|
|
**Description**:
|
|
- Input gate returns `STOP`.
|
|
- Workflow exits without export.
|
|
|
|
#### 2.1.3 REPEAT with history
|
|
**Summary**: Prior prediction is reused.
|
|
|
|
**Description**:
|
|
- Input gate returns `REPEAT`.
|
|
- `repeat_last_prediction` path is executed using existing historical row.
|
|
|
|
#### 2.1.4 REPEAT without history
|
|
**Summary**: Repeat requested but no previous prediction exists.
|
|
|
|
**Description**:
|
|
- Input gate returns `REPEAT`.
|
|
- No prior row is available to duplicate.
|
|
|
|
**Expected Outcome**:
|
|
- No new prediction rows are created for the model.
|
|
|
|
### 2.2 Transform Gate Decisions
|
|
|
|
#### 2.2.1 CONTINUE on transform response error
|
|
**Summary**: Transform response is degraded, but workflow continues.
|
|
|
|
#### 2.2.2 STOP on transform response error
|
|
**Summary**: Transform response error blocks downstream processing.
|
|
|
|
#### 2.2.3 REPEAT on transform response error
|
|
**Summary**: Transform response error triggers repeat-last-prediction path.
|
|
|
|
#### 2.2.4 STOP on transform content NaN
|
|
**Summary**: Content gate (`NAN_VALUES`) blocks on all-NaN transform payload.
|
|
|
|
### 2.3 Predict Gate Decisions
|
|
|
|
#### 2.3.1 CONTINUE on predict response error
|
|
**Summary**: Predict response degraded; workflow exports with degraded metadata.
|
|
|
|
#### 2.3.2 STOP on predict response error
|
|
**Summary**: Predict response error blocks export.
|
|
|
|
#### 2.3.3 REPEAT on predict response error
|
|
**Summary**: Predict response error routes to repeat-last-prediction.
|
|
|
|
### 2.4.1 Priority Conflict Resolution
|
|
**Summary**: Deterministic selection when multiple filters produce different flags.
|
|
|
|
**Description**:
|
|
- Multiple filters may produce `STOP`, `CONTINUE`, and/or `REPEAT`.
|
|
- `path_priority` defines precedence.
|
|
|
|
**Expected Outcome**:
|
|
- Highest-priority flag is applied consistently.
|
|
- Executed branch matches configured priority ordering.
|
|
|
|
---
|
|
|
|
## 3. Format and Export Scenarios
|
|
Source: `e2e/test_predictions_batch_format_export.py`
|
|
|
|
### 3.1 Output Combination Scenarios
|
|
|
|
#### 3.1.1 Default prediction export
|
|
**Summary**: Non-`None` path flag uses `format_default_prediction`.
|
|
|
|
**Description**:
|
|
- Default prediction is generated.
|
|
- Transform export is skipped.
|
|
- Optional outputs (PI/OPC) still execute when configured.
|
|
|
|
#### 3.1.2 OPC only
|
|
**Summary**: Postgres + OPC writes, PI Web API disabled.
|
|
|
|
#### 3.1.3 PI Web API only
|
|
**Summary**: Postgres + PI writes, OPC disabled.
|
|
|
|
#### 3.1.4 Postgres only
|
|
**Summary**: Both optional outputs disabled; only Postgres persistence and metrics.
|
|
|
|
#### 3.1.5 No transformed data export
|
|
**Summary**: Prediction is persisted; transformed table is not written.
|
|
|
|
### 3.2 Degraded-but-successful Completion
|
|
|
|
#### 3.2.1 PI Web API write error
|
|
**Summary**: PI write failure does not fail workflow.
|
|
|
|
**Expected Outcome**:
|
|
- Workflow completes.
|
|
- Prediction persisted with degraded confidence/comments (PI error semantics).
|
|
|
|
#### 3.2.2 OPC write error
|
|
**Summary**: OPC write failure does not fail workflow.
|
|
|
|
**Expected Outcome**:
|
|
- Workflow completes.
|
|
- Prediction persisted with OPC degraded confidence/comments.
|
|
|
|
#### 3.2.3 PI Web API partial write error
|
|
**Summary**: Partial PI acknowledgement is treated as degraded success.
|
|
|
|
**Expected Outcome**:
|
|
- Workflow completes.
|
|
- Prediction persisted with PI error confidence and descriptive comment.
|
|
|
|
#### 3.2.4 OPC session / channel error (confidence 14)
|
|
**Summary**: Tier-1 `BadSessionIdInvalid` (or equivalent session error) degrades the prediction without failing the workflow.
|
|
|
|
**Sources**:
|
|
- Mock: `e2e/test_predictions_batch_format_export.py::test_scenario_3_2_4_opc_session_bad_mock`
|
|
- Real server: `e2e/test_opc_real_server.py::test_scenario_3_2_4_opc_session_bad_real_server` (`@pytest.mark.opc`)
|
|
|
|
**Expected Outcome**:
|
|
- Workflow completes.
|
|
- `prediction_confidence` is 14.
|
|
- Comments contain `OPC UA session/channel error: BadSessionIdInvalid`.
|
|
|
|
#### 3.2.5 OPC write blocked during reconnect (confidence 14)
|
|
**Summary**: While reconnect holds the repository connection lock, writes fail fast with `reconnect_in_progress`.
|
|
|
|
**Sources**:
|
|
- Mock: `e2e/test_predictions_batch_format_export.py::test_scenario_3_2_5_opc_reconnect_in_progress_mock`
|
|
- Real server: `e2e/test_opc_real_server.py::test_scenario_3_2_5_opc_write_blocked_during_reconnect_real_server` (`@pytest.mark.opc`)
|
|
|
|
**Expected Outcome**:
|
|
- Workflow completes.
|
|
- `prediction_confidence` is 14.
|
|
- Comments contain `OPC UA reconnect in progress`.
|
|
|
|
### 3.3.1 Combined Optional Outputs (PI + OPC)
|
|
**Summary**: Both external output channels are enabled together.
|
|
|
|
**Description**:
|
|
- PI Web API and OPC configs are both present.
|
|
- Output mutation order matters for final persisted payload.
|
|
|
|
**Expected Outcome**:
|
|
- PI write executes before OPC write in workflow sequence.
|
|
- Final Postgres payload reflects any confidence/comment updates.
|
|
- OPC metrics are emitted when tag writes return response times.
|
|
|
|
---
|
|
|
|
## 4. MinIO Offload Scenarios
|
|
Source: `e2e/test_minio_offload.py`
|
|
|
|
### 4.1.1 Forced offload to MinIO
|
|
**Summary**: Very low threshold forces parquet upload.
|
|
|
|
**Description**:
|
|
- Payload is offloaded (`object_key` present, inline data absent/empty).
|
|
- Object is present in MinIO under `prediction_datasets/...`.
|
|
- Retrieval reconstructs the dataframe.
|
|
|
|
### 4.1.2 Full workflow with offloaded load payload
|
|
**Summary**: Offload path works during full `predictions_batch` execution.
|
|
|
|
**Expected Outcome**:
|
|
- Workflow completes.
|
|
- Prediction row is persisted.
|
|
|
|
### 4.2.1 Inline payload below threshold
|
|
**Summary**: Data remains inline when threshold is not exceeded.
|
|
|
|
**Expected Outcome**:
|
|
- Payload stores inline `data`.
|
|
- `object_key` is `None`.
|
|
- Downstream persistence behavior matches offload scenario semantics.
|
|
|
|
---
|
|
|
|
## 5. Drift Workflow Scenarios
|
|
Source: `e2e/test_drift.py`
|
|
|
|
The drift suite drives the **real** `sientia_model.analytics.drift_analysis.DriftAnalysis`
|
|
analyzer (no stubs / mocks). Each scenario exercises the full pipeline:
|
|
|
|
```
|
|
laborious_data (Postgres) -> load_custom_query
|
|
-> calculate_drift (DriftAnalysis univariate + multivariate)
|
|
-> export_data_to_postgres (sientia_data.drift_metrics)
|
|
```
|
|
|
|
The `mlflow_repository_stub` provides the reference-data CSV via
|
|
`download_artifacts`, and tests assert postgres rows in
|
|
`sientia_data.drift_metrics` against this canonical schema:
|
|
|
|
`id, model_id, feature, method, value, alert, chunk_index, chunk_start_date, chunk_end_date, accurate, timestamp, created_at`.
|
|
|
|
Tests assert behavioral / structural properties (column presence, NOT NULL
|
|
constraints, business-key invariants like uniform `timestamp` and stamped
|
|
`model_id`) rather than exact numeric drift scores, since those depend on
|
|
the real analyzer implementation and the synthetic data fed in.
|
|
|
|
### 5.1 Happy paths
|
|
|
|
#### D.1.1 Full pipeline persists all columns with reference data
|
|
**Summary**: 10 minutes of target data are inserted; a 10-row reference CSV
|
|
is configured via the MLflow stub. The `DriftAnalysis` runs end-to-end.
|
|
|
|
**Expected Outcome**:
|
|
- One row per `(chunk_index, feature, method)` plus a `multivariate` block
|
|
per chunk is persisted.
|
|
- Every column in the DDL is populated; `feature` is the only nullable column
|
|
per the new schema.
|
|
- `accurate=True` for every row (reference path).
|
|
- All three default univariate methods reach the analyzer.
|
|
- `model_id` is stamped as `text` and uniform across rows.
|
|
- `timestamp` equals `max(target_data.timestamp)` and is uniform across rows.
|
|
- `chunk_start_date` / `chunk_end_date` are persisted as ISO text and ordered.
|
|
- `p_value` is dropped before persistence.
|
|
|
|
#### D.1.2 30% fallback when reference data is unavailable
|
|
**Summary**: MLflow alias resolution is forced to fail so
|
|
`get_reference_data` returns `None`; `calculate_drift` falls back to the
|
|
first 30% of target rows as reference.
|
|
|
|
**Expected Outcome**:
|
|
- All persisted rows carry `accurate=False`.
|
|
- A `MODEL_METRICS_REFERENCE_DATA_WARNING` notification is emitted to MongoDB.
|
|
|
|
### 5.2 Failure paths
|
|
|
|
#### D.3.1 Empty target data short-circuits the workflow
|
|
**Summary**: `load_custom_query` returns no rows.
|
|
|
|
**Expected Outcome**:
|
|
- The workflow returns early and writes nothing to `sientia_data.drift_metrics`.
|
|
|
|
### 5.3 Configuration paths
|
|
|
|
#### D.4.2 Invalid `chunk_period` raises ValueError
|
|
**Summary**: Anything other than `min` / `s` is rejected by `calculate_drift`.
|
|
|
|
**Expected Outcome**:
|
|
- The workflow surfaces the `ValueError` ("Invalid chunk period: ...").
|
|
- No rows are persisted.
|
|
|
|
#### D.4.3 `chunk_period='s'` preserves seconds in `chunk_start_date`
|
|
**Summary**: Target data spans two minutes with samples at second-30
|
|
boundaries; the activity is configured with `chunk_period='s'`.
|
|
|
|
**Expected Outcome**:
|
|
- At least one persisted `chunk_start_date` carries `seconds=30`, proving
|
|
that the analyzer chunked at sub-minute granularity and the ISO-text
|
|
serialization preserved the boundary.
|
|
|
|
---
|
|
|
|
## 6. Simple Metrics Workflow Scenarios
|
|
Source: `e2e/test_simple_metrics.py`
|
|
|
|
Validates `sientia_data.simple_metrics` columns:
|
|
`id, model_id, metric, value, timestamp, data_size, interval_minutes, created_at`.
|
|
Note: ``timestamp`` is now nullable per the new DDL and ``model_id`` is ``text``.
|
|
|
|
### 6.1 Happy paths
|
|
|
|
#### S.1.1 rmse/mse/mae/r2 happy path
|
|
**Summary**: Prediction/target pairs are inserted; the activity computes all
|
|
four metrics with closed-form expected values.
|
|
|
|
**Expected Outcome**:
|
|
- One row per metric is persisted; all columns populated.
|
|
- `data_size` matches the joined row count and `interval_minutes=60`.
|
|
|
|
#### S.1.2 Subset metrics
|
|
**Summary**: Requesting `metrics=['rmse']` writes only the rmse row.
|
|
|
|
### 6.2 Edge cases
|
|
|
|
#### S.2.1 Zero-variance target returns r2=0
|
|
**Summary**: When all targets are equal, `ss_tot=0`; the activity must guard
|
|
against division by zero and return `r2=0`.
|
|
|
|
### 6.3 Failure paths
|
|
|
|
#### S.3.1 No overlapping data short-circuits persistence
|
|
**Summary**: With no `laborious_data` rows for the configured target variable
|
|
the workflow exits before `calculate_simple_metrics` and writes nothing.
|
|
|
|
---
|
|
|
|
## 7. Minimal Retrain Workflow Scenarios
|
|
Source: `e2e/test_minimal_retrain.py`
|
|
|
|
The MLflow registry is fully mocked (no real artifacts in test container).
|
|
Validates `sientia_data.log_retrain` columns:
|
|
`mlflow_experiment_id, mlflow_run_id, model_id, model_name, status, timestamp, version`.
|
|
Note: the new DDL drops the legacy ``id`` and ``created_at`` columns,
|
|
``mlflow_experiment_id`` is now ``int8`` and ``model_id`` is ``text``.
|
|
|
|
### 7.1 Happy path
|
|
|
|
#### MR.1.1 Successful retrain + promotion
|
|
**Summary**: Training data loads via MinIO offload, `wrapper.retrain` succeeds,
|
|
the new version is promoted to the `production` alias.
|
|
|
|
**Expected Outcome**:
|
|
- Report row has success status, `version='7'`, `mlflow_run_id='retrain-run-id'`,
|
|
`mlflow_experiment_id=4242` (`int8`).
|
|
- `mlflow.log_artifact` is called with the input CSV.
|
|
- `promote_to_alias` is called once with the resolved version and alias.
|
|
|
|
### 7.2 Failure paths
|
|
|
|
#### MR.2.1 Wrapper retrain raises
|
|
**Summary**: `wrapper.retrain` raises `RuntimeError`. The activity returns
|
|
`success=False`, `update_production_model` is NOT invoked.
|
|
|
|
**Expected Outcome**:
|
|
- Report row carries the error message and `version`/`mlflow_*` columns are NULL.
|
|
|
|
#### MR.2.2 Missing `model_config.target`
|
|
**Summary**: Empty model config short-circuits before any MLflow call.
|
|
|
|
**Expected Outcome**:
|
|
- Report row carries the explicit guard message.
|
|
- `get_cached_model` is never invoked.
|
|
|
|
#### MR.3.1 No training data
|
|
**Summary**: The training query returns no rows; the workflow does not
|
|
persist any report row. The current code raises plain `ValueError` from the
|
|
workflow function, which Temporal treats as a workflow-task failure (see
|
|
`CODE_ISSUES.md` issue MR-1).
|
|
|
|
---
|
|
|
|
## Input Contract Reference
|
|
|
|
Common scenario input fields:
|
|
- `schedule_name`
|
|
- `model_name`
|
|
- `model_id`
|
|
- `query`
|
|
- `schema`
|
|
- `table_name`
|
|
- `transform_table_name`
|
|
- `input_filters`
|
|
- `mlflow_transform_filters`
|
|
- `mlflow_predict_filters`
|
|
- `path_priority` (default order: `STOP`, `CONTINUE`, `REPEAT`)
|
|
- `save_transform`
|
|
- `prediction_store_policy`
|
|
- `model_config.target`
|
|
- `datetime_columns` (when query returns temporal fields)
|
|
|
|
Optional outputs:
|
|
- `opc_output_config`
|
|
- `pi_web_api_output_config` |