vitor-aignosi
10c7e292b9
Refactor ModelMetrics to utilize DriftAnalysis for drift detection - Replaced ModelAnalysis with DriftAnalysis in the ModelMetrics class to enhance drift detection capabilities. - Updated method signatures and documentation to reflect the changes in target_name and return values. - Adjusted data handling to ensure compatibility with the new analysis methods and improved clarity in the drift metrics dataframe preparation.
150 lines
8.1 KiB
Markdown
150 lines
8.1 KiB
Markdown
# E2E test run report
|
||
|
||
**Date:** 2026-05-08
|
||
**Command:** `source venv/bin/activate && rtk pytest e2e/ -v --tb=short`
|
||
**Environment:** Linux, Python 3.11.15, pytest 9.0.3
|
||
|
||
## Summary
|
||
|
||
| Metric | Count |
|
||
|--------|------:|
|
||
| Collected | 43 |
|
||
| **Passed** | **37** |
|
||
| **Failed** | **6** |
|
||
|
||
Full pytest output (compressed by `rtk`) was written to:
|
||
|
||
`~/.local/share/rtk/tee/1778268193_pytest.log`
|
||
|
||
---
|
||
|
||
## Failed tests (6)
|
||
|
||
1. `e2e/test_drift.py::test_drift_happy_path_persists_all_columns_with_reference_data`
|
||
2. `e2e/test_drift.py::test_drift_uses_30pct_fallback_when_reference_unavailable`
|
||
3. `e2e/test_drift.py::test_drift_chunk_period_seconds_preserves_seconds_in_chunk_start_date`
|
||
4. `e2e/test_predictions_batch_prediction_process.py::test_scenario_2_1_3_input_gate_triggers_repeat`
|
||
5. `e2e/test_predictions_batch_prediction_process.py::test_scenario_2_2_3_transform_gate_triggers_repeat`
|
||
6. `e2e/test_predictions_batch_prediction_process.py::test_scenario_2_3_3_predict_gate_triggers_repeat`
|
||
|
||
**Follow-up:** Jensen–Shannon NULLs for the drift happy-path scenario are fully traced (histogram out-of-range + `density=True`, vs NannyML’s leftover bin) in [DRIFT_JS_NULL_INVESTIGATION.md](./DRIFT_JS_NULL_INVESTIGATION.md).
|
||
|
||
---
|
||
|
||
## Failure group A — Drift: `drift_metrics.value` NOT NULL (2 tests)
|
||
|
||
### Error
|
||
|
||
`psycopg2.errors.NotNullViolation`: null value in column `value` of relation `sientia_data.drift_metrics` violates not-null constraint.
|
||
|
||
Example failing row (from logs): `feature=sensor_2`, `method=jensen_shannon`, `value=null`, with `kolmogorov_smirnov` / `wasserstein` populated for the same chunk.
|
||
|
||
The bulk INSERT built by `Activities.export_data_to_postgres` includes parameters such as `'value__4': None` for `jensen_shannon` on a given chunk.
|
||
|
||
### Root cause
|
||
|
||
`calculate_drift` (real `DriftAnalysis` + `ModelMetrics.get_drift_metrics`) can emit **NaN / missing** values for some metric methods (here **Jensen–Shannon**) on some features/chunks. Pandas/SQLAlchemy turns that into SQL `NULL`, while the E2E schema (mirroring production) defines:
|
||
|
||
```sql
|
||
value numeric NOT NULL
|
||
```
|
||
|
||
in `e2e/db_schema.sql` for `sientia_data.drift_metrics`.
|
||
|
||
### Recommended fixes (pick one consistent with product rules)
|
||
|
||
1. **Application layer (preferred if NULLs are never valid in production):** Before export, sanitize the drift dataframe — e.g. drop rows where `value` is null/NaN, or replace with a defined sentinel (only if product agrees), or skip emitting that method row when the statistic is undefined.
|
||
2. **Analytics layer:** Harden the Jensen–Shannon (and similar) paths so they always return a finite float for the supported inputs, or explicitly map “undefined” to an agreed numeric convention.
|
||
3. **Schema (only if product allows missing metrics):** Align DDL with reality by making `value` nullable — **only** if production and downstream consumers already expect missing metrics; the E2E comment in `test_drift.py` suggests `feature`/`timestamp` nullable cases exist, but `value` is still listed in `NON_NULL_DRIFT_COLUMNS`.
|
||
|
||
### Tests affected
|
||
|
||
- `test_drift_happy_path_persists_all_columns_with_reference_data`
|
||
- `test_drift_uses_30pct_fallback_when_reference_unavailable`
|
||
|
||
---
|
||
|
||
## Failure group B — Drift: chunk period seconds (1 test)
|
||
|
||
### Error
|
||
|
||
`AssertionError: expected at least one drift row to be persisted` (`e2e/test_drift.py:493`).
|
||
|
||
The workflow run completed without failing the test via `WorkflowFailureError`, but **no rows** were found in `sientia_data.drift_metrics` for the model.
|
||
|
||
### Likely cause
|
||
|
||
In `Drift.run`, persistence runs only when `if drift_data:` is truthy (`laborious/workflows/drift.py`). An **empty** drift result skips `export_data_to_postgres`, so the table stays empty.
|
||
|
||
Probable reasons:
|
||
|
||
- With **`chunk_period='s'`** and only **three** target rows (30 s spacing), `calculate_drift` / `DriftAnalysis` may produce **no output rows** (insufficient data per chunk or internal filters).
|
||
- Less likely here: time-window mismatch — timestamps are built from `datetime.now(UTC)` with `interval: 60` minutes from `drift_base.json`, so data should still fall in the window.
|
||
|
||
### Recommended fixes
|
||
|
||
1. **Test data:** Increase the number of second-spaced points (and/or span multiple chunk boundaries) so the analyzer reliably emits at least one chunk row.
|
||
2. **Product code:** If sub-minute chunking is required to always produce metrics when any data exists, adjust `ModelMetrics` / `DriftAnalysis` integration for small-N second buckets.
|
||
3. **Diagnostics:** Run the same scenario with `pytest -s` and confirm logs for “empty `drift_data`” vs export errors.
|
||
|
||
Solução a ser aplicada:
|
||
Dividir em dois testes:
|
||
|
||
1. Teste com dados suficientes para produzir pelo menos uma linha de drift
|
||
2. Teste com dados insuficientes para produzir pelo menos uma linha de drift, mas ja esperando os erros e validando que nao foi persistido nada
|
||
|
||
### Test affected
|
||
|
||
- `test_drift_chunk_period_seconds_preserves_seconds_in_chunk_start_date`
|
||
|
||
---
|
||
|
||
## Failure group C — Prediction REPEAT path: unique constraint on predictions (3 tests)
|
||
|
||
### Error
|
||
|
||
`psycopg2.errors.UniqueViolation`: duplicate key value violates unique constraint **`unique_model_id_timestamp`** on `sientia_data.predictions` (`model_id`, `timestamp`).
|
||
|
||
### Root cause
|
||
|
||
REPEAT is handled by `Activities.repeat_last_prediction` (registered from `sientia_do.temporal.activities.postgres_sync.Postgres`, via `Storage` inheritance in `laborious/activities/storage.py`). The E2E tests seed a prior row with a **fixed** timestamp:
|
||
|
||
```python
|
||
# e2e/test_predictions_batch_prediction_process.py — insert_sample_prediction
|
||
VALUES ({model_id}, '2024-01-01 12:00:00+00:00', ...)
|
||
```
|
||
|
||
The helper `assert_repeat` expects **two** rows with the same `(model_id, prediction, prediction_confidence, prediction_status)` but compares **only** those columns — not `timestamp` (`e2e/helpers.py`). So the intended behavior is: **duplicate business payload**, not necessarily **duplicate primary unique key** `(model_id, timestamp)`.
|
||
|
||
If `repeat_last_prediction` **INSERT**s a copy using the **same** `timestamp` as the last prediction, Postgres correctly rejects the second insert.
|
||
|
||
### Recommended fixes
|
||
|
||
1. **Implement REPEAT as UPSERT:** Use `ON CONFLICT (model_id, timestamp) DO UPDATE` (or the project’s existing `export_data_to_postgres` `on_conflict` pattern used in `FormatAndExportPrediction`) when writing the repeated prediction — if the product definition of REPEAT is “refresh same logical slot.”
|
||
2. **Insert with the current batch timestamp:** Copy numeric/status fields from the last row but set `timestamp` to the **new** batch instant (e.g. the workflow’s `last_timestamp` / slice timestamp). This matches `assert_repeat`, which does not assert on `timestamp`.
|
||
3. **Test-only change (weakest):** Relax assertions or change seed data — only if production behavior is “duplicate key is expected” (unlikely).
|
||
|
||
Solução a ser aplicada:
|
||
Alterar o teste para usar o timestamp da ultima execucao do batch, ao inves do timestamp do primeiro batch.
|
||
2. **Insert with the current batch timestamp:** Copy numeric/status fields from the last row but set `timestamp` to the **new** batch instant (e.g. the workflow’s `last_timestamp` / slice timestamp). This matches `assert_repeat`, which does not assert on `timestamp`.
|
||
|
||
### Tests affected
|
||
|
||
- `test_scenario_2_1_3_input_gate_triggers_repeat`
|
||
- `test_scenario_2_2_3_transform_gate_triggers_repeat`
|
||
- `test_scenario_2_3_3_predict_gate_triggers_repeat`
|
||
|
||
---
|
||
|
||
## Passing areas (sanity check)
|
||
|
||
All scenarios in `test_child_workflows_e2e.py`, `test_minimal_retrain.py`, `test_minio_offload.py`, `test_predictions_batch_format_export.py`, `test_predictions_batch_main_workflow.py` (except the three REPEAT cases above), and `test_simple_metrics.py` **passed** in this run.
|
||
|
||
---
|
||
|
||
## Suggested order of work
|
||
|
||
1. Fix **drift `value` NULL** — unblocks two high-value drift E2Es and may clarify the chunk-seconds scenario if exports start succeeding consistently.
|
||
2. Fix **`repeat_last_prediction` uniqueness** — unblocks three prediction-process E2Es; implementation likely lives in **`sientia_do`** Postgres activities, not in this repo.
|
||
3. Revisit **`test_drift_chunk_period_seconds`** data volume / expectations after drift export is stable.
|