gitea_admin/sientia-dataops-laborious-legacy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

QTZPOC-20

Browse Source 
QTZPOC-20: Improve structured logging and reporting for all model import pipeline failures by introducing a dedicated activity and centralizing error classification. This ensures consistent observability, with all workflow failures now reported before cleanup. Also, adjusted import step timeouts and refined manual run script cleanup logic.
This commit is contained in:
Vitor Pimentel dos Santos
2026-08-27 14:44:57 -03:00
parent e95e65b72b
commit 853c83e662
18 changed files with 857 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
docs/model-import.md
View File

@@ -353,7 +353,38 @@ change.
There is no rollback, no compensation, no resume and no reconciliation. A failed import stays failed;
trying again is a new upload and a new row.
## 15. How this contract is verified
## 15. What a failure looks like in the logs
Every failure — a row that cannot be claimed, a gate that rejected the bundle, an MLflow that refused
the connection, a verdict that could not be written — goes through the one channel in
`laborious/utils/models/import_errors.py`, which produces exactly two `logfmt` lines: the technical
one, whose message
carries the traceback (or the failure chain, root cause first, for anything that crossed an activity
boundary), and the user-facing sentence the notification logs. **Both carry the same fields**, so one
query returns the pair:
| Field | What it holds |
|---|---|
| `import_run_id` | the record's primary key — the id a user quotes to support |
| `import_workflow_id` | the Temporal workflow id |
| `import_step` | the step that failed, or `not_claimable` / `status_write` |
| `import_code` | the code of § 8, the same one written to the row |
| `error_type` | the failing exception's own type — `MlflowException`, not the `ApplicationError` wrapping it |
| `error_summary` | one bounded line of what it said |
| `import_gate` | the gate that rejected the bundle, when one did |
| `import_verdict_not_recorded` | the verdict a failed terminal write could not store |
The workflow metadata every other line already carries (`pod_id`, `runtime`, `model_id`, `model_name`,
`workflow_name`, `schedule_name`) is kept alongside them.
A failure at any pipeline step is reported from the workflow's `except`, before cleanup and before the
terminal write, so the log reads in the order things happened and the line's step can never disagree
with the row's. That report never changes the outcome: its own failure is swallowed, and the failure it
was reporting is re-raised untouched. The Temporal SDK still prints its own traceback for each failed
attempt — that is the per-attempt record, and it is not structured; the channel's line is the one to
query.
## 16. How this contract is verified
Two suites, and they prove different things.