SIENTIAPDE-1712

Enhance E2E testing with MinIO support and update documentation

- Updated `requirements-dev.txt` to include MinIO support in testcontainers.
- Added a new fixture for MinIO container setup in `conftest.py` to facilitate E2E tests involving S3-compatible storage.
- Introduced a new test fixture for activities using a real MinIO container in `conftest.py`.
- Updated E2E test scenarios and documentation to reflect the integration of MinIO for offload uploads and clarified error handling in workflows.
- Refactored existing tests to improve clarity and maintainability.

e2e/scenarios.md

@@ -2,6 +2,13 @@
 
 This document describes all possible test scenarios for the `predictions_batch` workflow and its child workflows `prediction_process` and `format_and_export_prediction`.
 
+## Running automated E2E tests (`e2e/`)
+
+- **Runtime**: Docker (or a Docker-compatible daemon) must be available so [testcontainers](https://testcontainers.com/) can start **PostgreSQL** and **MinIO** containers.
+- **Dependencies**: install dev requirements (includes `testcontainers[postgres,minio]`).
+- **Invocation**: run only integration-marked tests, for example: `pytest e2e/ -m integration`.
+- **MinIO tests**: `e2e/test_minio_offload.py` exercises real S3 uploads; other E2E modules continue to mock MinIO on the worker used by most scenarios.
+
 ## Workflow Overview
 
 The `predictions_batch` workflow:
@@ -445,6 +452,8 @@ The `predictions_batch` workflow:
 
 ### 3.2 Error Scenarios
 
+These paths do **not** rely on Temporal activity retries for export failures: the write activities run once, errors are handled inside the activity, and the **workflow completes successfully** with degraded metadata on the persisted prediction (`prediction_confidence` and `comments`).
+
 #### Scenario 3.2.1: PI Web API Write Error
 **Description**: PI Web API export fails
 
@@ -453,15 +462,16 @@ The `predictions_batch` workflow:
 - PI Web API service unavailable or invalid config
 
 **Expected Behavior**:
-- `write_pi_web_api_data` raises exception
-- Notification sent
-- Workflow fails after retries
-- PostgreSQL export may not execute (depends on execution order)
+- `write_pi_web_api_data` surfaces the failure (exception handled in the activity layer)
+- Notification may be sent
+- Workflow **completes** (does not fail)
+- Prediction row is still written to PostgreSQL with error confidence **13** and a comment describing the PI error
+- Subsequent steps (e.g. OPC, Postgres) still run per workflow order with the updated prediction payload
 
 **Assertions**:
-- PI Web API error notification sent
-- Workflow fails
-- May impact subsequent exports
+- PI Web API error notification sent (when applicable)
+- Workflow completes
+- PostgreSQL contains the prediction with `prediction_confidence` 13 and expected `comments`
 
 ---
 
@@ -473,14 +483,15 @@ The `predictions_batch` workflow:
 - OPC server unavailable or invalid configuration
 
 **Expected Behavior**:
-- `write_opc_data` raises exception
-- Notification sent
-- Workflow fails after retries
+- `write_opc_data` reports failure without aborting the workflow
+- Notification may be sent
+- Workflow **completes** (does not fail)
+- Prediction row is written to PostgreSQL with OPC error confidence **12** and a comment indicating OPC write issues
 
 **Assertions**:
-- OPC error notification sent
-- Workflow fails
-- PostgreSQL export may not execute
+- OPC error notification sent (when applicable)
+- Workflow completes
+- PostgreSQL contains the prediction with `prediction_confidence` 12 and expected `comments`
 
 ---
 
@@ -497,7 +508,7 @@ The `predictions_batch` workflow:
 - `process_pi_web_api_response` detects partial failure
 - Error confidence set (13)
 - Notification sent for failed tag
-- Workflow completes with error confidence
+- Workflow completes with error confidence (single activity attempt; no retry loop)
 
 **Assertions**:
 - One tag written successfully