SIENTIAPDE-1712

Implement MinIO Offload and Retention Features

- Added configuration options for MinIO retention hours and offload threshold in README.
- Introduced MinIO payload offloading for large DataFrame-derived payloads, storing them as parquet files.
- Updated activities to utilize MinIO for data loading and cleanup, including new methods for offloading and retention management.
- Refactored existing activities to integrate MinIO functionality, ensuring compatibility with previous workflows.
- Removed the legacy MinioRepository class, consolidating MinIO operations under a new manager structure.
- Updated requirements to use the latest version of the sientia-dataops-library.

README.md

@@ -716,10 +716,80 @@ The Laborious system exposes comprehensive Prometheus metrics for operational vi
 | `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
 | `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
 | `POD_ID` | Kubernetes pod identifier | `None` | No |
+| `SIENTIA_MINIO_RETENTION_HOURS` | Retention window for offloaded MinIO objects | `168` | No |
+| `SIENTIA_MINIO_OFFLOAD_THRESHOLD_BYTES` | Offload threshold for DataFrame-derived payloads | `int(1.5 * 1024 * 1024)` | No |
 
 
 
 
+### MinIO Payload Offload & Retention
+
+Laborious uses MinIO to prevent Temporal workflow history from carrying very large in-memory payloads (pandas `DataFrame`-derived dicts).
+Whenever a payload exceeds a configurable size threshold, it is stored as a parquet file in MinIO and the workflow history only keeps a lightweight reference.
+
+Notes:
+- `SIENTIA_MINIO_OFFLOAD_THRESHOLD_BYTES` supports:
+  - Integer bytes (e.g. `"1572864"`)
+  - Float MiB (e.g. `"1.5"`), converted to bytes as `MiB * 1024 * 1024`
+- Fallback behavior uses `1.5 MiB` when the env var is missing or invalid.
+
+#### Wire Contract: `MinioDataFramePayload`
+
+The payload is implemented in `laborious/utils/models/minio_dataframe_payload.py`.
+The dataclass does **not** store a pandas `DataFrame` field.
+Instead, the `DataFrame` is only used at build time by:
+- `MinioDataFramePayload.from_dataframe(...)`
+- `MinioDataFramePayload.from_dataframe_to_dict(...)`
+
+After evaluation, the payload is serialized for Temporal as a flat dict:
+- **Inline path**: `data` contains `df.to_dict()`, and MinIO keys (`object_key`, `bucket`, ...) are absent / `None`.
+- **MinIO path**: the dict contains:
+  - `bucket`
+  - `object_key` (full MinIO object name returned by `MinioRepository.upload_file`)
+  - `object_prefix` (directory prefix used for cleanup listing; relative to the repository namespace)
+  - `uri` (best-effort `s3://<bucket>/<...>` string)
+  - `data` is omitted / set to `None`.
+
+When an activity needs pandas operations, it resolves references using:
+- `MinioDataFramePayload.dataframe_from_wire(...)`
+
+#### MinIO Object Naming (Retention Parsing)
+
+MinIO object basename (required convention):
+`{model_name}-{operation}-{timestamp}.parquet`
+
+Where:
+- `model_name`: model identifier used by the pipeline
+- `operation`: `initial` (SQL/query load before transform) or `transform` (after MLFlow transform)
+- `timestamp`: `DATETIME_FORMAT_FILENAME` from `sientia_do.temporal.constants`
+
+The relative object key (under the repository namespace) is always shaped as:
+`training_datasets/{model_name}/{basename}`
+
+Retention cleanup parses timestamps from the basename using the `-initial-` / `-transform-` anchors.
+`model_name` may contain hyphens; parsing is resilient to it.
+
+#### Workflows / Activities Integration
+
+Predictions batch uses MinIO offload as follows:
+1. `predictions_batch` calls `Activities.load_query_with_minio_offload`
+   - On success, it puts the serialized `MinioDataFramePayload` dict into `prediction_input["data"]`.
+2. `sub_workflows/prediction_process`
+   - Tracks which MinIO prefixes were referenced for offloaded payloads.
+   - Runs `Activities.cleanup_minio_objects_expired` in a `finally` block (only when MinIO offload happened).
+3. `laborious/activities/gates.py` and `laborious/activities/mlflow.py`
+   - Resolve offloaded payloads transparently before constructing pandas `DataFrame` objects.
+
+#### Legacy: `query_to_minio` (Minimal Retrain)
+
+`Storage.query_to_minio` is intentionally kept with its legacy behavior for `minimal_retrain`.
+It always uploads parquet and returns `{success, object_key, uri}`.
+It is not used by predictions batch MinIO offload, and its objects are not part of the retention parser described above.
+
+Legacy MinIO object layout (relative key):
+`training_datasets/{model_name}/{object_prefix}_{timestamp}.parquet` where `object_prefix` is sanitized
+(slashes replaced by underscores) to keep a stable model-level directory.
+
 ### OPC Configuration
 
 For multiple OPC servers, use the `OPC_CONFIG` environment variable: