SIENTIAPDE-1712

Implement MinIO Offload and Retention Features

- Added configuration options for MinIO retention hours and offload threshold in README.
- Introduced MinIO payload offloading for large DataFrame-derived payloads, storing them as parquet files.
- Updated activities to utilize MinIO for data loading and cleanup, including new methods for offloading and retention management.
- Refactored existing activities to integrate MinIO functionality, ensuring compatibility with previous workflows.
- Removed the legacy MinioRepository class, consolidating MinIO operations under a new manager structure.
- Updated requirements to use the latest version of the sientia-dataops-library.

laborious/worker/worker.py

@@ -153,6 +153,7 @@ async def main():
             other_workflows=[],
             activities=[
                 activities.load_custom_query,
+                activities.load_query_with_minio_offload,
                 activities.query_to_minio,
                 activities.retrain_model,
                 activities.update_production_model,
@@ -202,8 +203,9 @@ async def main():
                 activities.get_last_timestamp,
                 # OPC
                 activities.write_opc_data,
-                # Postgres
-                activities.load_custom_query,
+                # Postgres / MinIO offload
+                activities.load_query_with_minio_offload,
+                activities.cleanup_minio_objects_expired,
                 activities.repeat_last_prediction,
                 activities.export_data_to_postgres,
                 activities.write_metrics,