SIENTIAPDE-1712

Implement MinIO Offload and Retention Features

- Added configuration options for MinIO retention hours and offload threshold in README.
- Introduced MinIO payload offloading for large DataFrame-derived payloads, storing them as parquet files.
- Updated activities to utilize MinIO for data loading and cleanup, including new methods for offloading and retention management.
- Refactored existing activities to integrate MinIO functionality, ensuring compatibility with previous workflows.
- Removed the legacy MinioRepository class, consolidating MinIO operations under a new manager structure.
- Updated requirements to use the latest version of the sientia-dataops-library.

laborious/workflows/minimal_retrain.py

@@ -73,26 +73,25 @@ class MinimalRetrain:
         model_config = input_data.get('model_config', {})
 
         storage_result = await workflow.execute_activity_method(
-            Activities.query_to_minio,
+            Activities.load_query_with_minio_offload,
             {
                 **metadata,
                 'query': input_data['query'],
                 'datetime_columns': input_data.get('datetime_columns', []),
                 'model_name': model_name,
-                'object_prefix': f'retrain_datasets/{model_name}/data',
             },
             retry_policy=retry_policy,
             start_to_close_timeout=timedelta(seconds=600),
         )
 
-        if not storage_result['success']:
+        if isinstance(storage_result, dict) and storage_result.get('success') is False:
             return
 
         experiment_response = await workflow.execute_activity_method(
             Activities.retrain_model,
             {
                 **metadata,
-                'object_key': storage_result['object_key'],
+                'data': storage_result,
                 'model_name': model_name,
                 'model_config': model_config,
             },