SIENTIAPDE-1717: Updated documentation to reflect the removal of MinIO data download and cleanup operations from Model Manager workflows.

README.md

@@ -80,7 +80,7 @@ An enterprise-grade ML model training orchestration platform built on Temporal.
 ### Core Functionality
 - **ML Model Training Pipeline**: Complete training workflow from validation to deployment using MLFlow
 - **Polynomial Regression Support**: Configurable polynomial degree with interaction terms and mandatory scaler validation
-- **Automated File Cleanup**: Scheduled cleanup of stale files from MinIO and local filesystem
+- **Automated File Cleanup**: Scheduled cleanup of stale files from local filesystem
 - **Temporal Workflow Orchestration**: Robust workflow management with granular retry policies and fault tolerance
 - **Parameter Validation**: Defense-in-depth validation with business rules and type checking
 - **Experiment Tracking**: Comprehensive status tracking in PostgreSQL database
@@ -184,13 +184,13 @@ The Model Manager system uses a Temporal-based workflow architecture with clear
   - No exception raising on failure - allows workflow to handle errors gracefully
   - Integration with TrainingRepository for business logic separation
   - MLFlow model saving and artifact management
-  - MinIO object storage operations
   - **Polynomial Regression**: Support for configurable degree and interaction terms
   - **Training Predictions**: Calculates y_train_pred before denormalization for accurate metrics
-- **Cleanup**: File and directory cleanup operations
+- **Cleanup**: Local directory cleanup operations
   - `cleanup_temp_directories()`: Cleans local temporary directories
   - Configurable retention period (default: 24 hours)
   - Dry-run mode for testing
+  - No MinIO cleanup (files are managed by external processes)
 - **Key Features**:
   - Multiple inheritance pattern for unified activity interface
   - Parameter validation with business rules
@@ -219,16 +219,15 @@ The Model Manager system uses a Temporal-based workflow architecture with clear
 
 #### **Model Training Pipeline**
 ```
-Training Request → Parameter Validation → MinIO Data Download → 
+Training Request → Parameter Validation → Model Training → 
-Model Training → MLFlow Model Save → Resource Cleanup → Status Update
+MLFlow Model Save → Resource Cleanup → Status Update
 ```
 
 **Key Stages:**
 1. **Validation**: Experiment run ID and training parameters validation
-2. **Data Acquisition**: Download training data from MinIO storage
+2. **Training**: Execute ML model training with validated parameters (data provided in request)
-3. **Training**: Execute ML model training with validated parameters
+3. **Persistence**: Save trained model and artifacts to MLFlow
-4. **Persistence**: Save trained model and artifacts to MLFlow
+4. **Cleanup**: Remove temporary local directories and update experiment status
-5. **Cleanup**: Remove temporary files and update experiment status
 
 ### Security Architecture
 
@@ -266,10 +265,9 @@ The **TrainModel** workflow orchestrates the complete ML model training pipeline
 #### Execution Flow
 1. **Validate Experiment Run ID**: Critical validation before any DB updates
 2. **Validate Training Parameters**: Type checking + business rules validation
-3. **Download Training Data**: Fetch file from MinIO storage
+3. **Train Model**: Execute ML model training with validated parameters
-4. **Train Model**: Execute ML model training with validated parameters
+4. **Save to MLFlow**: Save trained model and artifacts to MLFlow
-5. **Save to MLFlow**: Save trained model and artifacts to MLFlow
+5. **Cleanup Resources**: Delete temporary local directories
-6. **Cleanup Resources**: Delete temporary files and MinIO data
 
 #### Key Features
 - **Granular Retry Policies**: Different strategies for network, training, MLFlow, database, and filesystem operations
@@ -316,18 +314,13 @@ The **TrainModel** workflow orchestrates the complete ML model training pipeline
 ```mermaid
 flowchart TD
     A[1. validate_experiment_run_id] --> B[2. validate_train_params]
-    B --> C[3. fetch_file_from_minio]
+    B --> C[3. train_model]
-    C --> D[4. train_model]
+    C --> D[4. cleanup_run_directory]
-    D --> E[5. save_model]
-    E --> F[6. cleanup_run_directory]
-    F --> G[7. delete_file_from_minio]
     
     B -.-> DB[(PostgreSQL)]
-    C -.-> MinIO[MinIO Storage]
+    C -.-> Training[ML Training]
-    D -.-> Training[ML Training]
+    C -.-> MLFlow[MLFlow]
-    E -.-> MLFlow[MLFlow]
+    D -.-> FS[Filesystem]
-    F -.-> FS[Filesystem]
-    G -.-> MinIO
 ```
 
 #### Retry Strategies
@@ -336,11 +329,9 @@ The workflow implements 5 different retry policies optimized for each operation
 
 | Operation Type | Initial Interval | Max Interval | Backoff | Max Attempts | Use Case |
 |---------------|------------------|--------------|---------|--------------|----------|
-| **Network** | 1s | 10s | 2.0x | 5 | MinIO operations (transient network errors) |
+| **Network** | 1s | 10s | 2.0x | 5 | Network operations (transient errors) |
 | **No Retry** | - | - | - | 1 | Training/Validation (permanent data errors) |
-| **MLFlow** | 5s | 30s | 2.0x | 3 | MLFlow operations (API timeouts) |
 | **Database** | 2s | 20s | 2.0x | 5 | PostgreSQL updates (lock contention) |
-| **Filesystem** | 2s | 10s | 1.5x | 3 | Cleanup operations (busy resources) |
 
 #### Business Validation Rules
 
@@ -363,21 +354,20 @@ The workflow validates comprehensive business rules beyond type checking:
 
 ### Cleanup Files Workflow (`cleanup_files.py`)
 
-The **CleanupFiles** workflow provides automated cleanup of stale files from MinIO storage and local temporary directories. It runs on a scheduled basis (default: daily at midnight UTC) to maintain storage hygiene.
+The **CleanupFiles** workflow provides automated cleanup of stale local temporary directories. It runs on a scheduled basis (default: daily at midnight UTC) to maintain storage hygiene.
 
 #### Purpose
-- **Storage Management**: Automatic removal of old files from MinIO and local filesystem
+- **Storage Management**: Automatic removal of old temporary directories from local filesystem
 - **Retention Policy**: Configurable retention period (default: 24 hours)
 - **Scheduled Execution**: Cron-based scheduling for automated cleanup
-- **Resource Optimization**: Prevents storage bloat and reduces costs
+- **Resource Optimization**: Prevents storage bloat and reduces disk usage
 
 #### Execution Flow
-1. **Cleanup MinIO Files**: Scan and delete files older than retention period from MinIO bucket
+1. **Cleanup Local Directories**: Remove temporary directories older than retention period
-2. **Cleanup Local Directories**: Remove temporary directories older than retention period
 
 #### Key Features
-- **Timestamp-Based Cleanup**: Uses filename/directory timestamps for age determination
+- **Timestamp-Based Cleanup**: Uses directory timestamps for age determination
-- **Pattern Matching**: Regex patterns for MinIO (`timestamp-filename`) and directories (`name_YYYYMMDD_HHMMSS_microseconds`)
+- **Pattern Matching**: Regex pattern for directories (`name_YYYYMMDD_HHMMSS_microseconds`)
 - **Configurable Retention**: Environment variable-based retention period
 - **Dry-Run Mode**: Test cleanup operations without actual deletion
 - **Idempotent**: Safe to run multiple times
@@ -386,7 +376,7 @@ The **CleanupFiles** workflow provides automated cleanup of stale files from Min
 #### Input Parameters
 ```json
 {
-  "bucket_name": "model-training"  // Optional, defaults to DEFAULT_CLEANUP_BUCKET env var
+  "temp_path": "model_manager/reports/temp"  // Optional, defaults to 'model_manager/reports/temp'
 }
 ```
 
@@ -416,20 +406,15 @@ flowchart TD
 
 | Operation Type | Initial Interval | Max Interval | Backoff | Max Attempts | Use Case |
 |---------------|------------------|--------------|---------|--------------|----------|
-| **Network** | 1s | 10s | 2.0x | 5 | MinIO operations (transient network errors) |
 | **No Retry** | - | - | - | 1 | Local filesystem operations (permanent errors) |
 
 #### Cleanup Patterns
 
-**MinIO Files:**
-- Pattern: `{timestamp}-{filename}` where timestamp is milliseconds since epoch
-- Example: `1638360000000-training_data.csv`
-- Retention: Files older than `CLEANUP_RETENTION_HOURS` are deleted
-
 **Local Directories:**
 - Pattern: `{name}_{YYYYMMDD}_{HHMMSS}_{microseconds}`
 - Example: `temp_20231201_143052_123456`
 - Retention: Directories older than `CLEANUP_RETENTION_HOURS` are deleted
+- Location: `model_manager/reports/temp/` by default
 
 ## Installation & Setup
 
@@ -1052,7 +1037,6 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
 
 ### Cleanup Metrics
 - Cleanup execution success/failure rates
-- Number of files deleted from MinIO
 - Number of directories cleaned from local filesystem
 - Cleanup duration and performance
 
@@ -1114,7 +1098,6 @@ These timeouts control how long each activity in workflows can run before timing
 |----------|-------------|---------|-------------------|
 | `TIMEOUT_VALIDATE_PARAMS` | Parameter validation timeout | `30` | Fast operation, no I/O |
 | `TIMEOUT_TRAIN_MODEL` | Model training timeout | `2700` | Large dataset processing (45 min) |
-| `TIMEOUT_DELETE_FILE` | Delete file from MinIO timeout | `120` | MinIO delete operation (2 min) |
 | `TIMEOUT_UPDATE_DATABASE` | Database update timeout | `30` | PostgreSQL update query (30 sec) |
 
 **Cleanup Workflow Timeouts:**