From 14f91b40aa1400b7aea0894ba9408438526abfa1 Mon Sep 17 00:00:00 2001 From: Bruno Domingues Date: Mon, 30 Mar 2026 14:54:44 -0300 Subject: [PATCH] SIENTIAPDE-1717: Updated documentation to reflect the removal of MinIO data download and cleanup operations from Model Manager workflows. --- README.md | 67 +++++++++++++++++++++---------------------------------- 1 file changed, 25 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index 84005f0..60cd104 100644 --- a/README.md +++ b/README.md @@ -80,7 +80,7 @@ An enterprise-grade ML model training orchestration platform built on Temporal. ### Core Functionality - **ML Model Training Pipeline**: Complete training workflow from validation to deployment using MLFlow - **Polynomial Regression Support**: Configurable polynomial degree with interaction terms and mandatory scaler validation -- **Automated File Cleanup**: Scheduled cleanup of stale files from MinIO and local filesystem +- **Automated File Cleanup**: Scheduled cleanup of stale files from local filesystem - **Temporal Workflow Orchestration**: Robust workflow management with granular retry policies and fault tolerance - **Parameter Validation**: Defense-in-depth validation with business rules and type checking - **Experiment Tracking**: Comprehensive status tracking in PostgreSQL database @@ -184,13 +184,13 @@ The Model Manager system uses a Temporal-based workflow architecture with clear - No exception raising on failure - allows workflow to handle errors gracefully - Integration with TrainingRepository for business logic separation - MLFlow model saving and artifact management - - MinIO object storage operations - **Polynomial Regression**: Support for configurable degree and interaction terms - **Training Predictions**: Calculates y_train_pred before denormalization for accurate metrics -- **Cleanup**: File and directory cleanup operations +- **Cleanup**: Local directory cleanup operations - `cleanup_temp_directories()`: Cleans local temporary directories - Configurable retention period (default: 24 hours) - Dry-run mode for testing + - No MinIO cleanup (files are managed by external processes) - **Key Features**: - Multiple inheritance pattern for unified activity interface - Parameter validation with business rules @@ -219,16 +219,15 @@ The Model Manager system uses a Temporal-based workflow architecture with clear #### **Model Training Pipeline** ``` -Training Request → Parameter Validation → MinIO Data Download → -Model Training → MLFlow Model Save → Resource Cleanup → Status Update +Training Request → Parameter Validation → Model Training → +MLFlow Model Save → Resource Cleanup → Status Update ``` **Key Stages:** 1. **Validation**: Experiment run ID and training parameters validation -2. **Data Acquisition**: Download training data from MinIO storage -3. **Training**: Execute ML model training with validated parameters -4. **Persistence**: Save trained model and artifacts to MLFlow -5. **Cleanup**: Remove temporary files and update experiment status +2. **Training**: Execute ML model training with validated parameters (data provided in request) +3. **Persistence**: Save trained model and artifacts to MLFlow +4. **Cleanup**: Remove temporary local directories and update experiment status ### Security Architecture @@ -266,10 +265,9 @@ The **TrainModel** workflow orchestrates the complete ML model training pipeline #### Execution Flow 1. **Validate Experiment Run ID**: Critical validation before any DB updates 2. **Validate Training Parameters**: Type checking + business rules validation -3. **Download Training Data**: Fetch file from MinIO storage -4. **Train Model**: Execute ML model training with validated parameters -5. **Save to MLFlow**: Save trained model and artifacts to MLFlow -6. **Cleanup Resources**: Delete temporary files and MinIO data +3. **Train Model**: Execute ML model training with validated parameters +4. **Save to MLFlow**: Save trained model and artifacts to MLFlow +5. **Cleanup Resources**: Delete temporary local directories #### Key Features - **Granular Retry Policies**: Different strategies for network, training, MLFlow, database, and filesystem operations @@ -316,18 +314,13 @@ The **TrainModel** workflow orchestrates the complete ML model training pipeline ```mermaid flowchart TD A[1. validate_experiment_run_id] --> B[2. validate_train_params] - B --> C[3. fetch_file_from_minio] - C --> D[4. train_model] - D --> E[5. save_model] - E --> F[6. cleanup_run_directory] - F --> G[7. delete_file_from_minio] + B --> C[3. train_model] + C --> D[4. cleanup_run_directory] B -.-> DB[(PostgreSQL)] - C -.-> MinIO[MinIO Storage] - D -.-> Training[ML Training] - E -.-> MLFlow[MLFlow] - F -.-> FS[Filesystem] - G -.-> MinIO + C -.-> Training[ML Training] + C -.-> MLFlow[MLFlow] + D -.-> FS[Filesystem] ``` #### Retry Strategies @@ -336,11 +329,9 @@ The workflow implements 5 different retry policies optimized for each operation | Operation Type | Initial Interval | Max Interval | Backoff | Max Attempts | Use Case | |---------------|------------------|--------------|---------|--------------|----------| -| **Network** | 1s | 10s | 2.0x | 5 | MinIO operations (transient network errors) | +| **Network** | 1s | 10s | 2.0x | 5 | Network operations (transient errors) | | **No Retry** | - | - | - | 1 | Training/Validation (permanent data errors) | -| **MLFlow** | 5s | 30s | 2.0x | 3 | MLFlow operations (API timeouts) | | **Database** | 2s | 20s | 2.0x | 5 | PostgreSQL updates (lock contention) | -| **Filesystem** | 2s | 10s | 1.5x | 3 | Cleanup operations (busy resources) | #### Business Validation Rules @@ -363,21 +354,20 @@ The workflow validates comprehensive business rules beyond type checking: ### Cleanup Files Workflow (`cleanup_files.py`) -The **CleanupFiles** workflow provides automated cleanup of stale files from MinIO storage and local temporary directories. It runs on a scheduled basis (default: daily at midnight UTC) to maintain storage hygiene. +The **CleanupFiles** workflow provides automated cleanup of stale local temporary directories. It runs on a scheduled basis (default: daily at midnight UTC) to maintain storage hygiene. #### Purpose -- **Storage Management**: Automatic removal of old files from MinIO and local filesystem +- **Storage Management**: Automatic removal of old temporary directories from local filesystem - **Retention Policy**: Configurable retention period (default: 24 hours) - **Scheduled Execution**: Cron-based scheduling for automated cleanup -- **Resource Optimization**: Prevents storage bloat and reduces costs +- **Resource Optimization**: Prevents storage bloat and reduces disk usage #### Execution Flow -1. **Cleanup MinIO Files**: Scan and delete files older than retention period from MinIO bucket -2. **Cleanup Local Directories**: Remove temporary directories older than retention period +1. **Cleanup Local Directories**: Remove temporary directories older than retention period #### Key Features -- **Timestamp-Based Cleanup**: Uses filename/directory timestamps for age determination -- **Pattern Matching**: Regex patterns for MinIO (`timestamp-filename`) and directories (`name_YYYYMMDD_HHMMSS_microseconds`) +- **Timestamp-Based Cleanup**: Uses directory timestamps for age determination +- **Pattern Matching**: Regex pattern for directories (`name_YYYYMMDD_HHMMSS_microseconds`) - **Configurable Retention**: Environment variable-based retention period - **Dry-Run Mode**: Test cleanup operations without actual deletion - **Idempotent**: Safe to run multiple times @@ -386,7 +376,7 @@ The **CleanupFiles** workflow provides automated cleanup of stale files from Min #### Input Parameters ```json { - "bucket_name": "model-training" // Optional, defaults to DEFAULT_CLEANUP_BUCKET env var + "temp_path": "model_manager/reports/temp" // Optional, defaults to 'model_manager/reports/temp' } ``` @@ -416,20 +406,15 @@ flowchart TD | Operation Type | Initial Interval | Max Interval | Backoff | Max Attempts | Use Case | |---------------|------------------|--------------|---------|--------------|----------| -| **Network** | 1s | 10s | 2.0x | 5 | MinIO operations (transient network errors) | | **No Retry** | - | - | - | 1 | Local filesystem operations (permanent errors) | #### Cleanup Patterns -**MinIO Files:** -- Pattern: `{timestamp}-{filename}` where timestamp is milliseconds since epoch -- Example: `1638360000000-training_data.csv` -- Retention: Files older than `CLEANUP_RETENTION_HOURS` are deleted - **Local Directories:** - Pattern: `{name}_{YYYYMMDD}_{HHMMSS}_{microseconds}` - Example: `temp_20231201_143052_123456` - Retention: Directories older than `CLEANUP_RETENTION_HOURS` are deleted +- Location: `model_manager/reports/temp/` by default ## Installation & Setup @@ -1052,7 +1037,6 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa ### Cleanup Metrics - Cleanup execution success/failure rates -- Number of files deleted from MinIO - Number of directories cleaned from local filesystem - Cleanup duration and performance @@ -1114,7 +1098,6 @@ These timeouts control how long each activity in workflows can run before timing |----------|-------------|---------|-------------------| | `TIMEOUT_VALIDATE_PARAMS` | Parameter validation timeout | `30` | Fast operation, no I/O | | `TIMEOUT_TRAIN_MODEL` | Model training timeout | `2700` | Large dataset processing (45 min) | -| `TIMEOUT_DELETE_FILE` | Delete file from MinIO timeout | `120` | MinIO delete operation (2 min) | | `TIMEOUT_UPDATE_DATABASE` | Database update timeout | `30` | PostgreSQL update query (30 sec) | **Cleanup Workflow Timeouts:**