SIENTIAPDE-1350: Implement automated file cleanup workflow and update documentation. Added a new workflow for automated cleanup of stale files from MinIO and local filesystem, including scheduled execution and configurable retention. Updated the README with detailed information about the new workflow, its configuration, and related metrics. (118 additions, 5 deletions)

2025-11-27 10:22:38 -03:00
parent 5dcb62ef4b
commit 486e1756c9
2 changed files with 157 additions and 20 deletions
--- a/README.md
+++ b/README.md
@@ -15,6 +15,7 @@ An enterprise-grade ML model training orchestration platform built on Temporal.
  - [Security Architecture](#security-architecture)
 - [Workflows](#workflows)
  - [Train Model Workflow](#train-model-workflow-train_modelpy)
+  - [Cleanup Files Workflow](#cleanup-files-workflow-cleanup_filespy)
 - [Installation & Setup](#installation--setup)
  - [Prerequisites](#prerequisites)
  - [Environment Setup](#environment-setup)
@@ -66,11 +67,13 @@ An enterprise-grade ML model training orchestration platform built on Temporal.

 ### Core Functionality
 - **ML Model Training Pipeline**: Complete training workflow from validation to deployment using MLFlow
+- **Automated File Cleanup**: Scheduled cleanup of stale files from MinIO and local filesystem
 - **Temporal Workflow Orchestration**: Robust workflow management with granular retry policies and fault tolerance
 - **Parameter Validation**: Defense-in-depth validation with business rules and type checking
 - **Experiment Tracking**: Comprehensive status tracking in PostgreSQL database
 - **Resource Management**: Automatic cleanup of temporary files and storage
 - **Comprehensive Monitoring**: Prometheus metrics and detailed logging for operational visibility
+- **Scheduled Jobs**: Automated daily cleanup with configurable cron schedules

 ### Advanced Capabilities
 - **Granular Retry Policies**: Different strategies for network, training, MLFlow, database, and filesystem operations
@@ -133,38 +136,49 @@ The Model Manager system uses a Temporal-based workflow architecture with clear
  - Health check endpoints for Kubernetes liveness/readiness probes
  - Graceful shutdown with cleanup procedures
  - Multi-instance deployment support
-  - Dedicated task queue: `train_model-queue` for ML model training workflows
+  - Two dedicated task queues:
+    - `train_model-queue`: ML model training workflows
+    - `cleanup-queue`: File cleanup workflows
+  - Automated cleanup schedule management

 #### **Workflows (`model_manager/workflows/`)**
 - **TrainModel**: Complete ML model training pipeline from validation to deployment
+- **CleanupFiles**: Automated cleanup of stale files from MinIO and local filesystem
 - **Key Features**:
  - Temporal workflow definitions with granular retry policies
  - Parameter validation with business rules
  - Comprehensive error handling and status tracking
  - Configurable timeouts for different operation types
  - Automatic resource cleanup and management
+  - Scheduled cleanup jobs with cron expressions

 #### **Activities (`model_manager/activities/`)**
 - **Activities**: Main activity orchestrator combining all functionality through multiple inheritance
- **ExperimentTracking**: ML experiment lifecycle tracking and database operations (extends Postgres)
+- **ExperimentTracking**: ML experiment lifecycle tracking and database operations
  - Unified `update_experiment_run()` method for all experiment status updates
  - Support for three update types: STATUS, STATUS_WITH_ERROR, MODEL_SAVED
  - Automatic error message truncation (1024 chars)
-  - Connection pooling and retry logic via Postgres base class
- **Training**: ML model training operations (standalone activity, composition pattern)
+  - Connection pooling and retry logic
+- **Training**: ML model training operations with MLFlow and MinIO integration
  - Unified `train_model()` method for complete training pipeline
  - Receives pre-downloaded files (BytesIO) to avoid memory leaks
  - Returns success/failure status with TrainModelResult or error message
  - No exception raising on failure - allows workflow to handle errors gracefully
  - Integration with TrainingRepository for business logic separation
- **MLFlow**: Model saving and artifact management operations
- **MinIO**: Object storage operations for training data management
+  - MLFlow model saving and artifact management
+  - MinIO object storage operations
+- **Cleanup**: File and directory cleanup operations
+  - `cleanup_minio_files()`: Removes stale files from MinIO based on timestamp prefixes
+  - `cleanup_temp_directories()`: Cleans local temporary directories
+  - Configurable retention period (default: 24 hours)
+  - Dry-run mode for testing
 - **Key Features**:
  - Multiple inheritance pattern for unified activity interface
  - Parameter validation with business rules
  - MLFlow integration for model persistence
  - Comprehensive error handling and notification integration
  - Experiment tracking with automatic status management
+  - Timestamp-based file cleanup with regex pattern matching

 #### **Data Services (`model_manager/utils/`)**
 - **Connectors Config**: Environment variable-based configuration management
@@ -311,6 +325,79 @@ The workflow validates 10 business rules beyond type checking:
 5. **target_variable**: Must be in variable_columns
 6. **bucket_name, file_name, experiment_name**: Cannot be empty or whitespace

+### Cleanup Files Workflow (`cleanup_files.py`)
+
+The **CleanupFiles** workflow provides automated cleanup of stale files from MinIO storage and local temporary directories. It runs on a scheduled basis (default: daily at midnight UTC) to maintain storage hygiene.
+
+#### Purpose
+- **Storage Management**: Automatic removal of old files from MinIO and local filesystem
+- **Retention Policy**: Configurable retention period (default: 24 hours)
+- **Scheduled Execution**: Cron-based scheduling for automated cleanup
+- **Resource Optimization**: Prevents storage bloat and reduces costs
+
+#### Execution Flow
+1. **Cleanup MinIO Files**: Scan and delete files older than retention period from MinIO bucket
+2. **Cleanup Local Directories**: Remove temporary directories older than retention period
+
+#### Key Features
+- **Timestamp-Based Cleanup**: Uses filename/directory timestamps for age determination
+- **Pattern Matching**: Regex patterns for MinIO (`timestamp-filename`) and directories (`name_YYYYMMDD_HHMMSS_microseconds`)
+- **Configurable Retention**: Environment variable-based retention period
+- **Dry-Run Mode**: Test cleanup operations without actual deletion
+- **Idempotent**: Safe to run multiple times
+- **Error Handling**: Continues cleanup even if individual operations fail
+
+#### Input Parameters
+```json
+{
+  "bucket_name": "model-training"  // Optional, defaults to DEFAULT_CLEANUP_BUCKET env var
+}
+```
+
+#### Schedule Configuration
+
+The cleanup schedule is automatically created when the worker starts:
+
+| Configuration | Environment Variable | Default | Description |
+|--------------|---------------------|---------|-------------|
+| **Schedule ID** | `CLEANUP_SCHEDULE_ID` | `cleanup-files-daily` | Unique identifier for the schedule |
+| **Cron Expression** | `CLEANUP_CRON` | `0 0 * * *` | Daily at midnight UTC |
+| **Timezone** | `CLEANUP_TIMEZONE` | `UTC` | Timezone for cron execution |
+| **Task Queue** | `CLEANUP_TASK_QUEUE` | `cleanup-queue` | Dedicated task queue |
+| **Execution Timeout** | `CLEANUP_EXECUTION_TIMEOUT_HOURS` | `1` | Maximum execution time (hours) |
+| **Retention Period** | `CLEANUP_RETENTION_HOURS` | `24` | Files older than this are deleted |
+| **Dry Run** | `CLEANUP_DRY_RUN` | `false` | Test mode without actual deletion |
+| **Max Keys** | `MAX_KEYS_CLEANUP` | `1000` | MinIO list operation page size |
+
+#### Architecture Diagram
+```mermaid
+flowchart TD
+    A[Scheduled Trigger] --> B[cleanup_minio_files]
+    B --> C[cleanup_temp_directories]
+    
+    B -.-> MinIO[MinIO Storage]
+    C -.-> FS[Local Filesystem]
+```
+
+#### Retry Strategies
+
+| Operation Type | Initial Interval | Max Interval | Backoff | Max Attempts | Use Case |
+|---------------|------------------|--------------|---------|--------------|----------|
+| **Network** | 1s | 10s | 2.0x | 5 | MinIO operations (transient network errors) |
+| **No Retry** | - | - | - | 1 | Local filesystem operations (permanent errors) |
+
+#### Cleanup Patterns
+
+**MinIO Files:**
+- Pattern: `{timestamp}-{filename}` where timestamp is milliseconds since epoch
+- Example: `1638360000000-training_data.csv`
+- Retention: Files older than `CLEANUP_RETENTION_HOURS` are deleted
+
+**Local Directories:**
+- Pattern: `{name}_{YYYYMMDD}_{HHMMSS}_{microseconds}`
+- Example: `temp_20231201_143052_123456`
+- Retention: Directories older than `CLEANUP_RETENTION_HOURS` are deleted
+
 ## Installation & Setup

 ### Prerequisites
@@ -773,11 +860,23 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
 - `app_up`: Application health status (1=healthy, 0=unhealthy)
  - Labels: `pod_id`

+### Workflow Execution Metrics
+- `workflow_execution_total`: Total workflow executions
+  - Labels: `workflow_name`, `status` (success/failure)
+- `activity_execution_total`: Total activity executions
+  - Labels: `activity_name`, `status` (success/failure)
+
 ### Training Metrics
 - Training success/failure rates through notification system
 - Model save performance metrics
 - Experiment status tracking

+### Cleanup Metrics
+- Cleanup execution success/failure rates
+- Number of files deleted from MinIO
+- Number of directories cleaned from local filesystem
+- Cleanup duration and performance
+
 ## Configuration

 ### Environment Variables
@@ -786,6 +885,9 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
 |----------|-------------|---------|----------|
 | `TEMPORAL_HOST` | Temporal server address | `localhost:7233` | Yes |
 | `TEMPORAL_NAMESPACE` | Temporal namespace | `model-manager` | No |
+| `TEMPORAL_USE_TLS` | Enable TLS for Temporal connection | `false` | No |
+| `TRAIN_TASK_QUEUE` | Task queue for training workflows | `train_model-queue` | No |
+| `CLEANUP_TASK_QUEUE` | Task queue for cleanup workflows | `cleanup-queue` | No |
 | `POSTGRES_HOST` | PostgreSQL hostname | `localhost` | Yes |
 | `POSTGRES_PORT` | PostgreSQL port | `5432` | Yes |
 | `POSTGRES_USER` | PostgreSQL username | `sientia` | Yes |
@@ -811,6 +913,14 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
 | `MONGODB_PASSWORD` | MongoDB password | `wKZDbMNU1c` | Yes |
 | `MONGODB_DATABASE_NAME` | MongoDB database name | `sientia` | Yes |
 | `MONGODB_TTL_INDEX_HOURS` | MongoDB TTL index hours | `1` | No |
+| `CLEANUP_SCHEDULE_ID` | Cleanup schedule identifier | `cleanup-files-daily` | No |
+| `CLEANUP_CRON` | Cleanup cron expression | `0 0 * * *` | No |
+| `CLEANUP_TIMEZONE` | Cleanup schedule timezone | `UTC` | No |
+| `CLEANUP_EXECUTION_TIMEOUT_HOURS` | Cleanup execution timeout | `1` | No |
+| `CLEANUP_RETENTION_HOURS` | File retention period (hours) | `24` | No |
+| `CLEANUP_DRY_RUN` | Dry-run mode (no actual deletion) | `false` | No |
+| `MAX_KEYS_CLEANUP` | MinIO list operation page size | `1000` | No |
+| `DEFAULT_CLEANUP_BUCKET` | Default bucket for cleanup | `model-training` | No |
 | `LOG_LEVEL` | Application log level | `INFO` | No |
 | `PROJECT_NAME` | Project name for metrics | `model-manager` | No |
 | `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
@@ -819,18 +929,24 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa

 #### Workflow Activity Timeouts

-These timeouts control how long each activity in the training workflow can run before timing out. All values are in seconds and are designed to handle large files (up to 200MB).
+These timeouts control how long each activity in workflows can run before timing out. All values are in seconds.
+
+**Training Workflow Timeouts:**

 | Variable | Description | Default | Calculation Basis |
 |----------|-------------|---------|-------------------|
 | `TIMEOUT_VALIDATE_PARAMS` | Parameter validation timeout | `30` | Fast operation, no I/O |
-| `TIMEOUT_DOWNLOAD_FILE` | File download from MinIO timeout | `600` | 200MB @ 1MB/s with 3x buffer (10 min) |
-| `TIMEOUT_TRAIN_MODEL` | Model training timeout | `1800` | Large dataset processing (30 min) |
-| `TIMEOUT_SAVE_MODEL` | Save model to MLFlow timeout | `300` | Artifact upload and logging (5 min) |
-| `TIMEOUT_CLEANUP_DIRECTORY` | Cleanup temporary directory timeout | `60` | Local filesystem operation (1 min) |
-| `TIMEOUT_DELETE_FILE` | Delete file from MinIO timeout | `60` | MinIO delete operation (1 min) |
+| `TIMEOUT_TRAIN_MODEL` | Model training timeout | `2700` | Large dataset processing (45 min) |
+| `TIMEOUT_DELETE_FILE` | Delete file from MinIO timeout | `120` | MinIO delete operation (2 min) |
 | `TIMEOUT_UPDATE_DATABASE` | Database update timeout | `30` | PostgreSQL update query (30 sec) |

+**Cleanup Workflow Timeouts:**
+
+| Variable | Description | Default | Calculation Basis |
+|----------|-------------|---------|-------------------|
+| `TIMEOUT_CLEANUP_MINIO` | MinIO cleanup timeout | `300` | Scan and delete multiple files (5 min) |
+| `TIMEOUT_CLEANUP_LOCAL` | Local cleanup timeout | `120` | Scan and delete directories (2 min) |
+
 **Note**: These timeouts can be adjusted based on your infrastructure performance and file sizes. If you're processing files larger than 200MB or have slower network/compute resources, increase these values accordingly.

 ## Development
@@ -908,18 +1024,23 @@ model_manager/
 │   ├── __init__.py
 │   ├── activities.py       # Main activities orchestrator (combines all activities)
 │   ├── experiment_tracking.py # Experiment status tracking and database operations
-│   ├── training.py         # ML model training operations
-│   ├── minio.py            # MinIO object storage operations
-│   └── mlflow.py           # MLFlow model saving and artifact management
+│   ├── training.py         # ML model training operations (includes MLFlow & MinIO)
+│   └── cleanup.py          # File and directory cleanup operations
 ├── workflows/               # Temporal workflow definitions
 │   ├── __init__.py
-│   └── train_model.py      # Complete ML model training workflow
+│   ├── train_model.py      # Complete ML model training workflow
+│   └── cleanup_files.py    # Automated file cleanup workflow
+├── schedules/               # Temporal schedule configurations
+│   ├── __init__.py
+│   └── cleanup_schedule.py # Cleanup schedule creation and management
 ├── worker/                  # Worker implementation
 │   ├── __init__.py
 │   └── worker.py           # Main worker orchestrator (Temporal client setup)
 ├── utils/                   # Utility functions and helpers
 │   ├── __init__.py
 │   ├── connectors_config.py # Environment-based configuration builders
+│   ├── exceptions.py       # Custom exception definitions
+│   ├── logger_helper.py    # Logger initialization utilities
 │   ├── models/             # Data models and schemas
 │   │   ├── __init__.py
 │   │   ├── train_model_params.py # Training parameters model
@@ -928,7 +1049,19 @@ model_manager/
 │   └── repository/         # Data access layer
 │       ├── __init__.py
 │       ├── training_repository.py # Training business logic
-│       └── model_repository.py    # MLFlow artifact management
+│       ├── model_repository.py    # MLFlow artifact management
+│       └── storage_repository.py  # MinIO storage operations
+├── sientia/                 # Sientia-specific implementations
+│   ├── __init__.py
+│   ├── exceptions.py       # Custom exceptions
+│   ├── metrics.py          # Business metrics
+│   ├── models.py           # ML model implementations
+│   ├── model_serving.py    # Model serving utilities
+│   ├── reports.py          # Report generation
+│   └── utils.py            # Utility functions
+├── reports/                 # Report templates and temporary files
+│   ├── header.html         # HTML report header template
+│   └── temp/               # Temporary report files (cleaned up automatically)
 ├── metrics.py               # Prometheus metrics definitions
 └── __init__.py
 ```
@@ -994,16 +1127,22 @@ export LOG_LEVEL=DEBUG
 ### Key Parameters

 - **Worker Concurrency**: Adjust `max_concurrent_workflow_tasks` and `max_concurrent_activities`
+  - Training workflows: 10 concurrent tasks/activities
+  - Cleanup workflows: 20 concurrent tasks/activities
 - **Connection Pools**: Optimize database connection pool sizes
 - **Model Retention**: Configure MLFlow model retention based on requirements
 - **Batch Sizes**: Adjust data processing batch sizes for optimal throughput
+- **Cleanup Performance**: Tune `MAX_KEYS_CLEANUP` for MinIO list operation page size

 ### Scaling Considerations

 - **Horizontal Scaling**: Deploy multiple worker instances
- **Task Queue Distribution**: Use multiple task queues for different workflow types
+- **Task Queue Distribution**: Two dedicated task queues for workflow isolation
+  - `train_model-queue`: Training workflows
+  - `cleanup-queue`: Cleanup workflows
 - **Database Performance**: Optimize indexes and connection pooling
 - **MLFlow Performance**: Configure appropriate model serving resources
+- **Storage Management**: Adjust cleanup retention period based on storage capacity and costs

 ## Contributing

--- a/todo-list.txt
+++ b/todo-list.txt
@@ -1,6 +1,4 @@
 - Criar um gráfico no grafana para cada nova atividade.
- Atualizar a documentação dos métodos alterados.
- Atualizar a documentação do projeto.

 - Atualizar o .github/workflows/quality-gate.yml para usar os pipelines genéricos do github;
  Criar um workflow para fazer o deploy no suse.