SIENTIAPDE-1253: Update README.md to include documentation for the Train Model workflow and renumber the Minimal Retrain workflow. (+101, -1)

This commit is contained in:
Bruno Domingues
2025-10-15 16:35:13 -03:00
parent 98c216733b
commit 4ff796a56f

View File

@@ -17,7 +17,8 @@ A comprehensive AI model management platform for the complete machine learning l
- [Predictions Batch Workflow](#1-predictions-batch-workflow-predictions_batchpy)
- [Prediction Process Workflow](#2-prediction-process-workflow-prediction_processpy)
- [Format and Export Prediction Workflow](#3-format-and-export-prediction-workflow-format_and_export_predictionpy)
- [Minimal Retrain Workflow](#4-minimal-retrain-workflow-minimal_retrainpy)
- [Train Model Workflow](#4-train-model-workflow-train_modelpy)
- [Minimal Retrain Workflow](#5-minimal-retrain-workflow-minimal_retrainpy)
- [Installation & Setup](#installation--setup)
- [Prerequisites](#prerequisites)
- [Environment Setup](#environment-setup)
@@ -379,7 +380,99 @@ flowchart LR
C -.-> Prometheus[Prometheus]
```
### 4. Minimal Retrain Workflow (`minimal_retrain.py`)
### 4. Train Model Workflow (`train_model.py`)
The **TrainModel** workflow orchestrates the complete ML model training pipeline from parameter validation through model saving and cleanup.
#### Purpose
- **Model Training**: Complete ML model training pipeline
- **Parameter Validation**: Defense-in-depth validation with business rules
- **Resource Management**: Automatic cleanup of temporary resources
- **Status Tracking**: Comprehensive experiment tracking in database
- **Error Handling**: Robust error handling with detailed context logging
#### Execution Flow
1. **Validate Experiment Run ID**: Critical validation before any DB updates
2. **Validate Training Parameters**: Type checking + business rules validation
3. **Download Training Data**: Fetch file from MinIO storage
4. **Train Model**: Execute ML model training with validated parameters
5. **Save to MLFlow**: Save trained model and artifacts to MLFlow
6. **Cleanup Resources**: Delete temporary files and MinIO data
#### Key Features
- **Granular Retry Policies**: Different strategies for network, training, MLFlow, database, and filesystem operations
- **Configurable Timeouts**: Environment variable-based timeouts supporting files up to 200MB
- **Idempotent Cleanup**: Safe replay with Temporal workflow replay mechanism
- **Structured Logging**: Rich context in error messages for debugging
- **Business Validation**: 10 business rules including range checks, consistency validation, and data integrity
#### Input Parameters
```json
{
"experiment_run_id": 123,
"target_variable": "price",
"variable_columns": ["feature1", "feature2", "price"],
"train_size": 80,
"shuffle": true,
"use_scaler": true,
"include_ar": false,
"bucket_name": "ml-data",
"file_name": "training_data.csv",
"line_separator": "\n",
"decimal_separator": ".",
"lag_train": 5,
"lag_val": 3,
"rem_static_win": false,
"low_lim": {"feature1": 0.0, "feature2": 0.0, "price": 0.0},
"upp_lim": {"feature1": 100.0, "feature2": 100.0, "price": 1000.0},
"window": 10,
"experiment_name": "production_model_v1",
"removed_intervals": []
}
```
#### Architecture Diagram
```mermaid
flowchart TD
A[1. validate_experiment_run_id] --> B[2. validate_train_params]
B --> C[3. fetch_file_from_minio]
C --> D[4. train_model]
D --> E[5. save_model]
E --> F[6. cleanup_run_directory]
F --> G[7. delete_file_from_minio]
B -.-> DB[(PostgreSQL)]
C -.-> MinIO[MinIO Storage]
D -.-> Training[ML Training]
E -.-> MLFlow[MLFlow]
F -.-> FS[Filesystem]
G -.-> MinIO
```
#### Retry Strategies
The workflow implements 5 different retry policies optimized for each operation type:
| Operation Type | Initial Interval | Max Interval | Backoff | Max Attempts | Use Case |
|---------------|------------------|--------------|---------|--------------|----------|
| **Network** | 1s | 10s | 2.0x | 5 | MinIO operations (transient network errors) |
| **No Retry** | - | - | - | 1 | Training/Validation (permanent data errors) |
| **MLFlow** | 5s | 30s | 2.0x | 3 | MLFlow operations (API timeouts) |
| **Database** | 2s | 20s | 2.0x | 5 | PostgreSQL updates (lock contention) |
| **Filesystem** | 2s | 10s | 1.5x | 3 | Cleanup operations (busy resources) |
#### Business Validation Rules
The workflow validates 10 business rules beyond type checking:
1. **train_size**: Must be between 1-99%
2. **variable_columns**: Cannot be empty
3. **lag_train, lag_val, window**: Must be positive integers
4. **low_lim/upp_lim**: Must have same keys and low < upp for each variable
5. **target_variable**: Must be in variable_columns
6. **bucket_name, file_name, experiment_name**: Cannot be empty or whitespace
### 5. Minimal Retrain Workflow (`minimal_retrain.py`)
The **MinimalRetrain** workflow handles automated model retraining and production model updates.