SIENTIAPDE-1253: Update README.md to include documentation for the Train Model workflow and renumber the Minimal Retrain workflow. (+101, -1)
This commit is contained in:
97
README.md
97
README.md
@@ -17,7 +17,8 @@ A comprehensive AI model management platform for the complete machine learning l
|
||||
- [Predictions Batch Workflow](#1-predictions-batch-workflow-predictions_batchpy)
|
||||
- [Prediction Process Workflow](#2-prediction-process-workflow-prediction_processpy)
|
||||
- [Format and Export Prediction Workflow](#3-format-and-export-prediction-workflow-format_and_export_predictionpy)
|
||||
- [Minimal Retrain Workflow](#4-minimal-retrain-workflow-minimal_retrainpy)
|
||||
- [Train Model Workflow](#4-train-model-workflow-train_modelpy)
|
||||
- [Minimal Retrain Workflow](#5-minimal-retrain-workflow-minimal_retrainpy)
|
||||
- [Installation & Setup](#installation--setup)
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Environment Setup](#environment-setup)
|
||||
@@ -379,7 +380,99 @@ flowchart LR
|
||||
C -.-> Prometheus[Prometheus]
|
||||
```
|
||||
|
||||
### 4. Minimal Retrain Workflow (`minimal_retrain.py`)
|
||||
### 4. Train Model Workflow (`train_model.py`)
|
||||
|
||||
The **TrainModel** workflow orchestrates the complete ML model training pipeline from parameter validation through model saving and cleanup.
|
||||
|
||||
#### Purpose
|
||||
- **Model Training**: Complete ML model training pipeline
|
||||
- **Parameter Validation**: Defense-in-depth validation with business rules
|
||||
- **Resource Management**: Automatic cleanup of temporary resources
|
||||
- **Status Tracking**: Comprehensive experiment tracking in database
|
||||
- **Error Handling**: Robust error handling with detailed context logging
|
||||
|
||||
#### Execution Flow
|
||||
1. **Validate Experiment Run ID**: Critical validation before any DB updates
|
||||
2. **Validate Training Parameters**: Type checking + business rules validation
|
||||
3. **Download Training Data**: Fetch file from MinIO storage
|
||||
4. **Train Model**: Execute ML model training with validated parameters
|
||||
5. **Save to MLFlow**: Save trained model and artifacts to MLFlow
|
||||
6. **Cleanup Resources**: Delete temporary files and MinIO data
|
||||
|
||||
#### Key Features
|
||||
- **Granular Retry Policies**: Different strategies for network, training, MLFlow, database, and filesystem operations
|
||||
- **Configurable Timeouts**: Environment variable-based timeouts supporting files up to 200MB
|
||||
- **Idempotent Cleanup**: Safe replay with Temporal workflow replay mechanism
|
||||
- **Structured Logging**: Rich context in error messages for debugging
|
||||
- **Business Validation**: 10 business rules including range checks, consistency validation, and data integrity
|
||||
|
||||
#### Input Parameters
|
||||
```json
|
||||
{
|
||||
"experiment_run_id": 123,
|
||||
"target_variable": "price",
|
||||
"variable_columns": ["feature1", "feature2", "price"],
|
||||
"train_size": 80,
|
||||
"shuffle": true,
|
||||
"use_scaler": true,
|
||||
"include_ar": false,
|
||||
"bucket_name": "ml-data",
|
||||
"file_name": "training_data.csv",
|
||||
"line_separator": "\n",
|
||||
"decimal_separator": ".",
|
||||
"lag_train": 5,
|
||||
"lag_val": 3,
|
||||
"rem_static_win": false,
|
||||
"low_lim": {"feature1": 0.0, "feature2": 0.0, "price": 0.0},
|
||||
"upp_lim": {"feature1": 100.0, "feature2": 100.0, "price": 1000.0},
|
||||
"window": 10,
|
||||
"experiment_name": "production_model_v1",
|
||||
"removed_intervals": []
|
||||
}
|
||||
```
|
||||
|
||||
#### Architecture Diagram
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[1. validate_experiment_run_id] --> B[2. validate_train_params]
|
||||
B --> C[3. fetch_file_from_minio]
|
||||
C --> D[4. train_model]
|
||||
D --> E[5. save_model]
|
||||
E --> F[6. cleanup_run_directory]
|
||||
F --> G[7. delete_file_from_minio]
|
||||
|
||||
B -.-> DB[(PostgreSQL)]
|
||||
C -.-> MinIO[MinIO Storage]
|
||||
D -.-> Training[ML Training]
|
||||
E -.-> MLFlow[MLFlow]
|
||||
F -.-> FS[Filesystem]
|
||||
G -.-> MinIO
|
||||
```
|
||||
|
||||
#### Retry Strategies
|
||||
|
||||
The workflow implements 5 different retry policies optimized for each operation type:
|
||||
|
||||
| Operation Type | Initial Interval | Max Interval | Backoff | Max Attempts | Use Case |
|
||||
|---------------|------------------|--------------|---------|--------------|----------|
|
||||
| **Network** | 1s | 10s | 2.0x | 5 | MinIO operations (transient network errors) |
|
||||
| **No Retry** | - | - | - | 1 | Training/Validation (permanent data errors) |
|
||||
| **MLFlow** | 5s | 30s | 2.0x | 3 | MLFlow operations (API timeouts) |
|
||||
| **Database** | 2s | 20s | 2.0x | 5 | PostgreSQL updates (lock contention) |
|
||||
| **Filesystem** | 2s | 10s | 1.5x | 3 | Cleanup operations (busy resources) |
|
||||
|
||||
#### Business Validation Rules
|
||||
|
||||
The workflow validates 10 business rules beyond type checking:
|
||||
|
||||
1. **train_size**: Must be between 1-99%
|
||||
2. **variable_columns**: Cannot be empty
|
||||
3. **lag_train, lag_val, window**: Must be positive integers
|
||||
4. **low_lim/upp_lim**: Must have same keys and low < upp for each variable
|
||||
5. **target_variable**: Must be in variable_columns
|
||||
6. **bucket_name, file_name, experiment_name**: Cannot be empty or whitespace
|
||||
|
||||
### 5. Minimal Retrain Workflow (`minimal_retrain.py`)
|
||||
|
||||
The **MinimalRetrain** workflow handles automated model retraining and production model updates.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user