gitea_admin/sientia-dataops-model-manager
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SIENTIAPDE-1255: Refactor Model Manager to focus on ML model training pipeline and simplify architecture. This includes removing prediction workflows, updating core functionality, and improving parameter validation and resource management.

Browse Source 
README.md | 411 deletions(-) 117 insertions(+)
1 file changed, 117 insertions(+), 411 deletions(-)
This commit is contained in:
Bruno Domingues
2025-10-16 18:26:56 -03:00
parent 7abd951806
commit e191e7849f
10 changed files with 42 additions and 2081 deletions
Download Patch File Download Diff File Expand all files Collapse all files

368
README.md
View File

@@ -1,6 +1,6 @@
# Sientia DataOps Model Manager
A comprehensive AI model management platform for the complete machine learning lifecycle. Handles model training, versioning, deployment, monitoring, and governance. Streamlines MLOps workflows with centralized model registry, automated pipelines, performance tracking, and enterprise-grade compliance features.
An enterprise-grade ML model training orchestration platform built on Temporal. Provides robust, scalable workflows for training machine learning models with comprehensive validation, experiment tracking, and automated resource management. Integrates seamlessly with MLFlow for model persistence and PostgreSQL for experiment tracking.
## 📑 Table of Contents
@@ -14,11 +14,7 @@ A comprehensive AI model management platform for the complete machine learning l
- [Data Flow Architecture](#data-flow-architecture)
- [Security Architecture](#security-architecture)
- [Workflows](#workflows)
- [Predictions Batch Workflow](#1-predictions-batch-workflow-predictions_batchpy)
- [Prediction Process Workflow](#2-prediction-process-workflow-prediction_processpy)
- [Format and Export Prediction Workflow](#3-format-and-export-prediction-workflow-format_and_export_predictionpy)
- [Train Model Workflow](#4-train-model-workflow-train_modelpy)
- [Minimal Retrain Workflow](#5-minimal-retrain-workflow-minimal_retrainpy)
- [Train Model Workflow](#train-model-workflow-train_modelpy)
- [Installation & Setup](#installation--setup)
- [Prerequisites](#prerequisites)
- [Environment Setup](#environment-setup)
@@ -68,19 +64,19 @@ A comprehensive AI model management platform for the complete machine learning l
## Features
### Core Functionality
- **Batch Prediction Processing**: High-throughput ML model inference using MLFlow models
- **Temporal Workflow Orchestration**: Robust workflow management with automatic retry policies and fault tolerance
- **Data Quality Gates**: Configurable filtering for data validation, MLFlow API responses, and custom validation rules
- **Multi-Model Support**: Flexible ML model management with retention policies and versioning
- **Real-time Data Export**: PostgreSQL persistence for data storage
- **ML Model Training Pipeline**: Complete training workflow from validation to deployment using MLFlow
- **Temporal Workflow Orchestration**: Robust workflow management with granular retry policies and fault tolerance
- **Parameter Validation**: Defense-in-depth validation with business rules and type checking
- **Experiment Tracking**: Comprehensive status tracking in PostgreSQL database
- **Resource Management**: Automatic cleanup of temporary files and storage
- **Comprehensive Monitoring**: Prometheus metrics and detailed logging for operational visibility
### Advanced Capabilities
- **Incremental Data Processing**: Timestamp-based data loading to avoid reprocessing
- **Configurable Data Retention**: Model retention policies with automatic cleanup
- **Granular Retry Policies**: Different strategies for network, training, MLFlow, database, and filesystem operations
- **Configurable Timeouts**: Environment variable-based timeouts supporting large training files (up to 200MB)
- **Notification System**: Integrated alerting and notification management via MongoDB
- **Scalable Architecture**: Kubernetes-ready deployment with horizontal scaling support
- **Model Retraining**: Automated model retraining workflows with production model updates
- **MLFlow Integration**: Seamless model and artifact persistence to MLFlow tracking server
### Development & Quality Assurance
- **Code Quality Tools**: Ruff (linting/formatting), mypy (type checking), Bandit (security analysis)
@@ -136,18 +132,16 @@ The Model Manager system uses a Temporal-based workflow architecture with clear
- Health check endpoints for Kubernetes liveness/readiness probes
- Graceful shutdown with cleanup procedures
- Multi-instance deployment support
- Two dedicated task queues: `predictions_batch-queue` and `minimal_retrain-queue`
- Dedicated task queue: `train_model-queue` for ML model training workflows
#### **Workflows (`model_manager/workflows/`)**
- **PredictionsBatch**: Main entry point for batch prediction pipelines
- **PredictionProcess**: Core prediction pipeline with MLFlow integration
- **FormatAndExportPrediction**: Data formatting and export operations
- **MinimalRetrain**: Automated model retraining and deployment
- **TrainModel**: Complete ML model training pipeline from validation to deployment
- **Key Features**:
- Temporal workflow definitions with retry policies
- Child workflow orchestration and delegation
- Comprehensive error handling and recovery
- Configurable timeout and retry strategies
- Temporal workflow definitions with granular retry policies
- Parameter validation with business rules
- Comprehensive error handling and status tracking
- Configurable timeouts for different operation types
- Automatic resource cleanup and management
#### **Activities (`model_manager/activities/`)**
- **Activities**: Main activity orchestrator combining all functionality through multiple inheritance
@@ -162,13 +156,12 @@ The Model Manager system uses a Temporal-based workflow architecture with clear
- Returns success/failure status with TrainModelResult or error message
- No exception raising on failure - allows workflow to handle errors gracefully
- Integration with TrainingRepository for business logic separation
- **Gates**: Data quality validation and filtering mechanisms
- **MLFlow**: Model transformation and prediction operations
- **MinIO**: Object storage operations for file management
- **MLFlow**: Model saving and artifact management operations
- **MinIO**: Object storage operations for training data management
- **Key Features**:
- Multiple inheritance pattern for unified activity interface
- Configurable filter policies and validation rules
- MLFlow model serving integration with configurable flavors
- Parameter validation with business rules
- MLFlow integration for model persistence
- Comprehensive error handling and notification integration
- Experiment tracking with automatic status management
@@ -188,17 +181,18 @@ The Model Manager system uses a Temporal-based workflow architecture with clear
### Data Flow Architecture
#### **1. Batch Prediction Pipeline**
#### **Model Training Pipeline**
```
Input Data (PostgreSQL) → Data Quality Gates → MLFlow Transform →
MLFlow Prediction → Response Validation → Export (PostgreSQL)
Training Request → Parameter Validation → MinIO Data Download →
Model Training → MLFlow Model Save → Resource Cleanup → Status Update
```
#### **2. Model Retraining Pipeline**
```
Training Data → Model Retraining → Quality Validation →
Production Update → Notification & Monitoring
```
**Key Stages:**
1. **Validation**: Experiment run ID and training parameters validation
2. **Data Acquisition**: Download training data from MinIO storage
3. **Training**: Execute ML model training with validated parameters
4. **Persistence**: Save trained model and artifacts to MLFlow
5. **Cleanup**: Remove temporary files and update experiment status
### Security Architecture
@@ -222,166 +216,7 @@ Production Update → Notification & Monitoring
## Workflows
### 1. Predictions Batch Workflow (`predictions_batch.py`)
The **PredictionsBatch** workflow is the main entry point for batch prediction pipelines. It orchestrates the complete prediction process and implements a robust data loading and processing pattern.
#### Purpose
- **Batch Prediction Orchestration**: Coordinates data loading and prediction processing
- **Data Preparation**: Loads data using custom SQL queries with configurable schemas
- **Workflow Delegation**: Delegates actual prediction processing to the PredictionProcess workflow
- **Configuration Management**: Handles model configuration, filters, and retention policies
#### Execution Flow
1. **Data Loading**: Executes custom SQL query to load data from PostgreSQL
2. **Input Preparation**: Prepares prediction input with metadata and configuration
3. **Workflow Delegation**: Spawns PredictionProcess child workflow for actual processing
4. **Error Handling**: Implements comprehensive error handling with retry policies
#### Key Features
- **Custom Query Support**: Flexible SQL-based data loading
- **Schema Configuration**: Configurable data schema definitions
- **Automatic Retry**: Implements Temporal retry policies for fault tolerance
- **Timeout Management**: 60-second timeout for all activities
- **Comprehensive Error Handling**: Detailed error reporting and notification integration
#### Input Parameters
```json
{
"schedule_name": "hourly_predictions",
"model_name": "temperature_prediction_model",
"model_id": "temp_pred_001",
"query": "SELECT * FROM sensor_data WHERE timestamp > NOW() - INTERVAL '1 hour'",
"schema": {
"timestamp": "datetime",
"temperature": "float",
"humidity": "float"
},
"table_name": "predictions",
"input_filters": {
"EMPTY_DATA": {"POLICY": "STOP"}
},
"mlflow_transform_filters": {
"API_ERROR": {"POLICY": "STOP"}
},
"mlflow_predict_filters": {
"API_ERROR": {"POLICY": "STOP"}
},
"model_retention": 60,
"path_priority": ["STOP", "CONTINUE", "REPEAT"]
}
```
#### Architecture Diagram
```mermaid
flowchart LR
A[1. load_custom_query] --> B[2. prediction_process 🔃]
A -.-> Database[(Database)]
```
### 2. Prediction Process Workflow (`prediction_process.py`)
The **PredictionProcess** workflow implements the core prediction pipeline for ML model inference. It handles data quality validation, MLFlow model interactions, and prediction processing.
#### Purpose
- **Data Quality Validation**: Applies configurable filters for data integrity
- **MLFlow Integration**: Manages model transformation and prediction requests
- **Response Validation**: Filters MLFlow API responses for quality assurance
- **Prediction Export**: Delegates prediction formatting and export operations
#### Execution Flow
1. **Timestamp Retrieval**: Gets the last processed timestamp for incremental processing
2. **Input Data Gate**: Applies configured filters for data quality validation
3. **Path Decision**: Determines processing path based on filter results
4. **MLFlow Transform**: Requests data transformation using MLFlow models
5. **Response Validation**: Filters transform responses for quality assurance
6. **MLFlow Prediction**: Executes prediction using transformed data
7. **Content Validation**: Filters prediction responses for final quality check
8. **Export Delegation**: Delegates to FormatAndExportPrediction workflow
#### Key Features
- **Configurable Quality Gates**: Multiple filter types with policy-based configuration
- **Flexible Path Handling**: Configurable decision paths (STOP, CONTINUE, REPEAT)
- **MLFlow Integration**: Comprehensive model management and inference
- **Incremental Processing**: Timestamp-based data processing optimization
- **Comprehensive Monitoring**: Detailed metrics and error reporting
#### Input Parameters
```json
{
"metadata": {
"schedule_name": "hourly_predictions",
"model_name": "temperature_prediction_model",
"model_id": "temp_pred_001",
"workflow_name": "predictions_batch"
},
"data": {...},
"schema": {...},
"table_name": "predictions",
"model_id": "temp_pred_001",
"model_name": "temperature_prediction_model",
"input_filters": {
"EMPTY_DATA": {"POLICY": "STOP"},
"SPECIFIC_VARIABLES_NULL_VALUES": {
"POLICY": "STOP",
"config": {"variables": ["temperature", "humidity"]}
}
},
"mlflow_transform_filters": {
"API_ERROR": {"POLICY": "STOP"}
},
"mlflow_predict_filters": {
"API_ERROR": {"POLICY": "STOP"},
"NAN_VALUES": {"POLICY": "STOP"}
},
"model_retention": 60,
"path_priority": ["STOP", "CONTINUE", "REPEAT"]
}
```
#### Architecture Diagram
```mermaid
flowchart LR
A[1. get_last_timestamp] --> B[2. input_gate] --> C[3. request_transform] --> D[4. mlflow_response_gate] --> E[5. mlflow_content_gate] --> F[6. request_predict] --> G[7. mlflow_response_gate] --> H[8. format_and_export_prediction🔃]
C -.-> MLFlow[MLFlow]
F -.-> MLFlow[MLFlow]
G -.-> Filters[MLFlow Filters]
```
### 3. Format and Export Prediction Workflow (`format_and_export_prediction.py`)
The **FormatAndExportPrediction** workflow handles prediction data formatting and export operations to multiple destinations.
#### Purpose
- **Data Formatting**: Formats prediction data for database storage
- **PostgreSQL Export**: Persists predictions to database with metrics
- **Metrics Recording**: Tracks export operations and performance metrics
#### Execution Flow
1. **Path Decision**: Determines formatting path based on configuration
2. **Data Formatting**: Formats prediction data for specific output requirements
3. **PostgreSQL Export**: Writes formatted predictions to database
4. **Metrics Recording**: Records export performance and success metrics
#### Key Features
- **Flexible Formatting**: Configurable output formats for different destinations
- **Database Export**: PostgreSQL integration for data persistence
- **Performance Monitoring**: Comprehensive metrics for export operations
- **Error Handling**: Robust error handling with notification integration
#### Architecture Diagram
```mermaid
flowchart LR
A[1. format_prediction/format_default_prediction] --> B[2. export_data_to_postgres] --> C[3. write_metrics]
A -.-> Format[Data Formatting]
B -.-> PostgreSQL[(PostgreSQL)]
C -.-> Prometheus[Prometheus]
```
### 4. Train Model Workflow (`train_model.py`)
### Train Model Workflow (`train_model.py`)
The **TrainModel** workflow orchestrates the complete ML model training pipeline from parameter validation through model saving and cleanup.
@@ -473,34 +308,6 @@ The workflow validates 10 business rules beyond type checking:
5. **target_variable**: Must be in variable_columns
6. **bucket_name, file_name, experiment_name**: Cannot be empty or whitespace
### 5. Minimal Retrain Workflow (`minimal_retrain.py`)
The **MinimalRetrain** workflow handles automated model retraining and production model updates.
#### Purpose
- **Model Retraining**: Automates ML model retraining processes
- **Production Updates**: Manages production model version updates
- **Data Export**: Exports training data for model development
- **Quality Assurance**: Ensures model quality before production deployment
#### Execution Flow
1. **Data Loading**: Loads training data using custom queries
2. **Model Retraining**: Executes model retraining process
3. **Quality Validation**: Validates retrained model performance
4. **Production Update**: Updates production model if quality criteria met
5. **Data Export**: Exports training data for analysis
#### Architecture Diagram
```mermaid
flowchart LR
A[1. load_custom_query] --> B[2. retrain_model] --> C[3. update_production_model] --> D[4. export_data_to_postgres]
A -.-> Database[(Database)]
B -.-> MLFlow[MLFlow]
C -.-> MLFlow[MLFlow]
D -.-> PostgreSQL[(PostgreSQL)]
```
## Installation & Setup
### Prerequisites
@@ -986,8 +793,8 @@ pip install pytest pytest-cov pytest-asyncio
pytest --cov=model_manager --cov-report=html
# Run specific test modules
pytest tests/activities/test_gates.py
pytest tests/workflows/test_predictions_batch.py
pytest tests/activities/test_training.py
pytest tests/workflows/test_train_model.py
```
## Monitoring and Metrics
@@ -1067,94 +874,6 @@ These timeouts control how long each activity in the training workflow can run b
**Note**: These timeouts can be adjusted based on your infrastructure performance and file sizes. If you're processing files larger than 200MB or have slower network/compute resources, increase these values accordingly.
### Workflow Configuration
MongoDB pipeline configuration:
#### Predictions Batch Workflow configuration sample
This is the configuration for the Predictions Batch Workflow, to be inserted into the MongoDB pipeline collection.
```json
{
"schedule_name": "laborious-orchestrated-pipeline",
"model_id": "1",
"workflow_type": "predictions_batch",
"frequency": "30s",
"max_retry_policy": 1,
"query": "select * from sientia_data.laborious_data where model_id = 1 and \"timestamp\" > NOW() - INTERVAL '5 minutes' order by \"timestamp\" desc limit 30;",
"write_tags": [
{
"server_id": "server1",
"type": "prediction",
"addr": "ns=2;i=5",
"data_type": "double"
},
{
"server_id": "server1",
"type": "confidence",
"addr": "ns=2;i=6",
"data_type": "double"
}
],
"input_filters": {
"EMPTY_DATA": {"POLICY": "STOP"},
"SPECIFIC_VARIABLES_NULL_VALUES": {
"POLICY": "CONTINUE",
"config": {"variables": ["Counter"]}
}
},
"mlflow_transform_filters": {
"API_ERROR": {"POLICY": "REPEAT"},
"NAN_VALUES": {"POLICY": "STOP"}
},
"mlflow_predict_filters": {
"API_ERROR": {"POLICY": "CONTINUE"}
},
"path_priority": ["STOP", "CONTINUE", "REPEAT"],
"active": true,
"updated_at": {
"$date": "2025-09-16T10:00:00.000Z"
},
"datetime_columns": ["timestamp", "created_at"],
"predictions_storage_policy": "lts:1"
}
```
This is the configuration created by the Orchestrator in Temporal.
```json
{
"datetime_columns":["timestamp","created_at"],
"frequency":"15m",
"input_filters":{"EMPTY_DATA":{"config":{},"policy":"STOP"}},
"max_retry_policy":1,
"mlflow_predict_filters":{"API_ERROR":{"config":{},"policy":"CONTINUE"}},
"mlflow_transform_filters":{
"API_ERROR":{"config":{},"policy":"CONTINUE"},
"EMPTY_DATA":{"config":{},"policy":"STOP"}
},
"model_config":{
"is_compressed":true,
"predict_flavor":"pyfunc",
"retention_minutes":60,
"retention_target":"artifact",
"transform_function_keyword":"transform"
},
"model_id":"352",
"model_name":"courier",
"path_priority":["STOP","CONTINUE","REPEAT"],
"predictions_storage_policy":"lts:1",
"query":"select * from sientia_data.laborious_data where model_id = 352 order by \"timestamp\" desc limit 300;",
"retention_time":3600,
"schedule_name":"laborious-courier",
"schema":"sientia_data",
"table_name":"predictions",
"updated_at":"2025-09-12 19:35:01.600000+0000",
"workflow_type":"predictions_batch"
}
```
## Development
### Code Quality & Testing
@@ -1229,30 +948,27 @@ model_manager/
├── activities/ # Temporal activity implementations
│ ├── __init__.py
│ ├── activities.py # Main activities orchestrator (combines all activities)
│ ├── gates.py # Data quality gates and filtering logic
│ ├── experiment_tracking.py # Experiment status tracking and database operations
│ ├── training.py # ML model training operations
│ ├── minio.py # MinIO object storage operations
│ └── mlflow.py # MLFlow model operations (predict/transform)
│ └── mlflow.py # MLFlow model saving and artifact management
├── workflows/ # Temporal workflow definitions
│ ├── __init__.py
│ ├── predictions_batch.py # Main batch prediction workflow entry point
│ ├── minimal_retrain.py # Model retraining workflow
│ └── sub_workflows/ # Sub-workflow implementations
│ ├── __init__.py
│ ├── prediction_process.py # Core prediction pipeline
│ └── format_and_export_prediction.py # Data export workflow
│ └── train_model.py # Complete ML model training workflow
├── worker/ # Worker implementation
│ ├── __init__.py
│ └── worker.py # Main worker orchestrator (Temporal client setup)
├── utils/ # Utility functions and helpers
│ ├── __init__.py
│ ├── connectors_config.py # Environment-based configuration builders
│ ├── filters/ # Data quality validation filters
│ ├── models/ # Data models and schemas
│ │ ├── __init__.py
│ │ ├── conditional_filters.py # Input data validation filters
│ │ └── mlflow_filters.py # MLFlow response validation filters
│ │ ├── train_model_params.py # Training parameters model
│ │ ├── train_model_result.py # Training result model
│ │ └── experiment_status.py # Experiment status enum
│ └── repository/ # Data access layer
│ ├── __init__.py
│ └── model_repository.py # MLFlow model operations and retraining
│ └── training_repository.py # Training business logic
├── metrics.py # Prometheus metrics definitions
└── __init__.py
```