gitea_admin/sientia-dataops-model-manager
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
85a8ba1e68c5edd88be81d4f1440edda1886456d
sientia-dataops-model-manager/README.md

38 KiB
Raw Blame History

Sientia DataOps Model Manager

An enterprise-grade ML model training orchestration platform built on Temporal. Provides robust, scalable workflows for training machine learning models with comprehensive validation, experiment tracking, and automated resource management. Integrates seamlessly with MLFlow for model persistence and PostgreSQL for experiment tracking.

📑 Table of Contents

Features

Core Functionality

  • ML Model Training Pipeline: Complete training workflow from validation to deployment using MLFlow
  • Temporal Workflow Orchestration: Robust workflow management with granular retry policies and fault tolerance
  • Parameter Validation: Defense-in-depth validation with business rules and type checking
  • Experiment Tracking: Comprehensive status tracking in PostgreSQL database
  • Resource Management: Automatic cleanup of temporary files and storage
  • Comprehensive Monitoring: Prometheus metrics and detailed logging for operational visibility

Advanced Capabilities

  • Granular Retry Policies: Different strategies for network, training, MLFlow, database, and filesystem operations
  • Configurable Timeouts: Environment variable-based timeouts supporting large training files (up to 200MB)
  • Notification System: Integrated alerting and notification management via MongoDB
  • Scalable Architecture: Kubernetes-ready deployment with horizontal scaling support
  • MLFlow Integration: Seamless model and artifact persistence to MLFlow tracking server

Development & Quality Assurance

  • Code Quality Tools: Ruff (linting/formatting), mypy (type checking), Bandit (security analysis)
  • Automated Validation: Pre-commit validation script (validate.sh) and CI/CD integration
  • Comprehensive Testing: pytest with async support and 99%+ code coverage 🎯
  • Type Safety: Static type checking with mypy for improved code reliability
  • Coverage Visualization: Integration with Coverage Gutters for real-time coverage feedback
  • Automated Versioning: Semantic versioning based on branch patterns (release/, feature/, fix/, rc/)

Architecture

The Model Manager system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments.

Architecture Principles

1. Separation of Concerns

  • Worker Layer: Manages Temporal workers, task queues, and application lifecycle
  • Workflow Layer: Orchestrates business logic and process coordination
  • Activity Layer: Implements specific operations and external system interactions
  • Data Layer: Handles data persistence, caching, and external service connections

2. Fault Tolerance & Resilience

  • Automatic Retry Policies: Configurable retry strategies for transient failures
  • Circuit Breaker Pattern: Prevents cascading failures in external service calls
  • Graceful Degradation: System continues operating with reduced functionality
  • Comprehensive Error Handling: Detailed error reporting and notification integration

3. Scalability & Performance

  • Horizontal Scaling: Multiple worker instances for load distribution
  • Task Queue Isolation: Separate queues for different workflow types
  • Connection Pooling: Optimized database and external service connections
  • Asynchronous Processing: Non-blocking operations for improved throughput

4. Observability & Monitoring

  • Prometheus Metrics: Comprehensive system and business metrics
  • Structured Logging: Consistent log format with correlation IDs
  • Health Checks: Endpoint health monitoring and alerting
  • Performance Tracing: Request flow tracking and bottleneck identification

Key Components

Worker (model_manager/worker/worker.py)

  • Purpose: Main application orchestrator managing Temporal workers and task queues
  • Responsibilities:
    • Temporal client initialization and connection management
    • Worker lifecycle management and graceful shutdown
    • Task queue configuration and load balancing
    • Prometheus metrics server initialization
    • Notification handler setup and configuration
  • Key Features:
    • Automatic scaling with PollerBehaviorAutoscaling
    • Health check endpoints for Kubernetes liveness/readiness probes
    • Graceful shutdown with cleanup procedures
    • Multi-instance deployment support
    • Dedicated task queue: train_model-queue for ML model training workflows

Workflows (model_manager/workflows/)

  • TrainModel: Complete ML model training pipeline from validation to deployment
  • Key Features:
    • Temporal workflow definitions with granular retry policies
    • Parameter validation with business rules
    • Comprehensive error handling and status tracking
    • Configurable timeouts for different operation types
    • Automatic resource cleanup and management

Activities (model_manager/activities/)

  • Activities: Main activity orchestrator combining all functionality through multiple inheritance
  • ExperimentTracking: ML experiment lifecycle tracking and database operations (extends Postgres)
    • Unified update_experiment_run() method for all experiment status updates
    • Support for three update types: STATUS, STATUS_WITH_ERROR, MODEL_SAVED
    • Automatic error message truncation (1024 chars)
    • Connection pooling and retry logic via Postgres base class
  • Training: ML model training operations (standalone activity, composition pattern)
    • Unified train_model() method for complete training pipeline
    • Receives pre-downloaded files (BytesIO) to avoid memory leaks
    • Returns success/failure status with TrainModelResult or error message
    • No exception raising on failure - allows workflow to handle errors gracefully
    • Integration with TrainingRepository for business logic separation
  • MLFlow: Model saving and artifact management operations
  • MinIO: Object storage operations for training data management
  • Key Features:
    • Multiple inheritance pattern for unified activity interface
    • Parameter validation with business rules
    • MLFlow integration for model persistence
    • Comprehensive error handling and notification integration
    • Experiment tracking with automatic status management

Data Services (model_manager/utils/)

  • Connectors Config: Environment variable-based configuration management
  • Repository: Data access layer for training operations
    • training_repository.py: Training business logic and operations
  • Models: Data models and schemas
    • train_model_params.py: Training parameters model
    • train_model_result.py: Training result model
    • experiment_status.py: Experiment status enum
  • Key Features:
    • Environment variable-based configuration with sensible defaults
    • Connection pool management and optimization
    • Security credential management
    • Configuration validation and error handling
    • Type-safe data models with validation

Data Flow Architecture

Model Training Pipeline

Training Request → Parameter Validation → MinIO Data Download → 
Model Training → MLFlow Model Save → Resource Cleanup → Status Update

Key Stages:

  1. Validation: Experiment run ID and training parameters validation
  2. Data Acquisition: Download training data from MinIO storage
  3. Training: Execute ML model training with validated parameters
  4. Persistence: Save trained model and artifacts to MLFlow
  5. Cleanup: Remove temporary files and update experiment status

Security Architecture

Authentication & Authorization

  • MLFlow API Authentication: Username/password with secure transmission
  • Database Connection Security: Encrypted connections with credential management
  • Kubernetes Secrets Integration: Secure credential storage and access

Network Security

  • TLS/SSL Encryption: Secure communication channels
  • Network Isolation: Kubernetes network policies and service mesh
  • Firewall Rules: Controlled access to external services
  • VPN Integration: Secure remote access and management

Data Security

  • Data Encryption: At-rest and in-transit encryption
  • Access Control: Role-based access control (RBAC)
  • Audit Logging: Comprehensive access and operation logging
  • Data Retention: Configurable data lifecycle management

Workflows

Train Model Workflow (train_model.py)

The TrainModel workflow orchestrates the complete ML model training pipeline from parameter validation through model saving and cleanup.

Purpose

  • Model Training: Complete ML model training pipeline
  • Parameter Validation: Defense-in-depth validation with business rules
  • Resource Management: Automatic cleanup of temporary resources
  • Status Tracking: Comprehensive experiment tracking in database
  • Error Handling: Robust error handling with detailed context logging

Execution Flow

  1. Validate Experiment Run ID: Critical validation before any DB updates
  2. Validate Training Parameters: Type checking + business rules validation
  3. Download Training Data: Fetch file from MinIO storage
  4. Train Model: Execute ML model training with validated parameters
  5. Save to MLFlow: Save trained model and artifacts to MLFlow
  6. Cleanup Resources: Delete temporary files and MinIO data

Key Features

  • Granular Retry Policies: Different strategies for network, training, MLFlow, database, and filesystem operations
  • Configurable Timeouts: Environment variable-based timeouts supporting files up to 200MB
  • Idempotent Cleanup: Safe replay with Temporal workflow replay mechanism
  • Structured Logging: Rich context in error messages for debugging
  • Business Validation: 10 business rules including range checks, consistency validation, and data integrity

Input Parameters

{
  "experiment_run_id": 123,
  "target_variable": "price",
  "variable_columns": ["feature1", "feature2", "price"],
  "train_size": 80,
  "shuffle": true,
  "use_scaler": true,
  "include_ar": false,
  "bucket_name": "ml-data",
  "file_name": "training_data.csv",
  "line_separator": "\n",
  "decimal_separator": ".",
  "lag_train": 5,
  "lag_val": 3,
  "rem_static_win": false,
  "low_lim": {"feature1": 0.0, "feature2": 0.0, "price": 0.0},
  "upp_lim": {"feature1": 100.0, "feature2": 100.0, "price": 1000.0},
  "window": 10,
  "experiment_name": "production_model_v1",
  "removed_intervals": []
}

Architecture Diagram

flowchart TD
    A[1. validate_experiment_run_id] --> B[2. validate_train_params]
    B --> C[3. fetch_file_from_minio]
    C --> D[4. train_model]
    D --> E[5. save_model]
    E --> F[6. cleanup_run_directory]
    F --> G[7. delete_file_from_minio]
    
    B -.-> DB[(PostgreSQL)]
    C -.-> MinIO[MinIO Storage]
    D -.-> Training[ML Training]
    E -.-> MLFlow[MLFlow]
    F -.-> FS[Filesystem]
    G -.-> MinIO

Retry Strategies

The workflow implements 5 different retry policies optimized for each operation type:

Operation Type Initial Interval Max Interval Backoff Max Attempts Use Case
Network 1s 10s 2.0x 5 MinIO operations (transient network errors)
No Retry - - - 1 Training/Validation (permanent data errors)
MLFlow 5s 30s 2.0x 3 MLFlow operations (API timeouts)
Database 2s 20s 2.0x 5 PostgreSQL updates (lock contention)
Filesystem 2s 10s 1.5x 3 Cleanup operations (busy resources)

Business Validation Rules

The workflow validates 10 business rules beyond type checking:

  1. train_size: Must be between 1-99%
  2. variable_columns: Cannot be empty
  3. lag_train, lag_val, window: Must be positive integers
  4. low_lim/upp_lim: Must have same keys and low < upp for each variable
  5. target_variable: Must be in variable_columns
  6. bucket_name, file_name, experiment_name: Cannot be empty or whitespace

Installation & Setup

Prerequisites

  • Python 3.11+
  • Temporal server/cluster
  • PostgreSQL database
  • MLFlow server
  • MinIO object storage (for MLFlow artifacts)
  • MongoDB server (for notifications)

Note: External dependencies must be available either through:

  • Kubernetes cluster deployment
  • Docker Compose setup
  • Cloud-managed services
  • Local installations

MinIO Setup

MinIO is required for MLFlow artifact storage. For detailed installation and configuration instructions, refer to:

📚 Install MinIO via Helm Chart on K8s

This guide covers:

  • Helm chart installation on Kubernetes
  • Storage configuration and persistence
  • Access credentials setup
  • Integration with MLFlow

Environment Setup

  1. Clone the repository:

    git clone <repository-url>
cd sientia-dataops-model-manager

  2. Create virtual environment:

    python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

  3. Install dependencies:

    pip install -r requirements.txt

  4. Configure environment variables (see Configuration section)

  5. Run validation script:

    ./validate.sh

Temporal Namespace Setup

The Model Manager requires a dedicated Temporal namespace to isolate workflows and maintain proper execution history. The namespace must be created before starting the application.

Why Create a Namespace?

  • Isolation: Separates Model Manager workflows from other applications
  • Retention Control: Configures workflow history retention (default: 7 days)
  • Multi-tenancy: Enables multiple environments (dev, staging, prod) on same cluster
  • Security: Allows namespace-level access control and permissions

When to Create?

  • ✅ Before first deployment in any environment
  • ✅ Once per environment (dev, staging, production)
  • ✅ After Temporal cluster setup or upgrade

How to Create the Namespace

Option 1: Using Temporal Admin Tools Pod (Recommended for Kubernetes)

# 1. List Temporal pods
kubectl get pods -n temporal

# 2. Connect to admin tools pod
kubectl exec -it -n temporal <temporal-admin-tools-pod-name> -- bash

# 3. Create namespace
tctl --namespace model-manager namespace register \
    --retention 7 \
    --description "Model Manager - ML Model Orchestration Namespace"

# 4. Verify creation
tctl --namespace model-manager namespace describe

# 5. Exit pod
exit

Option 2: Using Port Forward (Local Development)

# 1. Port forward Temporal frontend
kubectl port-forward -n temporal svc/temporal-frontend 7233:7233

# 2. In another terminal, create namespace
tctl --address localhost:7233 \
     --namespace model-manager \
     namespace register \
     --retention 7 \
     --description "Model Manager - ML Model Orchestration Namespace"

# 3. Verify
tctl --address localhost:7233 --namespace model-manager namespace describe

Option 3: Direct kubectl exec (One-liner)

kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
    tctl --namespace model-manager namespace register \
    --retention 7 \
    --description "Model Manager - ML Model Orchestration Namespace"

Namespace Configuration

Parameter Value Description
Name model-manager Namespace identifier (configurable via TEMPORAL_NAMESPACE env var)
Retention 7 days Workflow history retention period
Description Model Manager - ML Model Orchestration Namespace Human-readable description

Verification

To verify the namespace was created successfully:

# List all namespaces
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- tctl namespace list

# Describe specific namespace
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
    tctl --namespace model-manager namespace describe

Troubleshooting

Error: "namespace already exists"

  • ✅ This is fine! The namespace is already configured
  • No action needed, proceed with application deployment

Error: "connection refused"

  • ❌ Temporal server is not accessible
  • Verify Temporal cluster is running: kubectl get pods -n temporal
  • Check network connectivity and port forwarding

Error: "permission denied"

  • ❌ Insufficient permissions to create namespace
  • Contact cluster administrator for namespace creation
  • Or request elevated permissions for your service account

Local Development Setup

  1. Clone the repository

    git clone <repository-url>
cd sientia-dataops-model-manager

  2. Create virtual environment

    conda create -p ./venv python=3.11
conda activate ./venv

  3. Install dependencies

  4. Install github cli bash sudo apt update sudo apt install gh -y

  5. Authenticate with github bash gh auth login

  6. Run the install_dependencies.sh script bash chmod +x install_dependencies.sh ./install_dependencies.sh

  7. Install Python dependencies bash python -m pip install --upgrade pip # Install production dependencies pip install -r requirements.txt # Install development and testing tools pip install -r requirements-dev.txt

  8. Create environment configuration file

    cp .env.example .env
# Edit .env with your connection details

  9. Configure external dependencies

    The Model Manager requires connections to several external services. For local development, you can use the provided port-forward script to establish connections to services running in your Kubernetes cluster.

Port Forward Setup Script

The setup_port_forwards.sh script automates the creation of port forwards to all required services:

Features:

  • 🔄 Automatic Cleanup: Kills existing port-forward jobs for the same services
  • ✅ Port Validation: Checks if ports are available before creating forwards
  • 🛡️ Safe Execution: Stops if any port is already in use by another process
  • 📊 Clear Output: Color-coded status messages and service information

Usage:

# Make script executable (first time only)
chmod +x setup_port_forwards.sh

# Run the script
./setup_port_forwards.sh

Services and Ports:

Local Port Service Description Namespace
5432 paradedb-rw PostgreSQL Database paradedb
45249 sientia-tracker-mlflow-tracking MLflow Tracking Server sientia-tracker
37463 temporal-frontend Temporal gRPC API temporal
8080 temporal-web Temporal Web UI temporal
42297 my-release-mongodb MongoDB Database mongodb
36577 minio MinIO Object Storage minio

Managing Port Forwards:

# View active port forwards
jobs -l

# Stop all port forwards
jobs -p | xargs kill

# Stop and restart (using the script)
./setup_port_forwards.sh

Troubleshooting:

If you encounter port conflicts:

  1. The script will show which ports are in use
  2. Stop the conflicting process or use the script to kill existing port-forwards
  3. Run the script again

Manual Port Forwarding:

If you prefer manual control or need different ports:

# PostgreSQL
kubectl -n paradedb port-forward svc/paradedb-rw 5432:5432 &

# MLflow
kubectl -n sientia-tracker port-forward svc/sientia-tracker-mlflow-tracking 45249:80 &

# Temporal
kubectl -n temporal port-forward svc/temporal-frontend 37463:7233 &

# Temporal UI
kubectl -n temporal port-forward svc/temporal-web 8080:8080 &

# MongoDB
kubectl -n mongodb port-forward svc/my-release-mongodb 42297:27017 &

# MinIO
kubectl -n minio port-forward svc/minio 36577:9000 &

How to Run

Running the Model Manager Application

Use the provided script to run the application locally:

# Make script executable (first time only)
chmod +x run_local.sh

# Run the application
./run_local.sh

The script will:

  • Activate the virtual environment
  • Load environment variables from .env
  • Start the model-manager worker application

Running Tests and Coverage

Use the provided script to run tests with coverage:

# Make script executable (first time only)
chmod +x run_coverage.sh

# Run tests with coverage
./run_coverage.sh

The script will:

  • Activate the virtual environment
  • Run pytest with coverage reporting
  • Generate HTML coverage report
  • Open the coverage report in your browser

Manual Test Execution

You can also run tests manually:

# Activate virtual environment
source ./venv/bin/activate

# Run all tests
pytest

# Run with coverage
pytest --cov=model_manager --cov-report=html

# Run specific test categories
pytest tests/activities/
pytest tests/workflow/

Manual Application Execution

For manual execution without scripts:

# Activate virtual environment
source ./venv/bin/activate

# Load environment variables (if using .env file)
if [ -f .env ]; then
    export $(cat .env | grep -v '^#' | xargs)
fi

# Start the model-manager worker
python -m model_manager.worker.worker

Code Quality & Validation

Overview

Since Python is not a compiled language, we use a robust set of tools to validate code quality, security, and correctness before execution. These tools detect errors, style issues, security vulnerabilities, and ensure code consistency.

Validation Tools

1. Ruff - Linting and Formatting ⚡

Modern and extremely fast tool (written in Rust) that replaces multiple tools:

  • Linting: Detects code errors, style issues (PEP 8), common bugs
  • Formatting: Automatically formats code consistently
  • Speed: 10-100x faster than Flake8/Black

2. mypy - Type Checking 🏷️

Static type checker that analyzes type hints:

  • Detects type errors before execution
  • Improves code documentation
  • Prevents bugs related to incorrect types

3. Bandit - Security Analysis 🔒

Security vulnerability scanner:

  • Detects insecure code patterns
  • Identifies hardcoded passwords, SQL injection, etc.
  • Ensures compliance with security practices

4. pytest - Automated Testing 🧪

Testing framework with code coverage:

  • Executes unit and integration tests
  • Measures code coverage
  • Supports asynchronous tests

Tools Installation

# Install development dependencies
pip install -r requirements-dev.txt

Complete Validation

Option 1: Automated Script (Recommended)

# Run all validations at once
./validate.sh

The validate.sh script automatically executes:

  1. ✅ Format checking (Ruff)
  2. ✅ Code linting (Ruff)
  3. ✅ Type checking (mypy)
  4. ✅ Security analysis (Bandit)
  5. ✅ Unit tests with coverage (pytest)

Option 2: Individual Commands

# 1. Check formatting
ruff format --check model_manager/ tests/

# 2. Check linting
ruff check model_manager/ tests/

# 3. Check types
mypy model_manager/

# 4. Security analysis
bandit -r model_manager/ -ll

# 5. Run tests
pytest tests/ --cov=model_manager --cov-report=term-missing

Automatic Fixes

Some tools can automatically fix issues:

# Format code automatically
ruff format model_manager/ tests/

# Fix linting issues automatically
ruff check --fix model_manager/ tests/

Configuration

All tools are configured in the pyproject.toml file:

  • Ruff: Linting rules, formatting, complexity
  • mypy: Type checking settings
  • pytest: Test and coverage options
  • Bandit: Security rules

CI/CD Integration

The .github/workflows/quality-gate.yml workflow automatically runs all validations on each push/PR:

  • ✅ Formatting and linting block merge if they fail
  • ⚠️ Type checking and security generate warnings but don't block
  • ✅ Tests must pass with minimum 80% coverage

Best Practices

  1. Before Commit: Run ./validate.sh to ensure quality
  2. During Development: Use ruff check --watch for real-time feedback
  3. Type Hints: Add type hints to new functions for better validation
  4. Tests: Maintain coverage above 80%
  5. Security: Review and fix all Bandit warnings

Testing

Test Structure

tests/
├── activities/      # Activity implementation tests
├── workflows/       # Workflow orchestration tests
├── utils/          # Utility function tests
└── worker/         # Worker tests

Test Execution

# Install test dependencies
pip install pytest pytest-cov pytest-asyncio

# Run tests with coverage
pytest --cov=model_manager --cov-report=html

# Run specific test modules
pytest tests/activities/test_training.py
pytest tests/workflows/test_train_model.py

Monitoring and Metrics

The Model Manager system exposes comprehensive Prometheus metrics for operational visibility and performance monitoring:

Application Health Metrics

  • app_up: Application health status (1=healthy, 0=unhealthy)
    • Labels: pod_id

Prediction Operation Metrics

  • model_manager_predictions_written_count: Counter for successful prediction exports
    • Labels: pod_id, model_name, pipeline_name
  • model_manager_prediction_confidence_monitor: Gauge for current prediction confidence levels
    • Labels: pod_id, model_name, pipeline_name
  • model_manager_prediction_response_time_monitor: Histogram for prediction response times
    • Labels: pod_id, model_name, pipeline_name
    • Buckets: [0.01, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0]

Training Metrics

  • Training success/failure rates through notification system
  • Model save performance metrics
  • Experiment status tracking

Configuration

Environment Variables

Variable Description Default Required
TEMPORAL_HOST Temporal server address localhost:7233 Yes
TEMPORAL_NAMESPACE Temporal namespace model-manager No
POSTGRES_HOST PostgreSQL hostname localhost Yes
POSTGRES_PORT PostgreSQL port 5432 Yes
POSTGRES_USER PostgreSQL username sientia Yes
POSTGRES_PASSWORD PostgreSQL password sientia Yes
POSTGRES_DBNAME PostgreSQL database sientia Yes
POSTGRES_MIN_CONNECTIONS Minimum PostgreSQL connections 5 No
POSTGRES_MAX_CONNECTIONS Maximum PostgreSQL connections 20 No
MLFLOW_HOST MLFlow server hostname http://localhost Yes
MLFLOW_PORT MLFlow server port 5080 Yes
MLFLOW_USERNAME MLFlow username aignosi Yes
MLFLOW_PASSWORD MLFlow password aignosi Yes
MINIO_ENDPOINT_URL MinIO server endpoint http://minio.minio.svc.cluster.local:9000 Yes
MINIO_ACCESS_KEY MinIO access key minioadmin Yes
MINIO_SECRET_KEY MinIO secret key minioadmin Yes
MINIO_REGION MinIO region us-east-1 No
MINIO_USE_SSL Enable SSL for MinIO false No
MINIO_MAX_RETRY_ATTEMPTS Maximum retry attempts 3 No
MINIO_RETRY_MODE Retry mode (standard/adaptive) adaptive No
MINIO_CONNECT_TIMEOUT Connection timeout (seconds) 10 No
MINIO_READ_TIMEOUT Read timeout (seconds) 60 No
MONGODB_URL MongoDB connection URI localhost:27018 Yes
MONGODB_USERNAME MongoDB username root Yes
MONGODB_PASSWORD MongoDB password wKZDbMNU1c Yes
MONGODB_DATABASE_NAME MongoDB database name sientia Yes
MONGODB_TTL_INDEX_HOURS MongoDB TTL index hours 1 No
LOG_LEVEL Application log level INFO No
PROJECT_NAME Project name for metrics model-manager No
HTTP_METRICS_PORT Prometheus metrics port 9090 No
HTTP_SDK_METRICS_PORT Temporal SDK metrics port 9091 No
POD_ID Kubernetes pod identifier None No

Workflow Activity Timeouts

These timeouts control how long each activity in the training workflow can run before timing out. All values are in seconds and are designed to handle large files (up to 200MB).

Variable Description Default Calculation Basis
TIMEOUT_VALIDATE_PARAMS Parameter validation timeout 30 Fast operation, no I/O
TIMEOUT_DOWNLOAD_FILE File download from MinIO timeout 600 200MB @ 1MB/s with 3x buffer (10 min)
TIMEOUT_TRAIN_MODEL Model training timeout 1800 Large dataset processing (30 min)
TIMEOUT_SAVE_MODEL Save model to MLFlow timeout 300 Artifact upload and logging (5 min)
TIMEOUT_CLEANUP_DIRECTORY Cleanup temporary directory timeout 60 Local filesystem operation (1 min)
TIMEOUT_DELETE_FILE Delete file from MinIO timeout 60 MinIO delete operation (1 min)
TIMEOUT_UPDATE_DATABASE Database update timeout 30 PostgreSQL update query (30 sec)

Note: These timeouts can be adjusted based on your infrastructure performance and file sizes. If you're processing files larger than 200MB or have slower network/compute resources, increase these values accordingly.

Development

Code Quality & Testing

The project maintains 99%+ code coverage with comprehensive unit and integration tests.

Running Tests

# Run all tests with coverage
pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=xml --cov-report=html

# Run specific test file
pytest tests/activities/test_training.py -v

# Run with coverage visualization
pytest tests/ --cov=model_manager --cov-report=xml
# Then open htmlcov/index.html in browser

Validation Script

The validate.sh script runs all quality checks before commit:

./validate.sh

This script performs:

  1. ✅ Code Formatting (Ruff)
  2. ✅ Code Linting (Ruff)
  3. ✅ Type Checking (mypy)
  4. ✅ Security Analysis (Bandit)
  5. ✅ Unit Tests (pytest with 80%+ coverage requirement)

Coverage Visualization

For real-time coverage feedback in VS Code/Windsurf:

  1. Install Coverage Gutters extension
  2. Configure .vscode/settings.json: 
    {
  "coverage-gutters.coverageBaseDir": "${workspaceFolder}",
  "coverage-gutters.coverageFileNames": ["coverage.xml"],
  "coverage-gutters.showLineCoverage": true,
  "coverage-gutters.showRulerCoverage": true
}
  3. Run tests to generate coverage: 
    pytest tests/ --cov=model_manager --cov-report=xml
  4. Activate Coverage Gutters: Press Ctrl+Shift+7 (or Cmd+Shift+7 on Mac)

Automated Versioning

The project uses semantic versioning based on branch patterns:

Branch Pattern Version Change Example
release/* Major version bump 2.0.0
feature/* Minor version bump 1.2.0
fix/* Patch version bump 1.1.3
rc/* Release candidate 1.1.2-rc2

Version is calculated automatically in the CI/CD pipeline and passed to SonarQube.

Project Structure

model_manager/
├── activities/              # Temporal activity implementations
│   ├── __init__.py
│   ├── activities.py       # Main activities orchestrator (combines all activities)
│   ├── experiment_tracking.py # Experiment status tracking and database operations
│   ├── training.py         # ML model training operations
│   ├── minio.py            # MinIO object storage operations
│   └── mlflow.py           # MLFlow model saving and artifact management
├── workflows/               # Temporal workflow definitions
│   ├── __init__.py
│   └── train_model.py      # Complete ML model training workflow
├── worker/                  # Worker implementation
│   ├── __init__.py
│   └── worker.py           # Main worker orchestrator (Temporal client setup)
├── utils/                   # Utility functions and helpers
│   ├── __init__.py
│   ├── connectors_config.py # Environment-based configuration builders
│   ├── models/             # Data models and schemas
│   │   ├── __init__.py
│   │   ├── train_model_params.py # Training parameters model
│   │   ├── train_model_result.py # Training result model
│   │   └── experiment_status.py  # Experiment status enum
│   └── repository/         # Data access layer
│       ├── __init__.py
│       └── training_repository.py # Training business logic
├── metrics.py               # Prometheus metrics definitions
└── __init__.py

Adding New Features

  1. Follow Temporal patterns for new workflows and activities
  2. Add comprehensive docstrings for all public methods
  3. Include Prometheus metrics for monitoring
  4. Add unit tests for new functionality (maintain 80%+ coverage)
  5. Run validation script (./validate.sh) before committing
  6. Update this README with new features and configuration

Test Coverage Guidelines

  • Minimum coverage: 80% (enforced by CI/CD)
  • Current coverage: 99%+ 🎯
  • Test all branches: Use Coverage Gutters to identify uncovered lines
  • Mock external dependencies: Use unittest.mock for external services
  • Async testing: Use pytest-asyncio for async activities and workflows
  • Test structure: 
    tests/
├── activities/      # Activity tests
├── workflows/       # Workflow tests
├── utils/          # Utility tests
└── worker/         # Worker tests

Troubleshooting

Common Issues

  1. Temporal Connection Failures

    • Verify Temporal server is running and accessible
    • Check namespace configuration and permissions
    • Review server logs for connection issues

  2. MLFlow Connection Issues

    • Verify MLFlow server is running and accessible
    • Check authentication credentials and permissions
    • Ensure model names and versions exist

  3. Database Connection Issues

    • Verify PostgreSQL service is running
    • Check connection credentials and network access
    • Ensure proper connection pool configuration

  4. Workflow Execution Failures

    • Review activity error logs and notifications
    • Check training parameter validation errors
    • Verify input data format and required fields

Debug Mode

Enable debug logging by setting the log level:

export LOG_LEVEL=DEBUG

Performance Tuning

Key Parameters

  • Worker Concurrency: Adjust max_concurrent_workflow_tasks and max_concurrent_activities
  • Connection Pools: Optimize database connection pool sizes
  • Model Retention: Configure MLFlow model retention based on requirements
  • Batch Sizes: Adjust data processing batch sizes for optimal throughput

Scaling Considerations

  • Horizontal Scaling: Deploy multiple worker instances
  • Task Queue Distribution: Use multiple task queues for different workflow types
  • Database Performance: Optimize indexes and connection pooling
  • MLFlow Performance: Configure appropriate model serving resources

Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes with comprehensive testing
  4. Update documentation and docstrings
  5. Submit a pull request

Code Quality Standards

  • Follow PEP 8 style guidelines
  • Include comprehensive docstrings for all public methods
  • Maintain test coverage above 80%
  • Use type hints where appropriate
  • Follow Temporal.io best practices

License

This project is licensed under the terms specified in the LICENSE file.

Support

For support and questions:

  • Check the troubleshooting section above
  • Review the metrics and logs for error patterns
  • Open an issue in the project repository
  • Contact the development team

Note: The Model Manager system is designed for production use in industrial ML environments. Ensure proper security configuration and network isolation for production deployments.

Reference in New Issue View Git Blame Copy Permalink