sientia-dataops-model-manager/README.md

# Sientia DataOps Model Manager

An enterprise-grade ML model training orchestration platform built on Temporal. Provides robust, scalable workflows for training machine learning models with comprehensive validation, experiment tracking, and automated resource management. Integrates seamlessly with MLFlow for model persistence and PostgreSQL for experiment tracking.

## 📑 Table of Contents

- [Features](#features)
  - [Core Functionality](#core-functionality)
  - [Advanced Capabilities](#advanced-capabilities)
  - [Development & Quality Assurance](#development--quality-assurance)
- [Architecture](#architecture)
  - [Architecture Principles](#architecture-principles)
  - [Key Components](#key-components)
  - [Data Flow Architecture](#data-flow-architecture)
  - [Security Architecture](#security-architecture)
- [Workflows](#workflows)
  - [Train Model Workflow](#train-model-workflow-train_modelpy)
- [Installation & Setup](#installation--setup)
  - [Prerequisites](#prerequisites)
  - [Environment Setup](#environment-setup)
  - [Temporal Namespace Setup](#temporal-namespace-setup)
  - [Local Development Setup](#local-development-setup)
    - [Port Forward Setup Script](#port-forward-setup-script)
- [How to Run](#how-to-run)
  - [Running the Model Manager Application](#running-the-model-manager-application)
  - [Running Tests and Coverage](#running-tests-and-coverage)
  - [Manual Test Execution](#manual-test-execution)
  - [Manual Application Execution](#manual-application-execution)
- [Code Quality & Validation](#code-quality--validation)
  - [Overview](#overview)
  - [Validation Tools](#validation-tools)
  - [Tools Installation](#tools-installation)
  - [Complete Validation](#complete-validation)
  - [Automatic Fixes](#automatic-fixes)
  - [Configuration](#configuration)
  - [CI/CD Integration](#cicd-integration)
  - [Best Practices](#best-practices)
- [Testing](#testing)
  - [Test Structure](#test-structure)
  - [Test Execution](#test-execution)
- [Monitoring and Metrics](#monitoring-and-metrics)
  - [Application Health Metrics](#application-health-metrics)
  - [Prediction Operation Metrics](#prediction-operation-metrics)
  - [Data Quality Metrics](#data-quality-metrics)
- [Configuration](#configuration-1)
  - [Environment Variables](#environment-variables)
  - [Workflow Configuration](#workflow-configuration)
- [Development](#development)
  - [Code Quality & Testing](#code-quality--testing)
  - [Project Structure](#project-structure)
  - [Adding New Features](#adding-new-features)
  - [Test Coverage Guidelines](#test-coverage-guidelines)
- [Troubleshooting](#troubleshooting)
  - [Common Issues](#common-issues)
  - [Debug Mode](#debug-mode)
- [Performance Tuning](#performance-tuning)
  - [Key Parameters](#key-parameters)
  - [Scaling Considerations](#scaling-considerations)
- [Contributing](#contributing)
  - [Code Quality Standards](#code-quality-standards)
- [License](#license)
- [Support](#support)

## Features

### Core Functionality
- **ML Model Training Pipeline**: Complete training workflow from validation to deployment using MLFlow
- **Temporal Workflow Orchestration**: Robust workflow management with granular retry policies and fault tolerance
- **Parameter Validation**: Defense-in-depth validation with business rules and type checking
- **Experiment Tracking**: Comprehensive status tracking in PostgreSQL database
- **Resource Management**: Automatic cleanup of temporary files and storage
- **Comprehensive Monitoring**: Prometheus metrics and detailed logging for operational visibility

### Advanced Capabilities
- **Granular Retry Policies**: Different strategies for network, training, MLFlow, database, and filesystem operations
- **Configurable Timeouts**: Environment variable-based timeouts supporting large training files (up to 200MB)
- **Notification System**: Integrated alerting and notification management via MongoDB
- **Scalable Architecture**: Kubernetes-ready deployment with horizontal scaling support
- **MLFlow Integration**: Seamless model and artifact persistence to MLFlow tracking server

### Development & Quality Assurance
- **Code Quality Tools**: Ruff (linting/formatting), mypy (type checking), Bandit (security analysis)
- **Automated Validation**: Pre-commit validation script (`validate.sh`) and CI/CD integration
- **Comprehensive Testing**: pytest with async support and **99%+ code coverage** 🎯
- **Type Safety**: Static type checking with mypy for improved code reliability
- **Coverage Visualization**: Integration with Coverage Gutters for real-time coverage feedback
- **Automated Versioning**: Semantic versioning based on branch patterns (release/*, feature/*, fix/*, rc/*)

## Architecture

The Model Manager system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments.


### Architecture Principles

#### 1. **Separation of Concerns**
- **Worker Layer**: Manages Temporal workers, task queues, and application lifecycle
- **Workflow Layer**: Orchestrates business logic and process coordination
- **Activity Layer**: Implements specific operations and external system interactions
- **Data Layer**: Handles data persistence, caching, and external service connections

#### 2. **Fault Tolerance & Resilience**
- **Automatic Retry Policies**: Configurable retry strategies for transient failures
- **Circuit Breaker Pattern**: Prevents cascading failures in external service calls
- **Graceful Degradation**: System continues operating with reduced functionality
- **Comprehensive Error Handling**: Detailed error reporting and notification integration

#### 3. **Scalability & Performance**
- **Horizontal Scaling**: Multiple worker instances for load distribution
- **Task Queue Isolation**: Separate queues for different workflow types
- **Connection Pooling**: Optimized database and external service connections
- **Asynchronous Processing**: Non-blocking operations for improved throughput

#### 4. **Observability & Monitoring**
- **Prometheus Metrics**: Comprehensive system and business metrics
- **Structured Logging**: Consistent log format with correlation IDs
- **Health Checks**: Endpoint health monitoring and alerting
- **Performance Tracing**: Request flow tracking and bottleneck identification

### Key Components

#### **Worker (`model_manager/worker/worker.py`)**
- **Purpose**: Main application orchestrator managing Temporal workers and task queues
- **Responsibilities**:
  - Temporal client initialization and connection management
  - Worker lifecycle management and graceful shutdown
  - Task queue configuration and load balancing
  - Prometheus metrics server initialization
  - Notification handler setup and configuration
- **Key Features**:
  - Automatic scaling with `PollerBehaviorAutoscaling`
  - Health check endpoints for Kubernetes liveness/readiness probes
  - Graceful shutdown with cleanup procedures
  - Multi-instance deployment support
  - Dedicated task queue: `train_model-queue` for ML model training workflows

#### **Workflows (`model_manager/workflows/`)**
- **TrainModel**: Complete ML model training pipeline from validation to deployment
- **Key Features**:
  - Temporal workflow definitions with granular retry policies
  - Parameter validation with business rules
  - Comprehensive error handling and status tracking
  - Configurable timeouts for different operation types
  - Automatic resource cleanup and management

#### **Activities (`model_manager/activities/`)**
- **Activities**: Main activity orchestrator combining all functionality through multiple inheritance
- **ExperimentTracking**: ML experiment lifecycle tracking and database operations (extends Postgres)
  - Unified `update_experiment_run()` method for all experiment status updates
  - Support for three update types: STATUS, STATUS_WITH_ERROR, MODEL_SAVED
  - Automatic error message truncation (1024 chars)
  - Connection pooling and retry logic via Postgres base class
- **Training**: ML model training operations (standalone activity, composition pattern)
  - Unified `train_model()` method for complete training pipeline
  - Receives pre-downloaded files (BytesIO) to avoid memory leaks
  - Returns success/failure status with TrainModelResult or error message
  - No exception raising on failure - allows workflow to handle errors gracefully
  - Integration with TrainingRepository for business logic separation
- **MLFlow**: Model saving and artifact management operations
- **MinIO**: Object storage operations for training data management
- **Key Features**:
  - Multiple inheritance pattern for unified activity interface
  - Parameter validation with business rules
  - MLFlow integration for model persistence
  - Comprehensive error handling and notification integration
  - Experiment tracking with automatic status management

#### **Data Services (`model_manager/utils/`)**
- **Connectors Config**: Environment variable-based configuration management
- **Repository**: Data access layer for training operations
  - `training_repository.py`: Training business logic and operations
- **Models**: Data models and schemas
  - `train_model_params.py`: Training parameters model
  - `train_model_result.py`: Training result model
  - `experiment_status.py`: Experiment status enum
- **Key Features**:
  - Environment variable-based configuration with sensible defaults
  - Connection pool management and optimization
  - Security credential management
  - Configuration validation and error handling
  - Type-safe data models with validation

### Data Flow Architecture

#### **Model Training Pipeline**
```
Training Request → Parameter Validation → MinIO Data Download → 
Model Training → MLFlow Model Save → Resource Cleanup → Status Update
```

**Key Stages:**
1. **Validation**: Experiment run ID and training parameters validation
2. **Data Acquisition**: Download training data from MinIO storage
3. **Training**: Execute ML model training with validated parameters
4. **Persistence**: Save trained model and artifacts to MLFlow
5. **Cleanup**: Remove temporary files and update experiment status

### Security Architecture

#### **Authentication & Authorization**

- **MLFlow API Authentication**: Username/password with secure transmission
- **Database Connection Security**: Encrypted connections with credential management
- **Kubernetes Secrets Integration**: Secure credential storage and access

#### **Network Security**
- **TLS/SSL Encryption**: Secure communication channels
- **Network Isolation**: Kubernetes network policies and service mesh
- **Firewall Rules**: Controlled access to external services
- **VPN Integration**: Secure remote access and management

#### **Data Security**
- **Data Encryption**: At-rest and in-transit encryption
- **Access Control**: Role-based access control (RBAC)
- **Audit Logging**: Comprehensive access and operation logging
- **Data Retention**: Configurable data lifecycle management

## Workflows

### Train Model Workflow (`train_model.py`)

The **TrainModel** workflow orchestrates the complete ML model training pipeline from parameter validation through model saving and cleanup.

#### Purpose
- **Model Training**: Complete ML model training pipeline
- **Parameter Validation**: Defense-in-depth validation with business rules
- **Resource Management**: Automatic cleanup of temporary resources
- **Status Tracking**: Comprehensive experiment tracking in database
- **Error Handling**: Robust error handling with detailed context logging

#### Execution Flow
1. **Validate Experiment Run ID**: Critical validation before any DB updates
2. **Validate Training Parameters**: Type checking + business rules validation
3. **Download Training Data**: Fetch file from MinIO storage
4. **Train Model**: Execute ML model training with validated parameters
5. **Save to MLFlow**: Save trained model and artifacts to MLFlow
6. **Cleanup Resources**: Delete temporary files and MinIO data

#### Key Features
- **Granular Retry Policies**: Different strategies for network, training, MLFlow, database, and filesystem operations
- **Configurable Timeouts**: Environment variable-based timeouts supporting files up to 200MB
- **Idempotent Cleanup**: Safe replay with Temporal workflow replay mechanism
- **Structured Logging**: Rich context in error messages for debugging
- **Business Validation**: 10 business rules including range checks, consistency validation, and data integrity

#### Input Parameters
```json
{
  "experiment_run_id": 123,
  "target_variable": "price",
  "variable_columns": ["feature1", "feature2", "price"],
  "train_size": 80,
  "shuffle": true,
  "use_scaler": true,
  "include_ar": false,
  "bucket_name": "ml-data",
  "file_name": "training_data.csv",
  "line_separator": "\n",
  "decimal_separator": ".",
  "lag_train": 5,
  "lag_val": 3,
  "rem_static_win": false,
  "low_lim": {"feature1": 0.0, "feature2": 0.0, "price": 0.0},
  "upp_lim": {"feature1": 100.0, "feature2": 100.0, "price": 1000.0},
  "window": 10,
  "experiment_name": "production_model_v1",
  "removed_intervals": []
}
```

#### Architecture Diagram
```mermaid
flowchart TD
    A[1. validate_experiment_run_id] --> B[2. validate_train_params]
    B --> C[3. fetch_file_from_minio]
    C --> D[4. train_model]
    D --> E[5. save_model]
    E --> F[6. cleanup_run_directory]
    F --> G[7. delete_file_from_minio]
    
    B -.-> DB[(PostgreSQL)]
    C -.-> MinIO[MinIO Storage]
    D -.-> Training[ML Training]
    E -.-> MLFlow[MLFlow]
    F -.-> FS[Filesystem]
    G -.-> MinIO
```

#### Retry Strategies

The workflow implements 5 different retry policies optimized for each operation type:

| Operation Type | Initial Interval | Max Interval | Backoff | Max Attempts | Use Case |
|---------------|------------------|--------------|---------|--------------|----------|
| **Network** | 1s | 10s | 2.0x | 5 | MinIO operations (transient network errors) |
| **No Retry** | - | - | - | 1 | Training/Validation (permanent data errors) |
| **MLFlow** | 5s | 30s | 2.0x | 3 | MLFlow operations (API timeouts) |
| **Database** | 2s | 20s | 2.0x | 5 | PostgreSQL updates (lock contention) |
| **Filesystem** | 2s | 10s | 1.5x | 3 | Cleanup operations (busy resources) |

#### Business Validation Rules

The workflow validates 10 business rules beyond type checking:

1. **train_size**: Must be between 1-99%
2. **variable_columns**: Cannot be empty
3. **lag_train, lag_val, window**: Must be positive integers
4. **low_lim/upp_lim**: Must have same keys and low < upp for each variable
5. **target_variable**: Must be in variable_columns
6. **bucket_name, file_name, experiment_name**: Cannot be empty or whitespace

## Installation & Setup

### Prerequisites

- Python 3.11+
- Temporal server/cluster
- PostgreSQL database
- MLFlow server
- MinIO object storage (for MLFlow artifacts)
- MongoDB server (for notifications)

**Note**: External dependencies must be available either through:
- Kubernetes cluster deployment
- Docker Compose setup
- Cloud-managed services
- Local installations

#### MinIO Setup

MinIO is required for MLFlow artifact storage. For detailed installation and configuration instructions, refer to:

📚 **[Install MinIO via Helm Chart on K8s](https://aignosi-wiki.atlassian.net/wiki/spaces/IT1/pages/225214465/Install+Minio+via+Helm+Chart+on+K8s)**

This guide covers:
- Helm chart installation on Kubernetes
- Storage configuration and persistence
- Access credentials setup
- Integration with MLFlow

### Environment Setup

1. **Clone the repository**:
   ```bash
   git clone <repository-url>
   cd sientia-dataops-model-manager
   ```

2. **Create virtual environment**:
   ```bash
   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   ```

3. **Install dependencies**:
   ```bash
   pip install -r requirements.txt
   ```

4. **Configure environment variables** (see [Configuration](#-configuration) section)

5. **Run validation script**:
   ```bash
   ./validate.sh
   ```

### Temporal Namespace Setup

The Model Manager requires a dedicated Temporal namespace to isolate workflows and maintain proper execution history. The namespace must be created **before** starting the application.

#### Why Create a Namespace?

- **Isolation**: Separates Model Manager workflows from other applications
- **Retention Control**: Configures workflow history retention (default: 7 days)
- **Multi-tenancy**: Enables multiple environments (dev, staging, prod) on same cluster
- **Security**: Allows namespace-level access control and permissions

#### When to Create?

- ✅ **Before first deployment** in any environment
- ✅ **Once per environment** (dev, staging, production)
- ✅ **After Temporal cluster setup** or upgrade

#### How to Create the Namespace

**Option 1: Using Temporal Admin Tools Pod (Recommended for Kubernetes)**

```bash
# 1. List Temporal pods
kubectl get pods -n temporal

# 2. Connect to admin tools pod
kubectl exec -it -n temporal <temporal-admin-tools-pod-name> -- bash

# 3. Create namespace
tctl --namespace model-manager namespace register \
    --retention 7 \
    --description "Model Manager - ML Model Orchestration Namespace"

# 4. Verify creation
tctl --namespace model-manager namespace describe

# 5. Exit pod
exit
```

**Option 2: Using Port Forward (Local Development)**

```bash
# 1. Port forward Temporal frontend
kubectl port-forward -n temporal svc/temporal-frontend 7233:7233

# 2. In another terminal, create namespace
tctl --address localhost:7233 \
     --namespace model-manager \
     namespace register \
     --retention 7 \
     --description "Model Manager - ML Model Orchestration Namespace"

# 3. Verify
tctl --address localhost:7233 --namespace model-manager namespace describe
```

**Option 3: Direct kubectl exec (One-liner)**

```bash
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
    tctl --namespace model-manager namespace register \
    --retention 7 \
    --description "Model Manager - ML Model Orchestration Namespace"
```

#### Namespace Configuration

| Parameter | Value | Description |
|-----------|-------|-------------|
| **Name** | `model-manager` | Namespace identifier (configurable via `TEMPORAL_NAMESPACE` env var) |
| **Retention** | `7 days` | Workflow history retention period |
| **Description** | `Model Manager - ML Model Orchestration Namespace` | Human-readable description |

#### Verification

To verify the namespace was created successfully:

```bash
# List all namespaces
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- tctl namespace list

# Describe specific namespace
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
    tctl --namespace model-manager namespace describe
```

#### Troubleshooting

**Error: "namespace already exists"**
- ✅ This is fine! The namespace is already configured
- No action needed, proceed with application deployment

**Error: "connection refused"**
- ❌ Temporal server is not accessible
- Verify Temporal cluster is running: `kubectl get pods -n temporal`
- Check network connectivity and port forwarding

**Error: "permission denied"**
- ❌ Insufficient permissions to create namespace
- Contact cluster administrator for namespace creation
- Or request elevated permissions for your service account

### Local Development Setup

1. **Clone the repository**
   ```bash
   git clone <repository-url>
   cd sientia-dataops-model-manager
   ```

2. **Create virtual environment**
   ```bash
   conda create -p ./venv python=3.11
   conda activate ./venv
   ```

3. **Install dependencies**

  1. **Install github cli**
    ```bash
    sudo apt update
    sudo apt install gh -y
    ```

  2. **Authenticate with github**
    ```bash
    gh auth login
    ```

  3. **Run the install_dependencies.sh script**
    ```bash
    chmod +x install_dependencies.sh
    ./install_dependencies.sh
    ```

  4. **Install Python dependencies**
    ```bash
    python -m pip install --upgrade pip
    # Install production dependencies
    pip install -r requirements.txt
    # Install development and testing tools
    pip install -r requirements-dev.txt
    ```

4. **Create environment configuration file**
   ```bash
   cp .env.example .env
   # Edit .env with your connection details
   ```

5. **Configure external dependencies**
   
   The Model Manager requires connections to several external services. For local development, you can use the provided port-forward script to establish connections to services running in your Kubernetes cluster.

#### Port Forward Setup Script

The `setup_port_forwards.sh` script automates the creation of port forwards to all required services:

**Features:**
- 🔄 **Automatic Cleanup**: Kills existing port-forward jobs for the same services
- ✅ **Port Validation**: Checks if ports are available before creating forwards
- 🛡️ **Safe Execution**: Stops if any port is already in use by another process
- 📊 **Clear Output**: Color-coded status messages and service information

**Usage:**

```bash
# Make script executable (first time only)
chmod +x setup_port_forwards.sh

# Run the script
./setup_port_forwards.sh
```

**Services and Ports:**

| Local Port | Service | Description | Namespace |
|------------|---------|-------------|-----------|
| `5432` | `paradedb-rw` | PostgreSQL Database | `paradedb` |
| `45249` | `sientia-tracker-mlflow-tracking` | MLflow Tracking Server | `sientia-tracker` |
| `37463` | `temporal-frontend` | Temporal gRPC API | `temporal` |
| `8080` | `temporal-web` | Temporal Web UI | `temporal` |
| `42297` | `my-release-mongodb` | MongoDB Database | `mongodb` |
| `36577` | `minio` | MinIO Object Storage | `minio` |

**Managing Port Forwards:**

```bash
# View active port forwards
jobs -l

# Stop all port forwards
jobs -p | xargs kill

# Stop and restart (using the script)
./setup_port_forwards.sh
```

**Troubleshooting:**

If you encounter port conflicts:
1. The script will show which ports are in use
2. Stop the conflicting process or use the script to kill existing port-forwards
3. Run the script again

**Manual Port Forwarding:**

If you prefer manual control or need different ports:

```bash
# PostgreSQL
kubectl -n paradedb port-forward svc/paradedb-rw 5432:5432 &

# MLflow
kubectl -n sientia-tracker port-forward svc/sientia-tracker-mlflow-tracking 45249:80 &

# Temporal
kubectl -n temporal port-forward svc/temporal-frontend 37463:7233 &

# Temporal UI
kubectl -n temporal port-forward svc/temporal-web 8080:8080 &

# MongoDB
kubectl -n mongodb port-forward svc/my-release-mongodb 42297:27017 &

# MinIO
kubectl -n minio port-forward svc/minio 36577:9000 &
```

## How to Run

### Running the Model Manager Application

Use the provided script to run the application locally:

```bash
# Make script executable (first time only)
chmod +x run_local.sh

# Run the application
./run_local.sh
```

The script will:
- Activate the virtual environment
- Load environment variables from `.env`
- Start the model-manager worker application

### Running Tests and Coverage

Use the provided script to run tests with coverage:

```bash
# Make script executable (first time only)
chmod +x run_coverage.sh

# Run tests with coverage
./run_coverage.sh
```

The script will:
- Activate the virtual environment
- Run pytest with coverage reporting
- Generate HTML coverage report
- Open the coverage report in your browser

### Manual Test Execution

You can also run tests manually:

```bash
# Activate virtual environment
source ./venv/bin/activate

# Run all tests
pytest

# Run with coverage
pytest --cov=model_manager --cov-report=html

# Run specific test categories
pytest tests/activities/
pytest tests/workflow/
```

### Manual Application Execution

For manual execution without scripts:

```bash
# Activate virtual environment
source ./venv/bin/activate

# Load environment variables (if using .env file)
if [ -f .env ]; then
    export $(cat .env | grep -v '^#' | xargs)
fi

# Start the model-manager worker
python -m model_manager.worker.worker
```

## Code Quality & Validation

### Overview

Since Python is not a compiled language, we use a robust set of tools to validate code quality, security, and correctness before execution. These tools detect errors, style issues, security vulnerabilities, and ensure code consistency.

### Validation Tools

#### 1. **Ruff** - Linting and Formatting ⚡
Modern and extremely fast tool (written in Rust) that replaces multiple tools:
- **Linting**: Detects code errors, style issues (PEP 8), common bugs
- **Formatting**: Automatically formats code consistently
- **Speed**: 10-100x faster than Flake8/Black

#### 2. **mypy** - Type Checking 🏷️
Static type checker that analyzes type hints:
- Detects type errors before execution
- Improves code documentation
- Prevents bugs related to incorrect types

#### 3. **Bandit** - Security Analysis 🔒
Security vulnerability scanner:
- Detects insecure code patterns
- Identifies hardcoded passwords, SQL injection, etc.
- Ensures compliance with security practices

#### 4. **pytest** - Automated Testing 🧪
Testing framework with code coverage:
- Executes unit and integration tests
- Measures code coverage
- Supports asynchronous tests

### Tools Installation

```bash
# Install development dependencies
pip install -r requirements-dev.txt
```

### Complete Validation

#### Option 1: Automated Script (Recommended)
```bash
# Run all validations at once
./validate.sh
```

The `validate.sh` script automatically executes:
1. ✅ Format checking (Ruff)
2. ✅ Code linting (Ruff)
3. ✅ Type checking (mypy)
4. ✅ Security analysis (Bandit)
5. ✅ Unit tests with coverage (pytest)

#### Option 2: Individual Commands
```bash
# 1. Check formatting
ruff format --check model_manager/ tests/

# 2. Check linting
ruff check model_manager/ tests/

# 3. Check types
mypy model_manager/

# 4. Security analysis
bandit -r model_manager/ -ll

# 5. Run tests
pytest tests/ --cov=model_manager --cov-report=term-missing
```

### Automatic Fixes

Some tools can automatically fix issues:

```bash
# Format code automatically
ruff format model_manager/ tests/

# Fix linting issues automatically
ruff check --fix model_manager/ tests/
```

### Configuration

All tools are configured in the `pyproject.toml` file:
- **Ruff**: Linting rules, formatting, complexity
- **mypy**: Type checking settings
- **pytest**: Test and coverage options
- **Bandit**: Security rules

### CI/CD Integration

The `.github/workflows/quality-gate.yml` workflow automatically runs all validations on each push/PR:
- ✅ Formatting and linting block merge if they fail
- ⚠️ Type checking and security generate warnings but don't block
- ✅ Tests must pass with minimum 80% coverage

### Best Practices

1. **Before Commit**: Run `./validate.sh` to ensure quality
2. **During Development**: Use `ruff check --watch` for real-time feedback
3. **Type Hints**: Add type hints to new functions for better validation
4. **Tests**: Maintain coverage above 80%
5. **Security**: Review and fix all Bandit warnings

## Testing

### Test Structure
```
tests/
├── activities/      # Activity implementation tests
├── workflows/       # Workflow orchestration tests
├── utils/          # Utility function tests
└── worker/         # Worker tests
```

### Test Execution
```bash
# Install test dependencies
pip install pytest pytest-cov pytest-asyncio

# Run tests with coverage
pytest --cov=model_manager --cov-report=html

# Run specific test modules
pytest tests/activities/test_training.py
pytest tests/workflows/test_train_model.py
```

## Monitoring and Metrics

The Model Manager system exposes comprehensive Prometheus metrics for operational visibility and performance monitoring:

### Application Health Metrics
- `app_up`: Application health status (1=healthy, 0=unhealthy)
  - Labels: `pod_id`

### Prediction Operation Metrics
- `model_manager_predictions_written_count`: Counter for successful prediction exports
  - Labels: `pod_id`, `model_name`, `pipeline_name`
- `model_manager_prediction_confidence_monitor`: Gauge for current prediction confidence levels
  - Labels: `pod_id`, `model_name`, `pipeline_name`
- `model_manager_prediction_response_time_monitor`: Histogram for prediction response times
  - Labels: `pod_id`, `model_name`, `pipeline_name`
  - Buckets: [0.01, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0]

### Training Metrics
- Training success/failure rates through notification system
- Model save performance metrics
- Experiment status tracking

## Configuration

### Environment Variables

| Variable | Description | Default | Required |
|----------|-------------|---------|----------|
| `TEMPORAL_HOST` | Temporal server address | `localhost:7233` | Yes |
| `TEMPORAL_NAMESPACE` | Temporal namespace | `model-manager` | No |
| `POSTGRES_HOST` | PostgreSQL hostname | `localhost` | Yes |
| `POSTGRES_PORT` | PostgreSQL port | `5432` | Yes |
| `POSTGRES_USER` | PostgreSQL username | `sientia` | Yes |
| `POSTGRES_PASSWORD` | PostgreSQL password | `sientia` | Yes |
| `POSTGRES_DBNAME` | PostgreSQL database | `sientia` | Yes |
| `POSTGRES_MIN_CONNECTIONS` | Minimum PostgreSQL connections | `5` | No |
| `POSTGRES_MAX_CONNECTIONS` | Maximum PostgreSQL connections | `20` | No |
| `MLFLOW_HOST` | MLFlow server hostname | `http://localhost` | Yes |
| `MLFLOW_PORT` | MLFlow server port | `5080` | Yes |
| `MLFLOW_USERNAME` | MLFlow username | `aignosi` | Yes |
| `MLFLOW_PASSWORD` | MLFlow password | `aignosi` | Yes |
| `MINIO_ENDPOINT_URL` | MinIO server endpoint | `http://minio.minio.svc.cluster.local:9000` | Yes |
| `MINIO_ACCESS_KEY` | MinIO access key | `minioadmin` | Yes |
| `MINIO_SECRET_KEY` | MinIO secret key | `minioadmin` | Yes |
| `MINIO_REGION` | MinIO region | `us-east-1` | No |
| `MINIO_USE_SSL` | Enable SSL for MinIO | `false` | No |
| `MINIO_MAX_RETRY_ATTEMPTS` | Maximum retry attempts | `3` | No |
| `MINIO_RETRY_MODE` | Retry mode (standard/adaptive) | `adaptive` | No |
| `MINIO_CONNECT_TIMEOUT` | Connection timeout (seconds) | `10` | No |
| `MINIO_READ_TIMEOUT` | Read timeout (seconds) | `60` | No |
| `MONGODB_URL` | MongoDB connection URI | `localhost:27018` | Yes |
| `MONGODB_USERNAME` | MongoDB username | `root` | Yes |
| `MONGODB_PASSWORD` | MongoDB password | `wKZDbMNU1c` | Yes |
| `MONGODB_DATABASE_NAME` | MongoDB database name | `sientia` | Yes |
| `MONGODB_TTL_INDEX_HOURS` | MongoDB TTL index hours | `1` | No |
| `LOG_LEVEL` | Application log level | `INFO` | No |
| `PROJECT_NAME` | Project name for metrics | `model-manager` | No |
| `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
| `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
| `POD_ID` | Kubernetes pod identifier | `None` | No |

#### Workflow Activity Timeouts

These timeouts control how long each activity in the training workflow can run before timing out. All values are in seconds and are designed to handle large files (up to 200MB).

| Variable | Description | Default | Calculation Basis |
|----------|-------------|---------|-------------------|
| `TIMEOUT_VALIDATE_PARAMS` | Parameter validation timeout | `30` | Fast operation, no I/O |
| `TIMEOUT_DOWNLOAD_FILE` | File download from MinIO timeout | `600` | 200MB @ 1MB/s with 3x buffer (10 min) |
| `TIMEOUT_TRAIN_MODEL` | Model training timeout | `1800` | Large dataset processing (30 min) |
| `TIMEOUT_SAVE_MODEL` | Save model to MLFlow timeout | `300` | Artifact upload and logging (5 min) |
| `TIMEOUT_CLEANUP_DIRECTORY` | Cleanup temporary directory timeout | `60` | Local filesystem operation (1 min) |
| `TIMEOUT_DELETE_FILE` | Delete file from MinIO timeout | `60` | MinIO delete operation (1 min) |
| `TIMEOUT_UPDATE_DATABASE` | Database update timeout | `30` | PostgreSQL update query (30 sec) |

**Note**: These timeouts can be adjusted based on your infrastructure performance and file sizes. If you're processing files larger than 200MB or have slower network/compute resources, increase these values accordingly.

## Development

### Code Quality & Testing

The project maintains **99%+ code coverage** with comprehensive unit and integration tests.

#### Running Tests

```bash
# Run all tests with coverage
pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=xml --cov-report=html

# Run specific test file
pytest tests/activities/test_training.py -v

# Run with coverage visualization
pytest tests/ --cov=model_manager --cov-report=xml
# Then open htmlcov/index.html in browser
```

#### Validation Script

The `validate.sh` script runs all quality checks before commit:

```bash
./validate.sh
```

This script performs:
1. ✅ **Code Formatting** (Ruff)
2. ✅ **Code Linting** (Ruff)
3. ✅ **Type Checking** (mypy)
4. ✅ **Security Analysis** (Bandit)
5. ✅ **Unit Tests** (pytest with 80%+ coverage requirement)

#### Coverage Visualization

For real-time coverage feedback in VS Code/Windsurf:

1. **Install Coverage Gutters extension**
2. **Configure `.vscode/settings.json`**:
   ```json
   {
     "coverage-gutters.coverageBaseDir": "${workspaceFolder}",
     "coverage-gutters.coverageFileNames": ["coverage.xml"],
     "coverage-gutters.showLineCoverage": true,
     "coverage-gutters.showRulerCoverage": true
   }
   ```
3. **Run tests to generate coverage**:
   ```bash
   pytest tests/ --cov=model_manager --cov-report=xml
   ```
4. **Activate Coverage Gutters**: Press `Ctrl+Shift+7` (or `Cmd+Shift+7` on Mac)

#### Automated Versioning

The project uses semantic versioning based on branch patterns:

| Branch Pattern | Version Change | Example |
|---------------|----------------|---------|
| `release/*` | Major version bump | `2.0.0` |
| `feature/*` | Minor version bump | `1.2.0` |
| `fix/*` | Patch version bump | `1.1.3` |
| `rc/*` | Release candidate | `1.1.2-rc2` |

Version is calculated automatically in the CI/CD pipeline and passed to SonarQube.

### Project Structure
```
model_manager/
├── activities/              # Temporal activity implementations
│   ├── __init__.py
│   ├── activities.py       # Main activities orchestrator (combines all activities)
│   ├── experiment_tracking.py # Experiment status tracking and database operations
│   ├── training.py         # ML model training operations
│   ├── minio.py            # MinIO object storage operations
│   └── mlflow.py           # MLFlow model saving and artifact management
├── workflows/               # Temporal workflow definitions
│   ├── __init__.py
│   └── train_model.py      # Complete ML model training workflow
├── worker/                  # Worker implementation
│   ├── __init__.py
│   └── worker.py           # Main worker orchestrator (Temporal client setup)
├── utils/                   # Utility functions and helpers
│   ├── __init__.py
│   ├── connectors_config.py # Environment-based configuration builders
│   ├── models/             # Data models and schemas
│   │   ├── __init__.py
│   │   ├── train_model_params.py # Training parameters model
│   │   ├── train_model_result.py # Training result model
│   │   └── experiment_status.py  # Experiment status enum
│   └── repository/         # Data access layer
│       ├── __init__.py
│       └── training_repository.py # Training business logic
├── metrics.py               # Prometheus metrics definitions
└── __init__.py
```

### Adding New Features

1. **Follow Temporal patterns** for new workflows and activities
2. **Add comprehensive docstrings** for all public methods
3. **Include Prometheus metrics** for monitoring
4. **Add unit tests** for new functionality (maintain 80%+ coverage)
5. **Run validation script** (`./validate.sh`) before committing
6. **Update this README** with new features and configuration

### Test Coverage Guidelines

- **Minimum coverage**: 80% (enforced by CI/CD)
- **Current coverage**: 99%+ 🎯
- **Test all branches**: Use Coverage Gutters to identify uncovered lines
- **Mock external dependencies**: Use `unittest.mock` for external services
- **Async testing**: Use `pytest-asyncio` for async activities and workflows
- **Test structure**:
  ```
  tests/
  ├── activities/      # Activity tests
  ├── workflows/       # Workflow tests
  ├── utils/          # Utility tests
  └── worker/         # Worker tests
  ```

## Troubleshooting

### Common Issues

1. **Temporal Connection Failures**
   - Verify Temporal server is running and accessible
   - Check namespace configuration and permissions
   - Review server logs for connection issues

2. **MLFlow Connection Issues**
   - Verify MLFlow server is running and accessible
   - Check authentication credentials and permissions
   - Ensure model names and versions exist

3. **Database Connection Issues**
   - Verify PostgreSQL service is running
   - Check connection credentials and network access
   - Ensure proper connection pool configuration

4. **Workflow Execution Failures**
   - Review activity error logs and notifications
   - Check training parameter validation errors
   - Verify input data format and required fields

### Debug Mode

Enable debug logging by setting the log level:
```bash
export LOG_LEVEL=DEBUG
```

## Performance Tuning

### Key Parameters

- **Worker Concurrency**: Adjust `max_concurrent_workflow_tasks` and `max_concurrent_activities`
- **Connection Pools**: Optimize database connection pool sizes
- **Model Retention**: Configure MLFlow model retention based on requirements
- **Batch Sizes**: Adjust data processing batch sizes for optimal throughput

### Scaling Considerations

- **Horizontal Scaling**: Deploy multiple worker instances
- **Task Queue Distribution**: Use multiple task queues for different workflow types
- **Database Performance**: Optimize indexes and connection pooling
- **MLFlow Performance**: Configure appropriate model serving resources

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes with comprehensive testing
4. Update documentation and docstrings
5. Submit a pull request

### Code Quality Standards

- Follow PEP 8 style guidelines
- Include comprehensive docstrings for all public methods
- Maintain test coverage above 80%
- Use type hints where appropriate
- Follow Temporal.io best practices

## License

This project is licensed under the terms specified in the LICENSE file.

## Support

For support and questions:
- Check the troubleshooting section above
- Review the metrics and logs for error patterns
- Open an issue in the project repository
- Contact the development team

---

**Note**: The Model Manager system is designed for production use in industrial ML environments. Ensure proper security configuration and network isolation for production deployments.