A comprehensive AI model management platform for the complete machine learning lifecycle. Handles model training, versioning, deployment, monitoring, and governance. Streamlines MLOps workflows with centralized model registry, automated pipelines, performance tracking, and enterprise-grade compliance features.

📑 Table of Contents

Features
Architecture
Workflows
Installation & Setup
How to Run
Code Quality & Validation
Testing
- Test Structure
- Test Execution
Monitoring and Metrics
Configuration
- Environment Variables
- Workflow Configuration
Development
Troubleshooting
- Common Issues
- Debug Mode
Performance Tuning
- Key Parameters
- Scaling Considerations
Contributing
- Code Quality Standards
License
Support

Features

Core Functionality

Batch Prediction Processing: High-throughput ML model inference using MLFlow models
Temporal Workflow Orchestration: Robust workflow management with automatic retry policies and fault tolerance
Data Quality Gates: Configurable filtering for data validation, MLFlow API responses, and custom validation rules
Multi-Model Support: Flexible ML model management with retention policies and versioning
Real-time Data Export: PostgreSQL persistence for data storage
Comprehensive Monitoring: Prometheus metrics and detailed logging for operational visibility

Advanced Capabilities

Incremental Data Processing: Timestamp-based data loading to avoid reprocessing
Configurable Data Retention: Model retention policies with automatic cleanup
Notification System: Integrated alerting and notification management via MongoDB
Scalable Architecture: Kubernetes-ready deployment with horizontal scaling support
Model Retraining: Automated model retraining workflows with production model updates

Development & Quality Assurance

Code Quality Tools: Ruff (linting/formatting), mypy (type checking), Bandit (security analysis)
Automated Validation: Pre-commit validation script (validate.sh) and CI/CD integration
Comprehensive Testing: pytest with async support and 99%+ code coverage 🎯
Type Safety: Static type checking with mypy for improved code reliability
Coverage Visualization: Integration with Coverage Gutters for real-time coverage feedback
Automated Versioning: Semantic versioning based on branch patterns (release/, feature/, fix/, rc/)

Architecture

The Model Manager system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments.

Architecture Principles

1. Separation of Concerns

Worker Layer: Manages Temporal workers, task queues, and application lifecycle
Workflow Layer: Orchestrates business logic and process coordination
Activity Layer: Implements specific operations and external system interactions
Data Layer: Handles data persistence, caching, and external service connections

2. Fault Tolerance & Resilience

Automatic Retry Policies: Configurable retry strategies for transient failures
Circuit Breaker Pattern: Prevents cascading failures in external service calls
Graceful Degradation: System continues operating with reduced functionality
Comprehensive Error Handling: Detailed error reporting and notification integration

3. Scalability & Performance

Horizontal Scaling: Multiple worker instances for load distribution
Task Queue Isolation: Separate queues for different workflow types
Connection Pooling: Optimized database and external service connections
Asynchronous Processing: Non-blocking operations for improved throughput

4. Observability & Monitoring

Prometheus Metrics: Comprehensive system and business metrics
Structured Logging: Consistent log format with correlation IDs
Health Checks: Endpoint health monitoring and alerting
Performance Tracing: Request flow tracking and bottleneck identification

Key Components

Worker (`model_manager/worker/worker.py`)

Purpose: Main application orchestrator managing Temporal workers and task queues
Responsibilities:
- Temporal client initialization and connection management
- Worker lifecycle management and graceful shutdown
- Task queue configuration and load balancing
- Prometheus metrics server initialization
- Notification handler setup and configuration
Key Features:
- Automatic scaling with PollerBehaviorAutoscaling
- Health check endpoints for Kubernetes liveness/readiness probes
- Graceful shutdown with cleanup procedures
- Multi-instance deployment support
- Two dedicated task queues: predictions_batch-queue and minimal_retrain-queue

Workflows (`model_manager/workflows/`)

PredictionsBatch: Main entry point for batch prediction pipelines
PredictionProcess: Core prediction pipeline with MLFlow integration
FormatAndExportPrediction: Data formatting and export operations
MinimalRetrain: Automated model retraining and deployment
Key Features:
- Temporal workflow definitions with retry policies
- Child workflow orchestration and delegation
- Comprehensive error handling and recovery
- Configurable timeout and retry strategies

Activities (`model_manager/activities/`)

Activities: Main activity orchestrator combining all functionality through multiple inheritance
ExperimentTracking: ML experiment lifecycle tracking and database operations (extends Postgres)
- Unified update_experiment_run() method for all experiment status updates
- Support for three update types: STATUS, STATUS_WITH_ERROR, MODEL_SAVED
- Automatic error message truncation (1024 chars)
- Connection pooling and retry logic via Postgres base class
Gates: Data quality validation and filtering mechanisms
MLFlow: Model transformation and prediction operations
MinIO: Object storage operations for file management
Key Features:
- Multiple inheritance pattern for unified activity interface
- Configurable filter policies and validation rules
- MLFlow model serving integration with configurable flavors
- Comprehensive error handling and notification integration
- Experiment tracking with automatic status management

Data Services (`model_manager/utils/`)

Connectors Config: Environment variable-based configuration management
Repository: Data access layer for MLFlow operations
- model_repository.py: MLFlow model operations and retraining
Filters: Data quality validation and MLFlow response filtering
- conditional_filters.py: Input data validation filters
- mlflow_filters.py: MLFlow API response validation filters
Key Features:
- Environment variable-based configuration with sensible defaults
- Connection pool management and optimization
- Security credential management
- Configuration validation and error handling
- Support for MLFlow model flavors

Data Flow Architecture

1. Batch Prediction Pipeline

Input Data (PostgreSQL) → Data Quality Gates → MLFlow Transform → 
MLFlow Prediction → Response Validation → Export (PostgreSQL)

2. Model Retraining Pipeline

Training Data → Model Retraining → Quality Validation → 
Production Update → Notification & Monitoring

Security Architecture

Authentication & Authorization

MLFlow API Authentication: Username/password with secure transmission
Database Connection Security: Encrypted connections with credential management
Kubernetes Secrets Integration: Secure credential storage and access

Network Security

TLS/SSL Encryption: Secure communication channels
Network Isolation: Kubernetes network policies and service mesh
Firewall Rules: Controlled access to external services
VPN Integration: Secure remote access and management

Data Security

Data Encryption: At-rest and in-transit encryption
Access Control: Role-based access control (RBAC)
Audit Logging: Comprehensive access and operation logging
Data Retention: Configurable data lifecycle management

Workflows

1. Predictions Batch Workflow (`predictions_batch.py`)

The PredictionsBatch workflow is the main entry point for batch prediction pipelines. It orchestrates the complete prediction process and implements a robust data loading and processing pattern.

Purpose

Batch Prediction Orchestration: Coordinates data loading and prediction processing
Data Preparation: Loads data using custom SQL queries with configurable schemas
Workflow Delegation: Delegates actual prediction processing to the PredictionProcess workflow
Configuration Management: Handles model configuration, filters, and retention policies

Execution Flow

Data Loading: Executes custom SQL query to load data from PostgreSQL
Input Preparation: Prepares prediction input with metadata and configuration
Workflow Delegation: Spawns PredictionProcess child workflow for actual processing
Error Handling: Implements comprehensive error handling with retry policies

Key Features

Custom Query Support: Flexible SQL-based data loading
Schema Configuration: Configurable data schema definitions
Automatic Retry: Implements Temporal retry policies for fault tolerance
Timeout Management: 60-second timeout for all activities
Comprehensive Error Handling: Detailed error reporting and notification integration

Input Parameters

{
  "schedule_name": "hourly_predictions",
  "model_name": "temperature_prediction_model",
  "model_id": "temp_pred_001",
  "query": "SELECT * FROM sensor_data WHERE timestamp > NOW() - INTERVAL '1 hour'",
  "schema": {
    "timestamp": "datetime",
    "temperature": "float",
    "humidity": "float"
  },
  "table_name": "predictions",
  "input_filters": {
    "EMPTY_DATA": {"POLICY": "STOP"}
  },
  "mlflow_transform_filters": {
    "API_ERROR": {"POLICY": "STOP"}
  },
  "mlflow_predict_filters": {
    "API_ERROR": {"POLICY": "STOP"}
  },
  "model_retention": 60,
  "path_priority": ["STOP", "CONTINUE", "REPEAT"]
}

Architecture Diagram

flowchart LR
    A[1. load_custom_query] --> B[2. prediction_process 🔃]
    
    A -.-> Database[(Database)]

2. Prediction Process Workflow (`prediction_process.py`)

The PredictionProcess workflow implements the core prediction pipeline for ML model inference. It handles data quality validation, MLFlow model interactions, and prediction processing.

Purpose

Data Quality Validation: Applies configurable filters for data integrity
MLFlow Integration: Manages model transformation and prediction requests
Response Validation: Filters MLFlow API responses for quality assurance
Prediction Export: Delegates prediction formatting and export operations

Execution Flow

Timestamp Retrieval: Gets the last processed timestamp for incremental processing
Input Data Gate: Applies configured filters for data quality validation
Path Decision: Determines processing path based on filter results
MLFlow Transform: Requests data transformation using MLFlow models
Response Validation: Filters transform responses for quality assurance
MLFlow Prediction: Executes prediction using transformed data
Content Validation: Filters prediction responses for final quality check
Export Delegation: Delegates to FormatAndExportPrediction workflow

Key Features

Configurable Quality Gates: Multiple filter types with policy-based configuration
Flexible Path Handling: Configurable decision paths (STOP, CONTINUE, REPEAT)
MLFlow Integration: Comprehensive model management and inference
Incremental Processing: Timestamp-based data processing optimization
Comprehensive Monitoring: Detailed metrics and error reporting

Input Parameters

{
  "metadata": {
    "schedule_name": "hourly_predictions",
    "model_name": "temperature_prediction_model",
    "model_id": "temp_pred_001",
    "workflow_name": "predictions_batch"
  },
  "data": {...},
  "schema": {...},
  "table_name": "predictions",
  "model_id": "temp_pred_001",
  "model_name": "temperature_prediction_model",
  "input_filters": {
    "EMPTY_DATA": {"POLICY": "STOP"},
    "SPECIFIC_VARIABLES_NULL_VALUES": {
      "POLICY": "STOP",
      "config": {"variables": ["temperature", "humidity"]}
    }
  },
  "mlflow_transform_filters": {
    "API_ERROR": {"POLICY": "STOP"}
  },
  "mlflow_predict_filters": {
    "API_ERROR": {"POLICY": "STOP"},
    "NAN_VALUES": {"POLICY": "STOP"}
  },
  "model_retention": 60,
  "path_priority": ["STOP", "CONTINUE", "REPEAT"]
}

Architecture Diagram

flowchart LR
    A[1. get_last_timestamp] --> B[2. input_gate] --> C[3. request_transform] --> D[4. mlflow_response_gate] --> E[5. mlflow_content_gate] --> F[6. request_predict] --> G[7. mlflow_response_gate] --> H[8. format_and_export_prediction🔃]
    
    C -.-> MLFlow[MLFlow]
    F -.-> MLFlow[MLFlow]
    G -.-> Filters[MLFlow Filters]

3. Format and Export Prediction Workflow (`format_and_export_prediction.py`)

The FormatAndExportPrediction workflow handles prediction data formatting and export operations to multiple destinations.

Purpose

Data Formatting: Formats prediction data for database storage
PostgreSQL Export: Persists predictions to database with metrics
Metrics Recording: Tracks export operations and performance metrics

Execution Flow

Path Decision: Determines formatting path based on configuration
Data Formatting: Formats prediction data for specific output requirements
PostgreSQL Export: Writes formatted predictions to database
Metrics Recording: Records export performance and success metrics

Key Features

Flexible Formatting: Configurable output formats for different destinations
Database Export: PostgreSQL integration for data persistence
Performance Monitoring: Comprehensive metrics for export operations
Error Handling: Robust error handling with notification integration

Architecture Diagram

flowchart LR
    A[1. format_prediction/format_default_prediction] --> B[2. export_data_to_postgres] --> C[3. write_metrics]
    
    A -.-> Format[Data Formatting]
    B -.-> PostgreSQL[(PostgreSQL)]
    C -.-> Prometheus[Prometheus]

4. Minimal Retrain Workflow (`minimal_retrain.py`)

The MinimalRetrain workflow handles automated model retraining and production model updates.

Purpose

Model Retraining: Automates ML model retraining processes
Production Updates: Manages production model version updates
Data Export: Exports training data for model development
Quality Assurance: Ensures model quality before production deployment

Execution Flow

Data Loading: Loads training data using custom queries
Model Retraining: Executes model retraining process
Quality Validation: Validates retrained model performance
Production Update: Updates production model if quality criteria met
Data Export: Exports training data for analysis

Architecture Diagram

flowchart LR
    A[1. load_custom_query] --> B[2. retrain_model] --> C[3. update_production_model] --> D[4. export_data_to_postgres]
    
    A -.-> Database[(Database)]
    B -.-> MLFlow[MLFlow]
    C -.-> MLFlow[MLFlow]
    D -.-> PostgreSQL[(PostgreSQL)]

Installation & Setup

Prerequisites

Python 3.11+
Temporal server/cluster
PostgreSQL database
MLFlow server
MinIO object storage (for MLFlow artifacts)
MongoDB server (for notifications)

Note: External dependencies must be available either through:

Kubernetes cluster deployment
Docker Compose setup
Cloud-managed services
Local installations

MinIO Setup

MinIO is required for MLFlow artifact storage. For detailed installation and configuration instructions, refer to:

📚 Install MinIO via Helm Chart on K8s

This guide covers:

Helm chart installation on Kubernetes
Storage configuration and persistence
Access credentials setup
Integration with MLFlow

Environment Setup

Clone the repository:

git clone <repository-url>
cd sientia-dataops-model-manager

Create virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Install dependencies:
```
pip install -r requirements.txt
```
Configure environment variables (see Configuration section)
Run validation script:
```
./validate.sh
```

Temporal Namespace Setup

The Model Manager requires a dedicated Temporal namespace to isolate workflows and maintain proper execution history. The namespace must be created before starting the application.

Why Create a Namespace?

Isolation: Separates Model Manager workflows from other applications
Retention Control: Configures workflow history retention (default: 7 days)
Multi-tenancy: Enables multiple environments (dev, staging, prod) on same cluster
Security: Allows namespace-level access control and permissions

When to Create?

✅ Before first deployment in any environment
✅ Once per environment (dev, staging, production)
✅ After Temporal cluster setup or upgrade

How to Create the Namespace

Option 1: Using Temporal Admin Tools Pod (Recommended for Kubernetes)

# 1. List Temporal pods
kubectl get pods -n temporal

# 2. Connect to admin tools pod
kubectl exec -it -n temporal <temporal-admin-tools-pod-name> -- bash

# 3. Create namespace
tctl --namespace model-manager namespace register \
    --retention 7 \
    --description "Model Manager - ML Model Orchestration Namespace"

# 4. Verify creation
tctl --namespace model-manager namespace describe

# 5. Exit pod
exit

Option 2: Using Port Forward (Local Development)

# 1. Port forward Temporal frontend
kubectl port-forward -n temporal svc/temporal-frontend 7233:7233

# 2. In another terminal, create namespace
tctl --address localhost:7233 \
     --namespace model-manager \
     namespace register \
     --retention 7 \
     --description "Model Manager - ML Model Orchestration Namespace"

# 3. Verify
tctl --address localhost:7233 --namespace model-manager namespace describe

Option 3: Direct kubectl exec (One-liner)

kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
    tctl --namespace model-manager namespace register \
    --retention 7 \
    --description "Model Manager - ML Model Orchestration Namespace"

Namespace Configuration

Parameter	Value	Description
Name	`model-manager`	Namespace identifier (configurable via `TEMPORAL_NAMESPACE` env var)
Retention	`7 days`	Workflow history retention period
Description	`Model Manager - ML Model Orchestration Namespace`	Human-readable description

Verification

To verify the namespace was created successfully:

# List all namespaces
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- tctl namespace list

# Describe specific namespace
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
    tctl --namespace model-manager namespace describe

Troubleshooting

Error: "namespace already exists"

✅ This is fine! The namespace is already configured
No action needed, proceed with application deployment

Error: "connection refused"

❌ Temporal server is not accessible
Verify Temporal cluster is running: kubectl get pods -n temporal
Check network connectivity and port forwarding

Error: "permission denied"

❌ Insufficient permissions to create namespace
Contact cluster administrator for namespace creation
Or request elevated permissions for your service account

Local Development Setup

Clone the repository

git clone <repository-url>
cd sientia-dataops-model-manager

Create virtual environment

conda create -p ./venv python=3.11
conda activate ./venv

Install dependencies
Install github cli bash sudo apt update sudo apt install gh -y
Authenticate with github bash gh auth login
Run the install_dependencies.sh script bash chmod +x install_dependencies.sh ./install_dependencies.sh
Install Python dependencies bash python -m pip install --upgrade pip # Install production dependencies pip install -r requirements.txt # Install development and testing tools pip install -r requirements-dev.txt

Create environment configuration file

cp .env.example .env
# Edit .env with your connection details

Configure external dependencies

You'll need to set up port forwarding or connections to external services. For example:

# Port forwarding from Kubernetes cluster
kubectl port-forward svc/postgresql 5432:5432
kubectl port-forward svc/mlflow 5000:5000
kubectl port-forward svc/mongodb 27017:27017

# Or connect to external services
# Ensure services are accessible on localhost with appropriate ports

How to Run

Running the Model Manager Application

Use the provided script to run the application locally:

# Make script executable (first time only)
chmod +x run_local.sh

# Run the application
./run_local.sh

The script will:

Activate the virtual environment
Load environment variables from .env
Start the model-manager worker application

Running Tests and Coverage

Use the provided script to run tests with coverage:

# Make script executable (first time only)
chmod +x run_coverage.sh

# Run tests with coverage
./run_coverage.sh

The script will:

Activate the virtual environment
Run pytest with coverage reporting
Generate HTML coverage report
Open the coverage report in your browser

Manual Test Execution

You can also run tests manually:

# Activate virtual environment
source ./venv/bin/activate

# Run all tests
pytest

# Run with coverage
pytest --cov=model_manager --cov-report=html

# Run specific test categories
pytest tests/activities/
pytest tests/workflow/

Manual Application Execution

For manual execution without scripts:

# Activate virtual environment
source ./venv/bin/activate

# Load environment variables (if using .env file)
if [ -f .env ]; then
    export $(cat .env | grep -v '^#' | xargs)
fi

# Start the model-manager worker
python -m model_manager.worker.worker

Code Quality & Validation

Overview

Since Python is not a compiled language, we use a robust set of tools to validate code quality, security, and correctness before execution. These tools detect errors, style issues, security vulnerabilities, and ensure code consistency.

Validation Tools

1. Ruff - Linting and Formatting ⚡

Modern and extremely fast tool (written in Rust) that replaces multiple tools:

Linting: Detects code errors, style issues (PEP 8), common bugs
Formatting: Automatically formats code consistently
Speed: 10-100x faster than Flake8/Black

2. mypy - Type Checking 🏷️

Static type checker that analyzes type hints:

Detects type errors before execution
Improves code documentation
Prevents bugs related to incorrect types

3. Bandit - Security Analysis 🔒

Security vulnerability scanner:

Detects insecure code patterns
Identifies hardcoded passwords, SQL injection, etc.
Ensures compliance with security practices

4. pytest - Automated Testing 🧪

Testing framework with code coverage:

Executes unit and integration tests
Measures code coverage
Supports asynchronous tests

Tools Installation

# Install development dependencies
pip install -r requirements-dev.txt

Complete Validation

Option 1: Automated Script (Recommended)

# Run all validations at once
./validate.sh

The validate.sh script automatically executes:

✅ Format checking (Ruff)
✅ Code linting (Ruff)
✅ Type checking (mypy)
✅ Security analysis (Bandit)
✅ Unit tests with coverage (pytest)

Option 2: Individual Commands

# 1. Check formatting
ruff format --check model_manager/ tests/

# 2. Check linting
ruff check model_manager/ tests/

# 3. Check types
mypy model_manager/

# 4. Security analysis
bandit -r model_manager/ -ll

# 5. Run tests
pytest tests/ --cov=model_manager --cov-report=term-missing

Automatic Fixes

Some tools can automatically fix issues:

# Format code automatically
ruff format model_manager/ tests/

# Fix linting issues automatically
ruff check --fix model_manager/ tests/

Configuration

All tools are configured in the pyproject.toml file:

Ruff: Linting rules, formatting, complexity
mypy: Type checking settings
pytest: Test and coverage options
Bandit: Security rules

CI/CD Integration

The .github/workflows/quality-gate.yml workflow automatically runs all validations on each push/PR:

✅ Formatting and linting block merge if they fail
⚠️ Type checking and security generate warnings but don't block
✅ Tests must pass with minimum 80% coverage

Best Practices

Before Commit: Run ./validate.sh to ensure quality
During Development: Use ruff check --watch for real-time feedback
Type Hints: Add type hints to new functions for better validation
Tests: Maintain coverage above 80%
Security: Review and fix all Bandit warnings

Testing

Test Structure

tests/
├── activities/      # Activity implementation tests
├── workflows/       # Workflow orchestration tests
├── utils/          # Utility function tests
└── worker/         # Worker tests

Test Execution

# Install test dependencies
pip install pytest pytest-cov pytest-asyncio

# Run tests with coverage
pytest --cov=model_manager --cov-report=html

# Run specific test modules
pytest tests/activities/test_gates.py
pytest tests/workflows/test_predictions_batch.py

Monitoring and Metrics

The Model Manager system exposes comprehensive Prometheus metrics for operational visibility and performance monitoring:

Application Health Metrics

app_up: Application health status (1=healthy, 0=unhealthy)
- Labels: pod_id

Prediction Operation Metrics

model_manager_predictions_written_count: Counter for successful prediction exports
- Labels: pod_id, model_name, pipeline_name
model_manager_prediction_confidence_monitor: Gauge for current prediction confidence levels
- Labels: pod_id, model_name, pipeline_name
model_manager_prediction_response_time_monitor: Histogram for prediction response times
- Labels: pod_id, model_name, pipeline_name
- Buckets: [0.01, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0]

Data Quality Metrics

Filter pass/fail rates through notification system
MLFlow API response validation metrics
Data quality gate performance tracking

Configuration

Environment Variables

Variable	Description	Default	Required
`TEMPORAL_HOST`	Temporal server address	`localhost:7233`	Yes
`TEMPORAL_NAMESPACE`	Temporal namespace	`model-manager`	No
`POSTGRES_HOST`	PostgreSQL hostname	`localhost`	Yes
`POSTGRES_PORT`	PostgreSQL port	`5432`	Yes
`POSTGRES_USER`	PostgreSQL username	`sientia`	Yes
`POSTGRES_PASSWORD`	PostgreSQL password	`sientia`	Yes
`POSTGRES_DBNAME`	PostgreSQL database	`sientia`	Yes
`POSTGRES_MIN_CONNECTIONS`	Minimum PostgreSQL connections	`5`	No
`POSTGRES_MAX_CONNECTIONS`	Maximum PostgreSQL connections	`20`	No
`MLFLOW_HOST`	MLFlow server hostname	`http://localhost`	Yes
`MLFLOW_PORT`	MLFlow server port	`5080`	Yes
`MLFLOW_USERNAME`	MLFlow username	`aignosi`	Yes
`MLFLOW_PASSWORD`	MLFlow password	`aignosi`	Yes
`MINIO_ENDPOINT_URL`	MinIO server endpoint	`http://minio.minio.svc.cluster.local:9000`	Yes
`MINIO_ACCESS_KEY`	MinIO access key	`minioadmin`	Yes
`MINIO_SECRET_KEY`	MinIO secret key	`minioadmin`	Yes
`MINIO_REGION`	MinIO region	`us-east-1`	No
`MINIO_USE_SSL`	Enable SSL for MinIO	`false`	No
`MINIO_MAX_RETRY_ATTEMPTS`	Maximum retry attempts	`3`	No
`MINIO_RETRY_MODE`	Retry mode (standard/adaptive)	`adaptive`	No
`MINIO_CONNECT_TIMEOUT`	Connection timeout (seconds)	`10`	No
`MINIO_READ_TIMEOUT`	Read timeout (seconds)	`60`	No
`MONGODB_URL`	MongoDB connection URI	`localhost:27018`	Yes
`MONGODB_USERNAME`	MongoDB username	`root`	Yes
`MONGODB_PASSWORD`	MongoDB password	`wKZDbMNU1c`	Yes
`MONGODB_DATABASE_NAME`	MongoDB database name	`sientia`	Yes
`MONGODB_TTL_INDEX_HOURS`	MongoDB TTL index hours	`1`	No
`LOG_LEVEL`	Application log level	`INFO`	No
`PROJECT_NAME`	Project name for metrics	`model-manager`	No
`HTTP_METRICS_PORT`	Prometheus metrics port	`9090`	No
`HTTP_SDK_METRICS_PORT`	Temporal SDK metrics port	`9091`	No
`POD_ID`	Kubernetes pod identifier	`None`	No

Workflow Configuration

MongoDB pipeline configuration:

Predictions Batch Workflow configuration sample

This is the configuration for the Predictions Batch Workflow, to be inserted into the MongoDB pipeline collection.

{
  "schedule_name": "laborious-orchestrated-pipeline",
  "model_id": "1",
  "workflow_type": "predictions_batch",
  "frequency": "30s",
  "max_retry_policy": 1,
  "query": "select * from sientia_data.laborious_data where model_id = 1 and \"timestamp\" > NOW() - INTERVAL '5 minutes' order by \"timestamp\" desc limit 30;",
  "write_tags": [
    {
      "server_id": "server1",
      "type": "prediction",
      "addr": "ns=2;i=5",
      "data_type": "double"
    },
    {
      "server_id": "server1",
      "type": "confidence",
      "addr": "ns=2;i=6",
      "data_type": "double"
    }
  ],
  "input_filters": {
    "EMPTY_DATA": {"POLICY": "STOP"},
    "SPECIFIC_VARIABLES_NULL_VALUES": {
      "POLICY": "CONTINUE",
      "config": {"variables": ["Counter"]}
    }
  },
  "mlflow_transform_filters": {
    "API_ERROR": {"POLICY": "REPEAT"},
    "NAN_VALUES": {"POLICY": "STOP"}
  },
  "mlflow_predict_filters": {
    "API_ERROR": {"POLICY": "CONTINUE"}
  },
  "path_priority": ["STOP", "CONTINUE", "REPEAT"],
  "active": true,
  "updated_at": {
    "$date": "2025-09-16T10:00:00.000Z"
  },
  "datetime_columns": ["timestamp", "created_at"],
  "predictions_storage_policy": "lts:1"
}

This is the configuration created by the Orchestrator in Temporal.

{
  "datetime_columns":["timestamp","created_at"],
  "frequency":"15m",
  "input_filters":{"EMPTY_DATA":{"config":{},"policy":"STOP"}},
  "max_retry_policy":1,
  "mlflow_predict_filters":{"API_ERROR":{"config":{},"policy":"CONTINUE"}},
  "mlflow_transform_filters":{
    "API_ERROR":{"config":{},"policy":"CONTINUE"},
    "EMPTY_DATA":{"config":{},"policy":"STOP"}
  },
  "model_config":{
    "is_compressed":true,
    "predict_flavor":"pyfunc",
    "retention_minutes":60,
    "retention_target":"artifact",
    "transform_function_keyword":"transform"
  },
  "model_id":"352",
  "model_name":"courier",
  "path_priority":["STOP","CONTINUE","REPEAT"],
  "predictions_storage_policy":"lts:1",
  "query":"select * from sientia_data.laborious_data where model_id = 352 order by \"timestamp\" desc limit 300;",
  "retention_time":3600,
  "schedule_name":"laborious-courier",
  "schema":"sientia_data",
  "table_name":"predictions",
  "updated_at":"2025-09-12 19:35:01.600000+0000",
  "workflow_type":"predictions_batch"
}

Development

Code Quality & Testing

The project maintains 99%+ code coverage with comprehensive unit and integration tests.

Running Tests

# Run all tests with coverage
pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=xml --cov-report=html

# Run specific test file
pytest tests/activities/test_gates.py -v

# Run with coverage visualization
pytest tests/ --cov=model_manager --cov-report=xml
# Then open htmlcov/index.html in browser

Validation Script

The validate.sh script runs all quality checks before commit:

./validate.sh

This script performs:

✅ Code Formatting (Ruff)
✅ Code Linting (Ruff)
✅ Type Checking (mypy)
✅ Security Analysis (Bandit)
✅ Unit Tests (pytest with 80%+ coverage requirement)

Coverage Visualization

For real-time coverage feedback in VS Code/Windsurf:

Install Coverage Gutters extension

Configure .vscode/settings.json:

{
  "coverage-gutters.coverageBaseDir": "${workspaceFolder}",
  "coverage-gutters.coverageFileNames": ["coverage.xml"],
  "coverage-gutters.showLineCoverage": true,
  "coverage-gutters.showRulerCoverage": true
}

Run tests to generate coverage:

pytest tests/ --cov=model_manager --cov-report=xml

Activate Coverage Gutters: Press Ctrl+Shift+7 (or Cmd+Shift+7 on Mac)

Automated Versioning

The project uses semantic versioning based on branch patterns:

Branch Pattern	Version Change	Example
`release/*`	Major version bump	`2.0.0`
`feature/*`	Minor version bump	`1.2.0`
`fix/*`	Patch version bump	`1.1.3`
`rc/*`	Release candidate	`1.1.2-rc2`

Version is calculated automatically in the CI/CD pipeline and passed to SonarQube.

Project Structure

model_manager/
├── activities/              # Temporal activity implementations
│   ├── __init__.py
│   ├── activities.py       # Main activities orchestrator (combines all activities)
│   ├── gates.py            # Data quality gates and filtering logic
│   ├── minio.py            # MinIO object storage operations
│   └── mlflow.py           # MLFlow model operations (predict/transform)
├── workflows/               # Temporal workflow definitions
│   ├── __init__.py
│   ├── predictions_batch.py # Main batch prediction workflow entry point
│   ├── minimal_retrain.py  # Model retraining workflow
│   └── sub_workflows/      # Sub-workflow implementations
│       ├── __init__.py
│       ├── prediction_process.py # Core prediction pipeline
│       └── format_and_export_prediction.py # Data export workflow
├── worker/                  # Worker implementation
│   ├── __init__.py
│   └── worker.py           # Main worker orchestrator (Temporal client setup)
├── utils/                   # Utility functions and helpers
│   ├── __init__.py
│   ├── connectors_config.py # Environment-based configuration builders
│   ├── filters/            # Data quality validation filters
│   │   ├── __init__.py
│   │   ├── conditional_filters.py # Input data validation filters
│   │   └── mlflow_filters.py # MLFlow response validation filters
│   └── repository/         # Data access layer
│       ├── __init__.py
│       └── model_repository.py # MLFlow model operations and retraining
├── metrics.py               # Prometheus metrics definitions
└── __init__.py

Adding New Features

Follow Temporal patterns for new workflows and activities
Add comprehensive docstrings for all public methods
Include Prometheus metrics for monitoring
Add unit tests for new functionality (maintain 80%+ coverage)
Run validation script (./validate.sh) before committing
Update this README with new features and configuration

Test Coverage Guidelines

Minimum coverage: 80% (enforced by CI/CD)
Current coverage: 99%+ 🎯
Test all branches: Use Coverage Gutters to identify uncovered lines
Mock external dependencies: Use unittest.mock for external services
Async testing: Use pytest-asyncio for async activities and workflows

Test structure:

tests/
├── activities/      # Activity tests
├── workflows/       # Workflow tests
├── utils/          # Utility tests
└── worker/         # Worker tests

Troubleshooting

Common Issues

Temporal Connection Failures
- Verify Temporal server is running and accessible
- Check namespace configuration and permissions
- Review server logs for connection issues
MLFlow Connection Issues
- Verify MLFlow server is running and accessible
- Check authentication credentials and permissions
- Ensure model names and versions exist
Database Connection Issues
- Verify PostgreSQL service is running
- Check connection credentials and network access
- Ensure proper connection pool configuration
Workflow Execution Failures
- Review activity error logs and notifications
- Check data quality filter configurations
- Verify input data format and required fields

Debug Mode

Enable debug logging by setting the log level:

export LOG_LEVEL=DEBUG

Performance Tuning

Key Parameters

Worker Concurrency: Adjust max_concurrent_workflow_tasks and max_concurrent_activities
Connection Pools: Optimize database connection pool sizes
Model Retention: Configure MLFlow model retention based on requirements
Batch Sizes: Adjust data processing batch sizes for optimal throughput

Scaling Considerations

Horizontal Scaling: Deploy multiple worker instances
Task Queue Distribution: Use multiple task queues for different workflow types
Database Performance: Optimize indexes and connection pooling
MLFlow Performance: Configure appropriate model serving resources

Contributing

Fork the repository
Create a feature branch
Make your changes with comprehensive testing
Update documentation and docstrings
Submit a pull request

Code Quality Standards

Follow PEP 8 style guidelines
Include comprehensive docstrings for all public methods
Maintain test coverage above 80%
Use type hints where appropriate
Follow Temporal.io best practices

License

This project is licensed under the terms specified in the LICENSE file.

Support

For support and questions:

Check the troubleshooting section above
Review the metrics and logs for error patterns
Open an issue in the project repository
Contact the development team

Note: The Model Manager system is designed for production use in industrial ML environments. Ensure proper security configuration and network isolation for production deployments.

README.md

Sientia DataOps Model Manager

📑 Table of Contents

Features

Core Functionality

Advanced Capabilities

Development & Quality Assurance

Architecture

Architecture Principles

1. Separation of Concerns

2. Fault Tolerance & Resilience

3. Scalability & Performance

4. Observability & Monitoring

Key Components

Worker (model_manager/worker/worker.py)

Workflows (model_manager/workflows/)

Activities (model_manager/activities/)

Data Services (model_manager/utils/)

Data Flow Architecture

1. Batch Prediction Pipeline

2. Model Retraining Pipeline

Security Architecture

Authentication & Authorization

Network Security

Data Security

Workflows

1. Predictions Batch Workflow (predictions_batch.py)

Purpose

Execution Flow

Key Features

Input Parameters

Architecture Diagram

2. Prediction Process Workflow (prediction_process.py)

Purpose

Execution Flow

Key Features

Input Parameters

Architecture Diagram

3. Format and Export Prediction Workflow (format_and_export_prediction.py)

Purpose

Execution Flow

Key Features

Architecture Diagram

4. Minimal Retrain Workflow (minimal_retrain.py)

Purpose

Execution Flow

Architecture Diagram

Installation & Setup

Prerequisites

MinIO Setup

Environment Setup

Temporal Namespace Setup

Why Create a Namespace?

When to Create?

How to Create the Namespace

Namespace Configuration

Verification

Troubleshooting

Local Development Setup

How to Run

Running the Model Manager Application

Running Tests and Coverage

Manual Test Execution

Manual Application Execution

Code Quality & Validation

Overview

Validation Tools

1. Ruff - Linting and Formatting ⚡

2. mypy - Type Checking 🏷️

3. Bandit - Security Analysis 🔒

4. pytest - Automated Testing 🧪

Tools Installation

Complete Validation

Option 1: Automated Script (Recommended)

Option 2: Individual Commands

Automatic Fixes

Configuration

CI/CD Integration

Best Practices

Testing

Worker (`model_manager/worker/worker.py`)

Workflows (`model_manager/workflows/`)

Activities (`model_manager/activities/`)

Data Services (`model_manager/utils/`)

1. Predictions Batch Workflow (`predictions_batch.py`)

2. Prediction Process Workflow (`prediction_process.py`)

3. Format and Export Prediction Workflow (`format_and_export_prediction.py`)

4. Minimal Retrain Workflow (`minimal_retrain.py`)