An enterprise-grade ML model training orchestration platform built on Temporal. Provides robust, scalable workflows for training machine learning models with comprehensive validation, experiment tracking, and automated resource management. Integrates seamlessly with MLFlow for model persistence and PostgreSQL for experiment tracking.

📑 Table of Contents

Features
Architecture
Workflows
- Train Model Workflow
- Cleanup Files Workflow
Installation & Setup
How to Run
Code Quality & Validation
Testing
Monitoring and Metrics
- Application Health Metrics
- Training Metrics
Configuration
- Environment Variables
- Workflow Configuration
Development
Troubleshooting
- Common Issues
- Debug Mode
Performance Tuning
- Key Parameters
- Scaling Considerations
Contributing
- Code Quality Standards
License
Support
Local GitHub Actions Testing (act)
Docker
Helm Chart

Features

Core Functionality

ML Model Training Pipeline: Complete training workflow from validation to deployment using MLFlow
Polynomial Regression Support: Configurable polynomial degree with interaction terms and mandatory scaler validation
Automated File Cleanup: Scheduled cleanup of stale files from MinIO and local filesystem
Temporal Workflow Orchestration: Robust workflow management with granular retry policies and fault tolerance
Parameter Validation: Defense-in-depth validation with business rules and type checking
Experiment Tracking: Comprehensive status tracking in PostgreSQL database
Resource Management: Automatic cleanup of temporary files and storage
Comprehensive Monitoring: Prometheus metrics and detailed logging for operational visibility
Scheduled Jobs: Automated daily cleanup with configurable cron schedules

Advanced Capabilities

Granular Retry Policies: Different strategies for network, training, MLFlow, database, and filesystem operations
Configurable Timeouts: Environment variable-based timeouts supporting large training files (up to 200MB)
Notification System: Integrated alerting and notification management via MongoDB
Scalable Architecture: Kubernetes-ready deployment with horizontal scaling support
MLFlow Integration: Seamless model and artifact persistence to MLFlow tracking server
Per-Variable Lag Configuration: Flexible lag settings for each variable independently
Date Range Filtering: Filter training data by start/end dates and removed intervals
NaN Treatment Options: Configurable handling of missing values (drop, linear interpolation)
RCE Drift Metrics: Reduced Coulomb Energy metrics for drift detection

Development & Quality Assurance

Code Quality Tools: Ruff (linting/formatting), mypy (type checking), Bandit (security analysis)
Automated Validation: Pre-commit validation script (validate.sh) and CI/CD integration
Comprehensive Testing: pytest with async support and 100% code coverage 🎯
Type Safety: Static type checking with mypy for improved code reliability
Coverage Visualization: Integration with Coverage Gutters for real-time coverage feedback
Automated Versioning: Semantic versioning based on branch patterns (release/, feature/, fix/, rc/)
Integration Test Scenarios: JSON-based test scenarios with batch execution support

Architecture

The Model Manager system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments.

Architecture Principles

1. Separation of Concerns

Worker Layer: Manages Temporal workers, task queues, and application lifecycle
Workflow Layer: Orchestrates business logic and process coordination
Activity Layer: Implements specific operations and external system interactions
Data Layer: Handles data persistence, caching, and external service connections

2. Fault Tolerance & Resilience

Automatic Retry Policies: Configurable retry strategies for transient failures
Circuit Breaker Pattern: Prevents cascading failures in external service calls
Graceful Degradation: System continues operating with reduced functionality
Comprehensive Error Handling: Detailed error reporting and notification integration

3. Scalability & Performance

Horizontal Scaling: Multiple worker instances for load distribution
Task Queue Isolation: Separate queues for different workflow types
Connection Pooling: Optimized database and external service connections
Asynchronous Processing: Non-blocking operations for improved throughput

4. Observability & Monitoring

Prometheus Metrics: Comprehensive system and business metrics
Structured Logging: Consistent log format with correlation IDs
Health Checks: Endpoint health monitoring and alerting
Performance Tracing: Request flow tracking and bottleneck identification

Key Components

Worker (`model_manager/worker/worker.py`)

Purpose: Main application orchestrator managing Temporal workers and task queues
Responsibilities:
- Temporal client initialization and connection management
- Worker lifecycle management and graceful shutdown
- Task queue configuration and load balancing
- Prometheus metrics server initialization
- Notification handler setup and configuration
Key Features:
- Automatic scaling with PollerBehaviorAutoscaling
- Health check endpoints for Kubernetes liveness/readiness probes
- Graceful shutdown with cleanup procedures
- Multi-instance deployment support
- Two dedicated task queues:
  - train_model-queue: ML model training workflows
  - cleanup-queue: File cleanup workflows
- Automated cleanup schedule management

Workflows (`model_manager/workflows/`)

TrainModel: Complete ML model training pipeline from validation to deployment
CleanupFiles: Automated cleanup of stale files from MinIO and local filesystem
Key Features:
- Temporal workflow definitions with granular retry policies
- Parameter validation with business rules
- Comprehensive error handling and status tracking
- Configurable timeouts for different operation types
- Automatic resource cleanup and management
- Scheduled cleanup jobs with cron expressions

Activities (`model_manager/activities/`)

Activities: Main activity orchestrator combining all functionality through multiple inheritance
ExperimentTracking: ML experiment lifecycle tracking and database operations
- Unified update_experiment_run() method for all experiment status updates
- Support for three update types: STATUS, STATUS_WITH_ERROR, MODEL_SAVED
- Automatic error message truncation (1024 chars)
- Connection pooling and retry logic
Training: ML model training operations with MLFlow and MinIO integration
- Unified train_model() method for complete training pipeline
- Receives pre-downloaded files (BytesIO) to avoid memory leaks
- Returns success/failure status with TrainModelResult or error message
- No exception raising on failure - allows workflow to handle errors gracefully
- Integration with TrainingRepository for business logic separation
- MLFlow model saving and artifact management
- MinIO object storage operations
- Polynomial Regression: Support for configurable degree and interaction terms
- Training Predictions: Calculates y_train_pred before denormalization for accurate metrics
Cleanup: File and directory cleanup operations
- cleanup_temp_directories(): Cleans local temporary directories
- Configurable retention period (default: 24 hours)
- Dry-run mode for testing
Key Features:
- Multiple inheritance pattern for unified activity interface
- Parameter validation with business rules
- MLFlow integration for model persistence
- Comprehensive error handling and notification integration
- Experiment tracking with automatic status management
- Timestamp-based file cleanup with regex pattern matching

Data Services (`model_manager/utils/`)

Connectors Config: Environment variable-based configuration management
Repository: Data access layer for training and MLFlow operations
- training_repository.py: Training business logic and operations
- model_repository.py: MLFlow artifact generation and model persistence
Models: Data models and schemas
- train_model_params.py: Training parameters model with comprehensive validation
- train_model_result.py: Training result model (includes y_train_pred)
- experiment_status.py: Experiment status enum
Key Features:
- Environment variable-based configuration with sensible defaults
- Connection pool management and optimization
- Security credential management
- Configuration validation and error handling
- Type-safe data models with validation

Data Flow Architecture

Model Training Pipeline

Training Request → Parameter Validation → MinIO Data Download → 
Model Training → MLFlow Model Save → Resource Cleanup → Status Update

Key Stages:

Validation: Experiment run ID and training parameters validation
Data Acquisition: Download training data from MinIO storage
Training: Execute ML model training with validated parameters
Persistence: Save trained model and artifacts to MLFlow
Cleanup: Remove temporary files and update experiment status

Security Architecture

Authentication & Authorization

MLFlow API Authentication: Username/password with secure transmission
Database Connection Security: Encrypted connections with credential management
Kubernetes Secrets Integration: Secure credential storage and access

Network Security

TLS/SSL Encryption: Secure communication channels
Network Isolation: Kubernetes network policies and service mesh
Firewall Rules: Controlled access to external services
VPN Integration: Secure remote access and management

Data Security

Data Encryption: At-rest and in-transit encryption
Access Control: Role-based access control (RBAC)
Audit Logging: Comprehensive access and operation logging
Data Retention: Configurable data lifecycle management

Workflows

Train Model Workflow (`train_model.py`)

The TrainModel workflow orchestrates the complete ML model training pipeline from parameter validation through model saving and cleanup.

Purpose

Model Training: Complete ML model training pipeline
Parameter Validation: Defense-in-depth validation with business rules
Resource Management: Automatic cleanup of temporary resources
Status Tracking: Comprehensive experiment tracking in database
Error Handling: Robust error handling with detailed context logging

Execution Flow

Validate Experiment Run ID: Critical validation before any DB updates
Validate Training Parameters: Type checking + business rules validation
Download Training Data: Fetch file from MinIO storage
Train Model: Execute ML model training with validated parameters
Save to MLFlow: Save trained model and artifacts to MLFlow
Cleanup Resources: Delete temporary files and MinIO data

Key Features

Granular Retry Policies: Different strategies for network, training, MLFlow, database, and filesystem operations
Configurable Timeouts: Environment variable-based timeouts supporting files up to 200MB
Idempotent Cleanup: Safe replay with Temporal workflow replay mechanism
Structured Logging: Rich context in error messages for debugging
Business Validation: 10 business rules including range checks, consistency validation, and data integrity

Input Parameters

{
  "experiment_run_id": 123,
  "target_variable": "price",
  "variable_columns": ["feature1", "feature2"],
  "train_size": 80,
  "shuffle": true,
  "use_scaler": true,
  "include_ar": false,
  "bucket_name": "ml-data",
  "file_name": "training_data.csv",
  "line_separator": "\n",
  "decimal_separator": ".",
  "lag_train": {"feature1": 0, "feature2": 2},
  "lag_val": {"feature1": 0, "feature2": 1},
  "rem_static_win": false,
  "static_threshold": null,
  "low_lim": {"feature1": 0.0, "feature2": 0.0},
  "upp_lim": {"feature1": 100.0, "feature2": 100.0},
  "window": 10,
  "experiment_name": "production_model_v1",
  "removed_intervals": [],
  "model_name": "Linear Regression",
  "degree": 1,
  "interaction_only": false,
  "nan_treatment": "drop",
  "start_date": null,
  "end_date": null,
  "scaler_name": "Standard Scaler",
  "support_filters": {}
}

Architecture Diagram

flowchart TD
    A[1. validate_experiment_run_id] --> B[2. validate_train_params]
    B --> C[3. fetch_file_from_minio]
    C --> D[4. train_model]
    D --> E[5. save_model]
    E --> F[6. cleanup_run_directory]
    F --> G[7. delete_file_from_minio]
    
    B -.-> DB[(PostgreSQL)]
    C -.-> MinIO[MinIO Storage]
    D -.-> Training[ML Training]
    E -.-> MLFlow[MLFlow]
    F -.-> FS[Filesystem]
    G -.-> MinIO

Retry Strategies

The workflow implements 5 different retry policies optimized for each operation type:

Operation Type	Initial Interval	Max Interval	Backoff	Max Attempts	Use Case
Network	1s	10s	2.0x	5	MinIO operations (transient network errors)
No Retry	-	-	-	1	Training/Validation (permanent data errors)
MLFlow	5s	30s	2.0x	3	MLFlow operations (API timeouts)
Database	2s	20s	2.0x	5	PostgreSQL updates (lock contention)
Filesystem	2s	10s	1.5x	3	Cleanup operations (busy resources)

Business Validation Rules

The workflow validates comprehensive business rules beyond type checking:

train_size: Must be between 10-100%
variable_columns: Cannot be empty
lag_train, lag_val: Per-variable dictionaries with non-negative values
window: Must be non-negative integer
low_lim/upp_lim: Must have same keys and low < upp for each variable
target_variable: Cannot be empty
bucket_name, file_name, experiment_name: Cannot be empty or whitespace
degree: Must be at least 1; must be >= 2 for Polynomial Regression
nan_treatment: Must be one of 'drop', 'linear interpolation', 'fill linear'
scaler_name: Must be 'Standard Scaler' or 'None'
model_name: Must be 'Linear Regression' or 'Polynomial Regression'
Polynomial Regression requires Scaler: Models with degree > 1 must have a scaler to prevent numerical overflow
Linear Regression requires degree 1: Linear models must have degree = 1
static_threshold: When rem_static_win is true and static_threshold has a value, it must be between 1 and 1000 (inclusive). If null, defaults to 1.

Cleanup Files Workflow (`cleanup_files.py`)

The CleanupFiles workflow provides automated cleanup of stale files from MinIO storage and local temporary directories. It runs on a scheduled basis (default: daily at midnight UTC) to maintain storage hygiene.

Purpose

Storage Management: Automatic removal of old files from MinIO and local filesystem
Retention Policy: Configurable retention period (default: 24 hours)
Scheduled Execution: Cron-based scheduling for automated cleanup
Resource Optimization: Prevents storage bloat and reduces costs

Execution Flow

Cleanup MinIO Files: Scan and delete files older than retention period from MinIO bucket
Cleanup Local Directories: Remove temporary directories older than retention period

Key Features

Timestamp-Based Cleanup: Uses filename/directory timestamps for age determination
Pattern Matching: Regex patterns for MinIO (timestamp-filename) and directories (name_YYYYMMDD_HHMMSS_microseconds)
Configurable Retention: Environment variable-based retention period
Dry-Run Mode: Test cleanup operations without actual deletion
Idempotent: Safe to run multiple times
Error Handling: Continues cleanup even if individual operations fail

Input Parameters

{
  "bucket_name": "model-training"  // Optional, defaults to DEFAULT_CLEANUP_BUCKET env var
}

Schedule Configuration

The cleanup schedule is automatically created when the worker starts:

Configuration	Environment Variable	Default	Description
Schedule ID	`CLEANUP_SCHEDULE_ID`	`cleanup-files-daily`	Unique identifier for the schedule
Cron Expression	`CLEANUP_CRON`	`0 0 * * *`	Daily at midnight UTC
Timezone	`CLEANUP_TIMEZONE`	`UTC`	Timezone for cron execution
Task Queue	`CLEANUP_TASK_QUEUE`	`cleanup-queue`	Dedicated task queue
Execution Timeout	`CLEANUP_EXECUTION_TIMEOUT_HOURS`	`1`	Maximum execution time (hours)
Retention Period	`CLEANUP_RETENTION_HOURS`	`24`	Files older than this are deleted
Dry Run	`CLEANUP_DRY_RUN`	`false`	Test mode without actual deletion

Architecture Diagram

flowchart TD
    A[Scheduled Trigger] --> B[cleanup_temp_directories]

    B -.-> FS[Local Filesystem]

Retry Strategies

Operation Type	Initial Interval	Max Interval	Backoff	Max Attempts	Use Case
Network	1s	10s	2.0x	5	MinIO operations (transient network errors)
No Retry	-	-	-	1	Local filesystem operations (permanent errors)

Cleanup Patterns

MinIO Files:

Pattern: {timestamp}-{filename} where timestamp is milliseconds since epoch
Example: 1638360000000-training_data.csv
Retention: Files older than CLEANUP_RETENTION_HOURS are deleted

Local Directories:

Pattern: {name}_{YYYYMMDD}_{HHMMSS}_{microseconds}
Example: temp_20231201_143052_123456
Retention: Directories older than CLEANUP_RETENTION_HOURS are deleted

Installation & Setup

Prerequisites

Python 3.11+
Temporal server/cluster
PostgreSQL database
MLFlow server
MinIO object storage (for MLFlow artifacts)
MongoDB server (for notifications)

Note: External dependencies must be available either through:

Kubernetes cluster deployment
Docker Compose setup
Cloud-managed services
Local installations

MinIO Setup

MinIO is required for MLFlow artifact storage. For detailed installation and configuration instructions, refer to:

📚 Install MinIO via Helm Chart on K8s

This guide covers:

Helm chart installation on Kubernetes
Storage configuration and persistence
Access credentials setup
Integration with MLFlow

Environment Setup

Clone the repository:

git clone <repository-url>
cd sientia-dataops-model-manager

Create virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Install dependencies:
```
pip install -r requirements.txt
```
Configure environment variables (see Configuration section)
Run validation script:
```
./validate.sh
```

Temporal Namespace Setup

The Model Manager requires a dedicated Temporal namespace to isolate workflows and maintain proper execution history. The namespace must be created before starting the application.

Why Create a Namespace?

Isolation: Separates Model Manager workflows from other applications
Retention Control: Configures workflow history retention (default: 7 days)
Multi-tenancy: Enables multiple environments (dev, staging, prod) on same cluster
Security: Allows namespace-level access control and permissions

When to Create?

✅ Before first deployment in any environment
✅ Once per environment (dev, staging, production)
✅ After Temporal cluster setup or upgrade

How to Create the Namespace

Option 1: Using Temporal Admin Tools Pod (Recommended for Kubernetes)

# 1. List Temporal pods
kubectl get pods -n temporal

# 2. Connect to admin tools pod
kubectl exec -it -n temporal <temporal-admin-tools-pod-name> -- bash

# 3. Create namespace
tctl --namespace model-manager namespace register \
    --retention 7 \
    --description "Model Manager - ML Model Orchestration Namespace"

# 4. Verify creation
tctl --namespace model-manager namespace describe

# 5. Exit pod
exit

Option 2: Using Port Forward (Local Development)

# 1. Port forward Temporal frontend
kubectl port-forward -n temporal svc/temporal-frontend 7233:7233

# 2. In another terminal, create namespace
tctl --address localhost:7233 \
     --namespace model-manager \
     namespace register \
     --retention 7 \
     --description "Model Manager - ML Model Orchestration Namespace"

# 3. Verify
tctl --address localhost:7233 --namespace model-manager namespace describe

Option 3: Direct kubectl exec (One-liner)

kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
    tctl --namespace model-manager namespace register \
    --retention 7 \
    --description "Model Manager - ML Model Orchestration Namespace"

Namespace Configuration

Parameter	Value	Description
Name	`model-manager`	Namespace identifier (configurable via `TEMPORAL_NAMESPACE` env var)
Retention	`7 days`	Workflow history retention period
Description	`Model Manager - ML Model Orchestration Namespace`	Human-readable description

Verification

To verify the namespace was created successfully:

# List all namespaces
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- tctl namespace list

# Describe specific namespace
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
    tctl --namespace model-manager namespace describe

Troubleshooting

Error: "namespace already exists"

✅ This is fine! The namespace is already configured
No action needed, proceed with application deployment

Error: "connection refused"

❌ Temporal server is not accessible
Verify Temporal cluster is running: kubectl get pods -n temporal
Check network connectivity and port forwarding

Error: "permission denied"

❌ Insufficient permissions to create namespace
Contact cluster administrator for namespace creation
Or request elevated permissions for your service account

Local Development Setup

Clone the repository

git clone <repository-url>
cd sientia-dataops-model-manager

Create virtual environment

conda create -p ./venv python=3.11
conda activate ./venv

Install dependencies
Install github cli bash sudo apt update sudo apt install gh -y
Authenticate with github bash gh auth login
Install Python dependencies bash python -m pip install --upgrade pip # Install production dependencies pip install -r requirements.txt # Install development and testing tools pip install -r requirements-dev.txt

Create environment configuration file

cp .env.example .env
# Edit .env with your connection details

Configure external dependencies

The Model Manager requires connections to several external services. For local development, you can use the provided port-forward script to establish connections to services running in your Kubernetes cluster.

Services and Ports:

Local Port	Service	Description	Namespace
`5432`	`paradedb-rw`	PostgreSQL Database	`paradedb`
`45249`	`sientia-tracker-mlflow-tracking`	MLflow Tracking Server	`sientia-tracker`
`37463`	`temporal-frontend`	Temporal gRPC API	`temporal`
`8080`	`temporal-web`	Temporal Web UI	`temporal`
`42297`	`my-release-mongodb`	MongoDB Database	`mongodb`
`36577`	`minio`	MinIO Object Storage	`minio`

Managing Port Forwards:

# View active port forwards
jobs -l

# Stop all port forwards
jobs -p | xargs kill

Manual Port Forwarding:

If you prefer manual control or need different ports:

# PostgreSQL
kubectl -n paradedb port-forward svc/paradedb-rw 5432:5432 &

# MLflow
kubectl -n sientia-tracker port-forward svc/sientia-tracker-mlflow-tracking 45249:80 &

# Temporal
kubectl -n temporal port-forward svc/temporal-frontend 37463:7233 &

# Temporal UI
kubectl -n temporal port-forward svc/temporal-web 8080:8080 &

# MongoDB
kubectl -n mongodb port-forward svc/my-release-mongodb 42297:27017 &

# MinIO
kubectl -n minio port-forward svc/minio 36577:9000 &

How to Run

Running the Model Manager Application

Use the provided script to run the application locally:

# Make script executable (first time only)
chmod +x run_local.sh

# Run the application
./run_local.sh

The script will:

Load environment variables from .env
Start the model-manager worker application

Note: Activate your virtual environment before running the script:

source ./venv/bin/activate  # or: conda activate ./venv
./run_local.sh

Running Tests and Coverage

Run tests with coverage using pytest directly:

# Run all tests with coverage
pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=html

# Open the coverage report in your browser
open htmlcov/index.html  # macOS
xdg-open htmlcov/index.html  # Linux

Manual Test Execution

You can also run tests manually:

# Activate virtual environment
source ./venv/bin/activate

# Run all tests
pytest

# Run with coverage
pytest --cov=model_manager --cov-report=html

# Run specific test categories
pytest tests/activities/
pytest tests/workflow/

Manual Application Execution

For manual execution without scripts:

# Activate virtual environment
source ./venv/bin/activate

# Load environment variables (if using .env file)
if [ -f .env ]; then
    export $(cat .env | grep -v '^#' | xargs)
fi

# Start the model-manager worker
python -m model_manager.worker.worker

Code Quality & Validation

Overview

Since Python is not a compiled language, we use a robust set of tools to validate code quality, security, and correctness before execution. These tools detect errors, style issues, security vulnerabilities, and ensure code consistency.

Validation Tools

1. Ruff - Linting and Formatting ⚡

Modern and extremely fast tool (written in Rust) that replaces multiple tools:

Linting: Detects code errors, style issues (PEP 8), common bugs
Formatting: Automatically formats code consistently
Speed: 10-100x faster than Flake8/Black

2. mypy - Type Checking 🏷️

Static type checker that analyzes type hints:

Detects type errors before execution
Improves code documentation
Prevents bugs related to incorrect types

3. Bandit - Security Analysis 🔒

Security vulnerability scanner:

Detects insecure code patterns
Identifies hardcoded passwords, SQL injection, etc.
Ensures compliance with security practices

4. pytest - Automated Testing 🧪

Testing framework with code coverage:

Executes unit and integration tests
Measures code coverage
Supports asynchronous tests

Tools Installation

# Install development dependencies
pip install -r requirements-dev.txt

Complete Validation

Option 1: Automated Script (Recommended)

# Run all validations at once
./validate.sh

The validate.sh script automatically executes:

✅ Format checking (Ruff)
✅ Code linting (Ruff)
✅ Type checking (mypy)
✅ Security analysis (Bandit)
✅ Unit tests with coverage (pytest)

Option 2: Individual Commands

# 1. Check formatting
ruff format --check model_manager/ tests/

# 2. Check linting
ruff check model_manager/ tests/

# 3. Check types
mypy model_manager/

# 4. Security analysis
bandit -r model_manager/ -ll

# 5. Run tests
pytest tests/ --cov=model_manager --cov-report=term-missing

Automatic Fixes

Some tools can automatically fix issues:

# Format code automatically
ruff format model_manager/ tests/

# Fix linting issues automatically
ruff check --fix model_manager/ tests/

Configuration

All tools are configured in the pyproject.toml file:

Ruff: Linting rules, formatting, complexity
mypy: Type checking settings
pytest: Test and coverage options
Bandit: Security rules

CI/CD Integration

The project uses reusable workflows from Aignosi/github_workflow_templates for CI/CD:

Quality Gate (`.github/workflows/quality-gate.yml`)

Runs automatically on each Pull Request to main:

✅ Formatting and linting block merge if they fail
⚠️ Type checking and security generate warnings but don't block
✅ Tests must pass with minimum 80% coverage
✅ SonarQube analysis for code quality metrics
✅ Automatic version calculation based on branch pattern

Deploy (`.github/workflows/deploy.yml`)

Runs automatically when a PR is merged to main:

✅ Builds and pushes Docker image to Azure Container Registry
✅ Creates GitHub release with calculated version
✅ Deploys to Kubernetes using Helm

Best Practices

Before Commit: Run ./validate.sh to ensure quality
During Development: Use ruff check --watch for real-time feedback
Type Hints: Add type hints to new functions for better validation
Tests: Maintain coverage above 80%
Security: Review and fix all Bandit warnings

Testing

Test Structure

tests/
├── activities/      # Activity implementation tests
├── workflows/       # Workflow orchestration tests
├── utils/           # Utility function tests
├── worker/          # Worker tests
├── schedules/       # Schedule configuration tests
└── sientia/         # Sientia module tests

Test Execution

# Install test dependencies
pip install pytest pytest-cov pytest-asyncio

# Run tests with coverage
pytest --cov=model_manager --cov-report=html

# Run specific test modules
pytest tests/activities/test_training.py
pytest tests/workflows/test_train_model.py

Integration Tests

The project includes integration tests that validate the complete training workflow against a running Temporal cluster. These tests use JSON-based scenario files for easy configuration and maintenance.

Running Integration Tests

# List available scenarios
python scripts/run_training_test.py --list

# Run a specific test scenario
python scripts/run_training_test.py --scenario 01-linear-regression-basic

# Run with custom CSV data file
python scripts/run_training_test.py --scenario 03-polynomial-regression-degree2 --csv /path/to/data.csv

# Run ALL scenarios sequentially with summary report
python scripts/run_training_test.py --all

# Run all scenarios with custom CSV
python scripts/run_training_test.py --all --csv docs/custom-data.csv

Batch Execution Output

When running all scenarios with --all, the script provides:

Progress indicators for each scenario ([1/10] Running scenario: ...)
Status symbols (✓ for passed, ✗ for failed)
Final summary with total/passed/failed counts
Detailed error messages for failed scenarios
Exit code 0 if all pass, 1 if any fail

Example output:

Running 10 scenarios...

[1/10] Running scenario: 01-linear-regression-basic
  Loaded scenario: 01-linear-regression-basic
  Uploaded CSV to MinIO: test-model-data-20231219-120000.csv
  Created experiment_run with ID: 42
  Workflow started: train-model-test-abc123
[1/10] ✓ 01-linear-regression-basic

...

============================================================
SUMMARY
============================================================
Total: 10 | Passed: 9 | Failed: 1
============================================================

✓ PASSED:
  - 01-linear-regression-basic
  - 02-linear-regression-with-scaler
  ...

✗ FAILED:
  - 05-linear-regression-with-lags
    Error: Failed to start Temporal workflow: connection refused

Test Scenarios

Test scenarios are defined as JSON files in docs/test-scenarios/. Each scenario configures a complete training workflow with specific parameters:

Scenario	Description	Key Features
`01-linear-regression-basic`	Basic linear regression	No scaler, no lags
`02-linear-regression-with-scaler`	Linear regression with normalization	Standard Scaler enabled
`03-polynomial-regression-degree2`	Polynomial regression (degree 2)	Requires scaler (mandatory)
`04-polynomial-regression-degree3`	Polynomial regression (degree 3)	Requires scaler (mandatory)
`05-linear-regression-with-lags`	Linear regression with lag features	Lag train/val configuration
`06-linear-regression-nan-interpolation`	Linear regression with NaN handling	`nanTreatment: "linear interpolation"`
`07-linear-regression-static-window-removal`	Linear regression with static window removal	`remStaticWin: true`
`08-linear-regression-with-limits`	Linear regression with variable limits	`lowLim`/`uppLim` configuration
`09-polynomial-degree2-with-scaler-and-lags`	Complete polynomial scenario	Scaler + lags + degree 2
`10-linear-regression-with-ar`	Linear regression with autoregressive variable	`includeAr: true`
`11-linear-regression-static-threshold-custom`	Linear regression with custom static threshold	`staticThreshold: 100`

Scenario File Structure

{
  "_description": "Human-readable description of the scenario",
  "experimentName": "test-experiment-name",
  "username": "user@example.com",
  "modelName": "Linear Regression",
  "targetVariable": "target_column_name",
  "variableColumns": ["feature1", "feature2"],
  "lagTrain": {"feature1": 0, "feature2": 0},
  "lagVal": {"feature1": 0, "feature2": 0},
  "remStaticWin": false,
  "staticThreshold": null,
  "lowLim": {},
  "uppLim": {},
  "window": 0,
  "useScaler": false,
  "includeAr": false,
  "trainSize": 80,
  "shuffle": true,
  "lineSeparator": ",",
  "decimalSeparator": ".",
  "removedIntervals": [],
  "degree": 1,
  "interactionOnly": false,
  "nanTreatment": "drop",
  "startDate": null,
  "endDate": null,
  "scalerName": "None",
  "supportFilters": {}
}

Creating New Scenarios

Copy an existing scenario file as a template
Modify parameters according to your test case
Save with a descriptive name: XX-description.json
Run with: python scripts/run_training_test.py --scenario XX-description

Important Validations

The training workflow enforces several business rules:

Polynomial Regression requires Scaler: Models with degree > 1 must have useScaler: true and a valid scalerName to prevent numerical overflow
Static Window Removal requires DatetimeIndex: Scenarios with remStaticWin: true require data with a timestamp column for the TimeSeriesDiscontinuityAnalyzer
Variable Limits Consistency: lowLim and uppLim must have matching keys, and lowLim[key] < uppLim[key] for all variables

Monitoring and Metrics

The Model Manager system exposes comprehensive Prometheus metrics for operational visibility and performance monitoring:

Application Health Metrics

app_up: Application health status (1=healthy, 0=unhealthy)
- Labels: pod_id

Workflow Execution Metrics

workflow_execution_total: Total workflow executions
- Labels: workflow_name, status (success/failure)
activity_execution_total: Total activity executions
- Labels: activity_name, status (success/failure)

Training Metrics

Training success/failure rates through notification system
Model save performance metrics
Experiment status tracking
RCE Drift Metrics: Reduced Coulomb Energy (RCE) for drift detection
- silverman_radius: Optimal bandwidth for kernel density estimation
- rce_reference: RCE value for reference data
- rce_current: RCE value for current data
- rce_drift: Drift score between reference and current distributions

Cleanup Metrics

Cleanup execution success/failure rates
Number of files deleted from MinIO
Number of directories cleaned from local filesystem
Cleanup duration and performance

Configuration

Environment Variables

Variable	Description	Default	Required
`TEMPORAL_HOST`	Temporal server address	`localhost:7233`	Yes
`TEMPORAL_NAMESPACE`	Temporal namespace	`model-manager`	No
`TEMPORAL_USE_TLS`	Enable TLS for Temporal connection	`false`	No
`TRAIN_TASK_QUEUE`	Task queue for training workflows	`train_model-queue`	No
`CLEANUP_TASK_QUEUE`	Task queue for cleanup workflows	`cleanup-queue`	No
`POSTGRES_HOST`	PostgreSQL hostname	`localhost`	Yes
`POSTGRES_PORT`	PostgreSQL port	`5432`	Yes
`POSTGRES_USER`	PostgreSQL username	`sientia`	Yes
`POSTGRES_PASSWORD`	PostgreSQL password	`sientia`	Yes
`POSTGRES_DBNAME`	PostgreSQL database	`sientia`	Yes
`POSTGRES_MIN_CONNECTIONS`	Minimum PostgreSQL connections	`5`	No
`POSTGRES_MAX_CONNECTIONS`	Maximum PostgreSQL connections	`20`	No
`MLFLOW_URL`	MLFlow server URL (full URL with protocol and port)	`http://localhost:5080`	Yes
`MLFLOW_USERNAME`	MLFlow username	`aignosi`	Yes
`MLFLOW_PASSWORD`	MLFlow password	`aignosi`	Yes
`MINIO_ENDPOINT_URL`	MinIO server endpoint	`http://minio.minio.svc.cluster.local:9000`	Yes
`MINIO_ACCESS_KEY`	MinIO access key	`minioadmin`	Yes
`MINIO_SECRET_KEY`	MinIO secret key	`minioadmin`	Yes
`MINIO_REGION`	MinIO region	`us-east-1`	No
`MINIO_USE_SSL`	Enable SSL for MinIO	`false`	No
`MINIO_MAX_RETRY_ATTEMPTS`	Maximum retry attempts	`3`	No
`MINIO_RETRY_MODE`	Retry mode (standard/adaptive)	`adaptive`	No
`MINIO_CONNECT_TIMEOUT`	Connection timeout (seconds)	`10`	No
`MINIO_READ_TIMEOUT`	Read timeout (seconds)	`60`	No
`MONGODB_URL`	MongoDB connection URI	`localhost:27018`	Yes
`MONGODB_USERNAME`	MongoDB username	`root`	Yes
`MONGODB_PASSWORD`	MongoDB password	`wKZDbMNU1c`	Yes
`MONGODB_DATABASE`	MongoDB database name	`sientia`	Yes
`MONGODB_TTL_INDEX_HOURS`	MongoDB TTL index hours	`1`	No
`CLEANUP_SCHEDULE_ID`	Cleanup schedule identifier	`cleanup-files-daily`	No
`CLEANUP_CRON`	Cleanup cron expression	`0 0 * * *`	No
`CLEANUP_TIMEZONE`	Cleanup schedule timezone	`UTC`	No
`CLEANUP_EXECUTION_TIMEOUT_HOURS`	Cleanup execution timeout	`1`	No
`CLEANUP_RETENTION_HOURS`	File retention period (hours)	`24`	No
`CLEANUP_DRY_RUN`	Dry-run mode (no actual deletion)	`false`	No
`LOG_LEVEL`	Application log level	`INFO`	No
`PROJECT_NAME`	Project name for metrics	`model-manager`	No
`HTTP_METRICS_PORT`	Prometheus metrics port	`9090`	No
`HTTP_SDK_METRICS_PORT`	Temporal SDK metrics port	`9091`	No
`POD_ID`	Kubernetes pod identifier	`None`	No
`EXTRA_PIP_REQUIREMENTS`	Extra pip requirements for MLFlow model serving	`None`	No

Workflow Activity Timeouts

These timeouts control how long each activity in workflows can run before timing out. All values are in seconds.

Training Workflow Timeouts:

Variable	Description	Default	Calculation Basis
`TIMEOUT_VALIDATE_PARAMS`	Parameter validation timeout	`30`	Fast operation, no I/O
`TIMEOUT_TRAIN_MODEL`	Model training timeout	`2700`	Large dataset processing (45 min)
`TIMEOUT_DELETE_FILE`	Delete file from MinIO timeout	`120`	MinIO delete operation (2 min)
`TIMEOUT_UPDATE_DATABASE`	Database update timeout	`30`	PostgreSQL update query (30 sec)

Cleanup Workflow Timeouts:

Variable	Description	Default	Calculation Basis
`TIMEOUT_CLEANUP_LOCAL`	Local cleanup timeout	`120`	Scan and delete directories (2 min)

Note: These timeouts can be adjusted based on your infrastructure performance and file sizes. If you're processing files larger than 200MB or have slower network/compute resources, increase these values accordingly.

Development

Code Quality & Testing

The project maintains 100% code coverage with comprehensive unit and integration tests.

Running Tests

# Run all tests with coverage
pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=xml --cov-report=html

# Run specific test file
pytest tests/activities/test_training.py -v

# Run with coverage visualization
pytest tests/ --cov=model_manager --cov-report=xml
# Then open htmlcov/index.html in browser

Validation Script

The validate.sh script runs all quality checks before commit:

./validate.sh

This script performs:

✅ Code Formatting (Ruff)
✅ Code Linting (Ruff)
✅ Type Checking (mypy)
✅ Security Analysis (Bandit)
✅ Unit Tests (pytest with 80%+ coverage requirement)

Coverage Visualization

For real-time coverage feedback in VS Code/Windsurf:

Install Coverage Gutters extension

Configure .vscode/settings.json:

{
  "coverage-gutters.coverageBaseDir": "${workspaceFolder}",
  "coverage-gutters.coverageFileNames": ["coverage.xml"],
  "coverage-gutters.showLineCoverage": true,
  "coverage-gutters.showRulerCoverage": true
}

Run tests to generate coverage:

pytest tests/ --cov=model_manager --cov-report=xml

Activate Coverage Gutters: Press Ctrl+Shift+7 (or Cmd+Shift+7 on Mac)

Automated Versioning

The project uses semantic versioning based on branch patterns:

Branch Pattern	Version Change	Example
`release/*`	Major version bump	`2.0.0`
`feature/*`	Minor version bump	`1.2.0`
`fix/*`	Patch version bump	`1.1.3`
`rc/*`	Release candidate	`1.1.2-rc2`

Version is calculated automatically in the CI/CD pipeline and passed to SonarQube.

Project Structure

sientia-dataops-model-manager/
├── model_manager/           # Main application package
│   ├── activities/          # Temporal activity implementations
│   │   ├── __init__.py
│   │   ├── activities.py   # Main activities orchestrator (combines all activities)
│   │   ├── experiment_tracking.py # Experiment status tracking and database operations
│   │   ├── training.py     # ML model training operations (includes MLFlow & MinIO)
│   │   └── cleanup.py      # File and directory cleanup operations
│   ├── workflows/           # Temporal workflow definitions
│   │   ├── __init__.py
│   │   ├── train_model.py  # Complete ML model training workflow
│   │   └── cleanup_files.py # Automated file cleanup workflow
│   ├── schedules/           # Temporal schedule configurations
│   │   ├── __init__.py
│   │   └── cleanup_schedule.py # Cleanup schedule creation and management
│   ├── worker/              # Worker implementation
│   │   ├── __init__.py
│   │   └── worker.py       # Main worker orchestrator (Temporal client setup)
│   ├── utils/               # Utility functions and helpers
│   │   ├── __init__.py
│   │   ├── connectors_config.py # Environment-based configuration builders
│   │   ├── exceptions.py   # Custom exception definitions
│   │   ├── logger_helper.py # Logger initialization utilities
│   │   ├── models/         # Data models and schemas
│   │   │   ├── __init__.py
│   │   │   ├── train_model_params.py # Training parameters model
│   │   │   ├── train_model_result.py # Training result model
│   │   │   └── experiment_status.py  # Experiment status enum
│   │   └── repository/     # Data access layer
│   │       ├── training_repository.py # Training business logic
│   │       ├── model_repository.py    # MLFlow artifact management
│   │       └── storage_repository.py  # MinIO storage operations
│   ├── sientia/             # Sientia-specific implementations
│   │   ├── __init__.py
│   │   ├── exceptions.py   # Custom exceptions
│   │   ├── metrics.py      # Business metrics (includes RCE drift detection)
│   │   ├── models.py       # ML model implementations (Linear & Polynomial Regression)
│   │   ├── model_serving.py # Model serving utilities
│   │   ├── reports.py      # Report generation
│   │   └── utils.py        # Utility functions
│   ├── reports/             # Report templates and temporary files
│   │   ├── header.html     # HTML report header template
│   │   └── temp/           # Temporary report files (cleaned up automatically)
│   ├── metrics.py           # Prometheus metrics definitions
│   └── __init__.py
├── scripts/                 # Test and utility scripts
│   ├── run_cleanup_test.py  # Manual cleanup workflow test
│   └── run_training_test.py # Training test with scenario support (--all for batch)
├── tests/                   # Test suite
│   ├── activities/          # Activity tests
│   ├── workflows/           # Workflow tests
│   ├── utils/               # Utility tests
│   ├── worker/              # Worker tests
│   ├── schedules/           # Schedule tests
│   └── sientia/             # Sientia module tests
├── docs/                    # Documentation and test data
│   ├── test-scenarios/      # JSON test scenario files for integration tests
│   └── test-model-data.csv  # Sample CSV data for testing
├── .github/workflows/       # CI/CD workflows
│   ├── quality-gate.yml     # PR quality checks
│   └── deploy.yml           # Deployment workflow
├── Dockerfile               # Container image definition
├── values.yaml              # Helm chart values
├── pyproject.toml           # Project configuration
├── requirements.txt         # Production dependencies
├── requirements-dev.txt     # Development dependencies
├── validate.sh              # Code quality validation script
├── run_local.sh             # Local execution script
└── README.md                # This file

Adding New Features

Follow Temporal patterns for new workflows and activities
Add comprehensive docstrings for all public methods
Include Prometheus metrics for monitoring
Add unit tests for new functionality (maintain 80%+ coverage)
Run validation script (./validate.sh) before committing
Update this README with new features and configuration

Test Coverage Guidelines

Minimum coverage: 80% (enforced by CI/CD)
Current coverage: 100% 🎯
Test all branches: Use Coverage Gutters to identify uncovered lines
Mock external dependencies: Use unittest.mock for external services
Async testing: Use pytest-asyncio for async activities and workflows

Test structure:

tests/
├── activities/      # Activity tests
├── workflows/       # Workflow tests
├── utils/           # Utility tests
├── worker/          # Worker tests
├── schedules/       # Schedule tests
└── sientia/         # Sientia module tests

Troubleshooting

Common Issues

Temporal Connection Failures
- Verify Temporal server is running and accessible
- Check namespace configuration and permissions
- Review server logs for connection issues
MLFlow Connection Issues
- Verify MLFlow server is running and accessible
- Check authentication credentials and permissions
- Ensure model names and versions exist
Database Connection Issues
- Verify PostgreSQL service is running
- Check connection credentials and network access
- Ensure proper connection pool configuration
Workflow Execution Failures
- Review activity error logs and notifications
- Check training parameter validation errors
- Verify input data format and required fields

Debug Mode

Enable debug logging by setting the log level:

export LOG_LEVEL=DEBUG

Performance Tuning

Key Parameters

Worker Concurrency: Adjust max_concurrent_workflow_tasks and max_concurrent_activities
- Training workflows: 10 concurrent tasks/activities
- Cleanup workflows: 20 concurrent tasks/activities
Connection Pools: Optimize database connection pool sizes
Model Retention: Configure MLFlow model retention based on requirements
Batch Sizes: Adjust data processing batch sizes for optimal throughput

Scaling Considerations

Horizontal Scaling: Deploy multiple worker instances
Task Queue Distribution: Two dedicated task queues for workflow isolation
- train_model-queue: Training workflows
- cleanup-queue: Cleanup workflows
Database Performance: Optimize indexes and connection pooling
MLFlow Performance: Configure appropriate model serving resources
Storage Management: Adjust cleanup retention period based on storage capacity and costs

Contributing

Fork the repository
Create a feature branch
Make your changes with comprehensive testing
Update documentation and docstrings
Submit a pull request

Code Quality Standards

Follow PEP 8 style guidelines
Include comprehensive docstrings for all public methods
Maintain test coverage above 80%
Use type hints where appropriate
Follow Temporal.io best practices

License

This project is licensed under the terms specified in the LICENSE file.

Support

For support and questions:

Check the troubleshooting section above
Review the metrics and logs for error patterns
Open an issue in the project repository
Contact the development team

Local GitHub Actions Testing (act)

What is act?

act is a tool that allows you to run GitHub Actions locally using Docker. This is useful for:

Testing workflows before pushing to the repository
Debugging issues in workflows without creating commits
Speeding up development by avoiding push/wait/check cycles
Saving GitHub Actions minutes during development

Installation

Prerequisites

Docker installed and running
Go (for installation via go install)

Installation Steps

# 1. Update packages
sudo apt-get update

# 2. Install Go (if not already installed)
sudo apt-get install golang

# 3. Install act
go install github.com/nektos/act@latest

# 4. Add Go bin to PATH
echo 'export PATH="$PATH:$HOME/go/bin"' >> ~/.bashrc
source ~/.bashrc

# 5. Verify installation
act --version

On first run, act will ask which Docker image to use:

Large (~17GB): Full image, compatible with almost all actions
Medium (~500MB): Balanced image, compatible with most actions ✅ Recommended
Micro (<200MB): Minimal image, Node.js only

Configuration

`.secrets` File

Create a .secrets file in the project root to store tokens and credentials:

# .secrets
GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxx
SONAR_TOKEN=sqp_xxxxxxxxxxxxxxxxxxxx
SONAR_HOST_URL=https://sonarqube.example.com
CI_DEPS_APP_ID=123456
CI_DEPS_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"

⚠️ Important: The .secrets file is already in .gitignore. Never commit this file!

`.event.json` File

Create a .event.json file to simulate GitHub events (e.g., pull request):

{
  "pull_request": {
    "head": {
      "ref": "feature/my-feature"
    },
    "number": 1
  }
}

⚠️ Important: The .event.json file is already in .gitignore. Never commit this file!

Usage

List Available Jobs

act -l

This command lists all available workflows and jobs in the repository.

Run Quality Gate Locally

act pull_request -j quality-gate \
  --secret-file .secrets \
  --env SONAR_SCANNER_OPTS="-Dsonar.ci.autoconfig.disabled=true" \
  --eventpath .event.json

Run Deploy Workflow with Local Repository

If you have workflows that reference external repositories (e.g., reusable workflows), you can map them locally:

act pull_request \
  -e .event.json \
  --secret-file .secrets \
  -W .github/workflows/deploy.yml \
  --local-repository Aignosi/github_workflow_templates=/path/to/local/github_workflow_templates \
  --container-daemon-socket /var/run/docker.sock \
  --container-options "--user $(id -u):$(id -g)"

Docker Socket Permissions

If you encounter permission issues with Docker socket:

# Grant temporary access to Docker socket
sudo chmod 666 /var/run/docker.sock

# Fix file ownership after running act (if needed)
sudo chown -R $USER:$USER /path/to/project

Command Reference

Command Parameters

Parameter	Description
`pull_request`	Event type to simulate (can be `push`, `pull_request`, `workflow_dispatch`, etc.)
`-j quality-gate`	Specific job name to execute (use `act -l` to see available jobs)
`--secret-file .secrets`	File containing secrets (tokens, credentials)
`--env VAR=value`	Sets environment variables for execution
`--eventpath .event.json`	JSON file with the simulated event payload

Special Parameter: `SONAR_SCANNER_OPTS`

--env SONAR_SCANNER_OPTS="-Dsonar.ci.autoconfig.disabled=true"

This parameter is required because SonarQube tries to automatically detect the CI environment. When running locally with act, the complete GitHub Actions context is not available, causing errors. The -Dsonar.ci.autoconfig.disabled=true flag disables this automatic detection.

Other Useful Commands

# List all jobs
act -l

# Run with verbose output
act pull_request -j quality-gate --secret-file .secrets -v

# Run a push event
act push -j build --secret-file .secrets

# Use a specific Docker image
act -P ubuntu-latest=catthehacker/ubuntu:act-latest

# Dry-run (doesn't execute, only shows what would be done)
act -n

Troubleshooting

Problem	Solution
`SyntaxError: Unexpected end of JSON input`	Check if `.event.json` is properly formatted
`NullPointerException` in SonarQube	Add `--env SONAR_SCANNER_OPTS="-Dsonar.ci.autoconfig.disabled=true"`
`Not Found` when accessing GitHub API	Check if `GITHUB_TOKEN` in `.secrets` is valid
Job not found	Use `act -l` to see the correct job names

Docker

Create image

$ docker build --ssh default --no-cache --progress=plain -t aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0 .

Create container

$ docker run --env-file .env --network="host" --name sientia-dataops-model-manager -d aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0

$ docker logs -f sientia-dataops-model-manager

$ docker login -u <username> -p <access-token> aignosi.azurecr.io

Push image to repository

$ docker push aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0

Helm Chart

Reference

https://aignosi-wiki.atlassian.net/wiki/spaces/IT1/pages/274563074/Como+utilizar+o+Helm+Repo+Privado

Add Helm Chart repository

$ helm repo add sientia \
    https://raw.githubusercontent.com/Aignosi/sientia-dataops-helm-repo/refs/heads/main/ \
    --username $GITHUB_USER \
    --password $GITHUB_PASS

# Update repository
$ helm repo update

# List repositories
$ helm repo list

# List versions of a specific chart
$ helm search repo sientia --versions

# List all charts available
$ helm search repo sientia

# List chart details
$ helm show all sientia/sientia-module

# Download chart to current directory
$ helm pull sientia/sientia-module --version 0.6.0 --untar

# Remove chart directory
$ rm -rf sientia-module

Helm Install

$ helm upgrade --install sientia-dataops-model-manager sientia/sientia-module -n sientia --create-namespace -f ./values.yaml --version 0.6.0

Uninstall Helm Chart

$ helm uninstall sientia-dataops-model-manager -n sientia

Note: The Model Manager system is designed for production use in industrial ML environments. Ensure proper security configuration and network isolation for production deployments.

README.md

Sientia DataOps Model Manager

📑 Table of Contents

Features

Core Functionality

Advanced Capabilities

Development & Quality Assurance

Architecture

Architecture Principles

1. Separation of Concerns

2. Fault Tolerance & Resilience

3. Scalability & Performance

4. Observability & Monitoring

Key Components

Worker (model_manager/worker/worker.py)

Workflows (model_manager/workflows/)

Activities (model_manager/activities/)

Data Services (model_manager/utils/)

Data Flow Architecture

Model Training Pipeline

Security Architecture

Authentication & Authorization

Network Security

Data Security

Workflows

Train Model Workflow (train_model.py)

Purpose

Execution Flow

Key Features

Input Parameters

Architecture Diagram

Retry Strategies

Business Validation Rules

Cleanup Files Workflow (cleanup_files.py)

Purpose

Execution Flow

Key Features

Input Parameters

Schedule Configuration

Architecture Diagram

Retry Strategies

Cleanup Patterns

Installation & Setup

Prerequisites

MinIO Setup

Environment Setup

Temporal Namespace Setup

Why Create a Namespace?

When to Create?

How to Create the Namespace

Namespace Configuration

Verification

Troubleshooting

Local Development Setup

How to Run

Running the Model Manager Application

Running Tests and Coverage

Manual Test Execution

Manual Application Execution

Code Quality & Validation

Overview

Validation Tools

1. Ruff - Linting and Formatting ⚡

2. mypy - Type Checking 🏷️

3. Bandit - Security Analysis 🔒

4. pytest - Automated Testing 🧪

Tools Installation

Complete Validation

Option 1: Automated Script (Recommended)

Option 2: Individual Commands

Automatic Fixes

Configuration

CI/CD Integration

Quality Gate (.github/workflows/quality-gate.yml)

Deploy (.github/workflows/deploy.yml)

Best Practices

Testing

Test Structure

Test Execution

Integration Tests

Worker (`model_manager/worker/worker.py`)

Workflows (`model_manager/workflows/`)

Activities (`model_manager/activities/`)

Data Services (`model_manager/utils/`)

Train Model Workflow (`train_model.py`)

Cleanup Files Workflow (`cleanup_files.py`)

Quality Gate (`.github/workflows/quality-gate.yml`)

Deploy (`.github/workflows/deploy.yml`)

`.secrets` File

`.event.json` File

Special Parameter: `SONAR_SCANNER_OPTS`