gitea_admin/sientia-dataops-model-manager
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
43cbc31e10d53bbd30da89e10e4c401be501ff77
sientia-dataops-model-manager/README.md
Bruno Domingues 06571011f2 SIENTIAPDE-1430: Introduce static_threshold parameter for static window removal. 
This parameter allows customizing the threshold (1-1000) used when rem_static_win is enabled, defaulting to 1 if null.
Updates include parameter definition, business rule validation, repository logic for passing the threshold, documentation in README.md and PIPELINE_PARAMS_CHANGELOG.md, and new unit and integration tests.
2025-12-19 15:44:18 -03:00

1633 lines
60 KiB
Markdown
Raw Blame History

# Sientia DataOps Model Manager
An enterprise-grade ML model training orchestration platform built on Temporal. Provides robust, scalable workflows for training machine learning models with comprehensive validation, experiment tracking, and automated resource management. Integrates seamlessly with MLFlow for model persistence and PostgreSQL for experiment tracking.
## 📑 Table of Contents
- [Features](#features)
- [Core Functionality](#core-functionality)
- [Advanced Capabilities](#advanced-capabilities)
- [Development & Quality Assurance](#development--quality-assurance)
- [Architecture](#architecture)
- [Architecture Principles](#architecture-principles)
- [Key Components](#key-components)
- [Data Flow Architecture](#data-flow-architecture)
- [Security Architecture](#security-architecture)
- [Workflows](#workflows)
- [Train Model Workflow](#train-model-workflow-train_modelpy)
- [Cleanup Files Workflow](#cleanup-files-workflow-cleanup_filespy)
- [Installation & Setup](#installation--setup)
- [Prerequisites](#prerequisites)
- [Environment Setup](#environment-setup)
- [Temporal Namespace Setup](#temporal-namespace-setup)
- [Local Development Setup](#local-development-setup)
- [Port Forward Setup Script](#port-forward-setup-script)
- [How to Run](#how-to-run)
- [Running the Model Manager Application](#running-the-model-manager-application)
- [Running Tests and Coverage](#running-tests-and-coverage)
- [Manual Test Execution](#manual-test-execution)
- [Manual Application Execution](#manual-application-execution)
- [Code Quality & Validation](#code-quality--validation)
- [Overview](#overview)
- [Validation Tools](#validation-tools)
- [Tools Installation](#tools-installation)
- [Complete Validation](#complete-validation)
- [Automatic Fixes](#automatic-fixes)
- [Configuration](#configuration)
- [CI/CD Integration](#cicd-integration)
- [Best Practices](#best-practices)
- [Testing](#testing)
- [Test Structure](#test-structure)
- [Test Execution](#test-execution)
- [Integration Tests](#integration-tests)
- [Running Integration Tests](#running-integration-tests)
- [Test Scenarios](#test-scenarios)
- [Scenario File Structure](#scenario-file-structure)
- [Creating New Scenarios](#creating-new-scenarios)
- [Important Validations](#important-validations)
- [Monitoring and Metrics](#monitoring-and-metrics)
- [Application Health Metrics](#application-health-metrics)
- [Training Metrics](#training-metrics)
- [Configuration](#configuration-1)
- [Environment Variables](#environment-variables)
- [Workflow Configuration](#workflow-configuration)
- [Development](#development)
- [Code Quality & Testing](#code-quality--testing)
- [Project Structure](#project-structure)
- [Adding New Features](#adding-new-features)
- [Test Coverage Guidelines](#test-coverage-guidelines)
- [Troubleshooting](#troubleshooting)
- [Common Issues](#common-issues)
- [Debug Mode](#debug-mode)
- [Performance Tuning](#performance-tuning)
- [Key Parameters](#key-parameters)
- [Scaling Considerations](#scaling-considerations)
- [Contributing](#contributing)
- [Code Quality Standards](#code-quality-standards)
- [License](#license)
- [Support](#support)
- [Local GitHub Actions Testing (act)](#local-github-actions-testing-act)
- [What is act?](#what-is-act)
- [Installation](#installation)
- [Configuration](#configuration-2)
- [Usage](#usage)
- [Command Reference](#command-reference)
- [Docker](#docker)
- [Helm Chart](#helm-chart)
## Features
### Core Functionality
- **ML Model Training Pipeline**: Complete training workflow from validation to deployment using MLFlow
- **Polynomial Regression Support**: Configurable polynomial degree with interaction terms and mandatory scaler validation
- **Automated File Cleanup**: Scheduled cleanup of stale files from MinIO and local filesystem
- **Temporal Workflow Orchestration**: Robust workflow management with granular retry policies and fault tolerance
- **Parameter Validation**: Defense-in-depth validation with business rules and type checking
- **Experiment Tracking**: Comprehensive status tracking in PostgreSQL database
- **Resource Management**: Automatic cleanup of temporary files and storage
- **Comprehensive Monitoring**: Prometheus metrics and detailed logging for operational visibility
- **Scheduled Jobs**: Automated daily cleanup with configurable cron schedules
### Advanced Capabilities
- **Granular Retry Policies**: Different strategies for network, training, MLFlow, database, and filesystem operations
- **Configurable Timeouts**: Environment variable-based timeouts supporting large training files (up to 200MB)
- **Notification System**: Integrated alerting and notification management via MongoDB
- **Scalable Architecture**: Kubernetes-ready deployment with horizontal scaling support
- **MLFlow Integration**: Seamless model and artifact persistence to MLFlow tracking server
- **Per-Variable Lag Configuration**: Flexible lag settings for each variable independently
- **Date Range Filtering**: Filter training data by start/end dates and removed intervals
- **NaN Treatment Options**: Configurable handling of missing values (drop, linear interpolation)
- **RCE Drift Metrics**: Reduced Coulomb Energy metrics for drift detection
### Development & Quality Assurance
- **Code Quality Tools**: Ruff (linting/formatting), mypy (type checking), Bandit (security analysis)
- **Automated Validation**: Pre-commit validation script (`validate.sh`) and CI/CD integration
- **Comprehensive Testing**: pytest with async support and **100% code coverage** 🎯
- **Type Safety**: Static type checking with mypy for improved code reliability
- **Coverage Visualization**: Integration with Coverage Gutters for real-time coverage feedback
- **Automated Versioning**: Semantic versioning based on branch patterns (release/*, feature/*, fix/*, rc/*)
- **Integration Test Scenarios**: JSON-based test scenarios with batch execution support
## Architecture
The Model Manager system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments.
### Architecture Principles
#### 1. **Separation of Concerns**
- **Worker Layer**: Manages Temporal workers, task queues, and application lifecycle
- **Workflow Layer**: Orchestrates business logic and process coordination
- **Activity Layer**: Implements specific operations and external system interactions
- **Data Layer**: Handles data persistence, caching, and external service connections
#### 2. **Fault Tolerance & Resilience**
- **Automatic Retry Policies**: Configurable retry strategies for transient failures
- **Circuit Breaker Pattern**: Prevents cascading failures in external service calls
- **Graceful Degradation**: System continues operating with reduced functionality
- **Comprehensive Error Handling**: Detailed error reporting and notification integration
#### 3. **Scalability & Performance**
- **Horizontal Scaling**: Multiple worker instances for load distribution
- **Task Queue Isolation**: Separate queues for different workflow types
- **Connection Pooling**: Optimized database and external service connections
- **Asynchronous Processing**: Non-blocking operations for improved throughput
#### 4. **Observability & Monitoring**
- **Prometheus Metrics**: Comprehensive system and business metrics
- **Structured Logging**: Consistent log format with correlation IDs
- **Health Checks**: Endpoint health monitoring and alerting
- **Performance Tracing**: Request flow tracking and bottleneck identification
### Key Components
#### **Worker (`model_manager/worker/worker.py`)**
- **Purpose**: Main application orchestrator managing Temporal workers and task queues
- **Responsibilities**:
- Temporal client initialization and connection management
- Worker lifecycle management and graceful shutdown
- Task queue configuration and load balancing
- Prometheus metrics server initialization
- Notification handler setup and configuration
- **Key Features**:
- Automatic scaling with `PollerBehaviorAutoscaling`
- Health check endpoints for Kubernetes liveness/readiness probes
- Graceful shutdown with cleanup procedures
- Multi-instance deployment support
- Two dedicated task queues:
- `train_model-queue`: ML model training workflows
- `cleanup-queue`: File cleanup workflows
- Automated cleanup schedule management
#### **Workflows (`model_manager/workflows/`)**
- **TrainModel**: Complete ML model training pipeline from validation to deployment
- **CleanupFiles**: Automated cleanup of stale files from MinIO and local filesystem
- **Key Features**:
- Temporal workflow definitions with granular retry policies
- Parameter validation with business rules
- Comprehensive error handling and status tracking
- Configurable timeouts for different operation types
- Automatic resource cleanup and management
- Scheduled cleanup jobs with cron expressions
#### **Activities (`model_manager/activities/`)**
- **Activities**: Main activity orchestrator combining all functionality through multiple inheritance
- **ExperimentTracking**: ML experiment lifecycle tracking and database operations
- Unified `update_experiment_run()` method for all experiment status updates
- Support for three update types: STATUS, STATUS_WITH_ERROR, MODEL_SAVED
- Automatic error message truncation (1024 chars)
- Connection pooling and retry logic
- **Training**: ML model training operations with MLFlow and MinIO integration
- Unified `train_model()` method for complete training pipeline
- Receives pre-downloaded files (BytesIO) to avoid memory leaks
- Returns success/failure status with TrainModelResult or error message
- No exception raising on failure - allows workflow to handle errors gracefully
- Integration with TrainingRepository for business logic separation
- MLFlow model saving and artifact management
- MinIO object storage operations
- **Polynomial Regression**: Support for configurable degree and interaction terms
- **Training Predictions**: Calculates y_train_pred before denormalization for accurate metrics
- **Cleanup**: File and directory cleanup operations
- `cleanup_minio_files()`: Removes stale files from MinIO based on timestamp prefixes
- `cleanup_temp_directories()`: Cleans local temporary directories
- Configurable retention period (default: 24 hours)
- Dry-run mode for testing
- **Key Features**:
- Multiple inheritance pattern for unified activity interface
- Parameter validation with business rules
- MLFlow integration for model persistence
- Comprehensive error handling and notification integration
- Experiment tracking with automatic status management
- Timestamp-based file cleanup with regex pattern matching
#### **Data Services (`model_manager/utils/`)**
- **Connectors Config**: Environment variable-based configuration management
- **Repository**: Data access layer for training and MLFlow operations
- `training_repository.py`: Training business logic and operations
- `model_repository.py`: MLFlow artifact generation and model persistence
- **Models**: Data models and schemas
- `train_model_params.py`: Training parameters model with comprehensive validation
- `train_model_result.py`: Training result model (includes y_train_pred)
- `experiment_status.py`: Experiment status enum
- **Key Features**:
- Environment variable-based configuration with sensible defaults
- Connection pool management and optimization
- Security credential management
- Configuration validation and error handling
- Type-safe data models with validation
### Data Flow Architecture
#### **Model Training Pipeline**
```
Training Request → Parameter Validation → MinIO Data Download →
Model Training → MLFlow Model Save → Resource Cleanup → Status Update
```
**Key Stages:**
1. **Validation**: Experiment run ID and training parameters validation
2. **Data Acquisition**: Download training data from MinIO storage
3. **Training**: Execute ML model training with validated parameters
4. **Persistence**: Save trained model and artifacts to MLFlow
5. **Cleanup**: Remove temporary files and update experiment status
### Security Architecture
#### **Authentication & Authorization**
- **MLFlow API Authentication**: Username/password with secure transmission
- **Database Connection Security**: Encrypted connections with credential management
- **Kubernetes Secrets Integration**: Secure credential storage and access
#### **Network Security**
- **TLS/SSL Encryption**: Secure communication channels
- **Network Isolation**: Kubernetes network policies and service mesh
- **Firewall Rules**: Controlled access to external services
- **VPN Integration**: Secure remote access and management
#### **Data Security**
- **Data Encryption**: At-rest and in-transit encryption
- **Access Control**: Role-based access control (RBAC)
- **Audit Logging**: Comprehensive access and operation logging
- **Data Retention**: Configurable data lifecycle management
## Workflows
### Train Model Workflow (`train_model.py`)
The **TrainModel** workflow orchestrates the complete ML model training pipeline from parameter validation through model saving and cleanup.
#### Purpose
- **Model Training**: Complete ML model training pipeline
- **Parameter Validation**: Defense-in-depth validation with business rules
- **Resource Management**: Automatic cleanup of temporary resources
- **Status Tracking**: Comprehensive experiment tracking in database
- **Error Handling**: Robust error handling with detailed context logging
#### Execution Flow
1. **Validate Experiment Run ID**: Critical validation before any DB updates
2. **Validate Training Parameters**: Type checking + business rules validation
3. **Download Training Data**: Fetch file from MinIO storage
4. **Train Model**: Execute ML model training with validated parameters
5. **Save to MLFlow**: Save trained model and artifacts to MLFlow
6. **Cleanup Resources**: Delete temporary files and MinIO data
#### Key Features
- **Granular Retry Policies**: Different strategies for network, training, MLFlow, database, and filesystem operations
- **Configurable Timeouts**: Environment variable-based timeouts supporting files up to 200MB
- **Idempotent Cleanup**: Safe replay with Temporal workflow replay mechanism
- **Structured Logging**: Rich context in error messages for debugging
- **Business Validation**: 10 business rules including range checks, consistency validation, and data integrity
#### Input Parameters
```json
{
"experiment_run_id": 123,
"target_variable": "price",
"variable_columns": ["feature1", "feature2"],
"train_size": 80,
"shuffle": true,
"use_scaler": true,
"include_ar": false,
"bucket_name": "ml-data",
"file_name": "training_data.csv",
"line_separator": "\n",
"decimal_separator": ".",
"lag_train": {"feature1": 0, "feature2": 2},
"lag_val": {"feature1": 0, "feature2": 1},
"rem_static_win": false,
"static_threshold": null,
"low_lim": {"feature1": 0.0, "feature2": 0.0},
"upp_lim": {"feature1": 100.0, "feature2": 100.0},
"window": 10,
"experiment_name": "production_model_v1",
"removed_intervals": [],
"model_name": "Linear Regression",
"degree": 1,
"interaction_only": false,
"nan_treatment": "drop",
"start_date": null,
"end_date": null,
"scaler_name": "Standard Scaler",
"support_filters": {}
}
```
#### Architecture Diagram
```mermaid
flowchart TD
A[1. validate_experiment_run_id] --> B[2. validate_train_params]
B --> C[3. fetch_file_from_minio]
C --> D[4. train_model]
D --> E[5. save_model]
E --> F[6. cleanup_run_directory]
F --> G[7. delete_file_from_minio]
B -.-> DB[(PostgreSQL)]
C -.-> MinIO[MinIO Storage]
D -.-> Training[ML Training]
E -.-> MLFlow[MLFlow]
F -.-> FS[Filesystem]
G -.-> MinIO
```
#### Retry Strategies
The workflow implements 5 different retry policies optimized for each operation type:
| Operation Type | Initial Interval | Max Interval | Backoff | Max Attempts | Use Case |
|---------------|------------------|--------------|---------|--------------|----------|
| **Network** | 1s | 10s | 2.0x | 5 | MinIO operations (transient network errors) |
| **No Retry** | - | - | - | 1 | Training/Validation (permanent data errors) |
| **MLFlow** | 5s | 30s | 2.0x | 3 | MLFlow operations (API timeouts) |
| **Database** | 2s | 20s | 2.0x | 5 | PostgreSQL updates (lock contention) |
| **Filesystem** | 2s | 10s | 1.5x | 3 | Cleanup operations (busy resources) |
#### Business Validation Rules
The workflow validates comprehensive business rules beyond type checking:
1. **train_size**: Must be between 10-100%
2. **variable_columns**: Cannot be empty
3. **lag_train, lag_val**: Per-variable dictionaries with non-negative values
4. **window**: Must be non-negative integer
5. **low_lim/upp_lim**: Must have same keys and low < upp for each variable
6. **target_variable**: Cannot be empty
7. **bucket_name, file_name, experiment_name**: Cannot be empty or whitespace
8. **degree**: Must be at least 1; must be >= 2 for Polynomial Regression
9. **nan_treatment**: Must be one of 'drop', 'linear interpolation', 'fill linear'
10. **scaler_name**: Must be 'Standard Scaler' or 'None'
11. **model_name**: Must be 'Linear Regression' or 'Polynomial Regression'
12. **Polynomial Regression requires Scaler**: Models with degree > 1 must have a scaler to prevent numerical overflow
13. **Linear Regression requires degree 1**: Linear models must have degree = 1
14. **static_threshold**: When `rem_static_win` is true and `static_threshold` has a value, it must be between 1 and 1000 (inclusive). If null, defaults to 1.
### Cleanup Files Workflow (`cleanup_files.py`)
The **CleanupFiles** workflow provides automated cleanup of stale files from MinIO storage and local temporary directories. It runs on a scheduled basis (default: daily at midnight UTC) to maintain storage hygiene.
#### Purpose
- **Storage Management**: Automatic removal of old files from MinIO and local filesystem
- **Retention Policy**: Configurable retention period (default: 24 hours)
- **Scheduled Execution**: Cron-based scheduling for automated cleanup
- **Resource Optimization**: Prevents storage bloat and reduces costs
#### Execution Flow
1. **Cleanup MinIO Files**: Scan and delete files older than retention period from MinIO bucket
2. **Cleanup Local Directories**: Remove temporary directories older than retention period
#### Key Features
- **Timestamp-Based Cleanup**: Uses filename/directory timestamps for age determination
- **Pattern Matching**: Regex patterns for MinIO (`timestamp-filename`) and directories (`name_YYYYMMDD_HHMMSS_microseconds`)
- **Configurable Retention**: Environment variable-based retention period
- **Dry-Run Mode**: Test cleanup operations without actual deletion
- **Idempotent**: Safe to run multiple times
- **Error Handling**: Continues cleanup even if individual operations fail
#### Input Parameters
```json
{
"bucket_name": "model-training" // Optional, defaults to DEFAULT_CLEANUP_BUCKET env var
}
```
#### Schedule Configuration
The cleanup schedule is automatically created when the worker starts:
| Configuration | Environment Variable | Default | Description |
|--------------|---------------------|---------|-------------|
| **Schedule ID** | `CLEANUP_SCHEDULE_ID` | `cleanup-files-daily` | Unique identifier for the schedule |
| **Cron Expression** | `CLEANUP_CRON` | `0 0 * * *` | Daily at midnight UTC |
| **Timezone** | `CLEANUP_TIMEZONE` | `UTC` | Timezone for cron execution |
| **Task Queue** | `CLEANUP_TASK_QUEUE` | `cleanup-queue` | Dedicated task queue |
| **Execution Timeout** | `CLEANUP_EXECUTION_TIMEOUT_HOURS` | `1` | Maximum execution time (hours) |
| **Retention Period** | `CLEANUP_RETENTION_HOURS` | `24` | Files older than this are deleted |
| **Dry Run** | `CLEANUP_DRY_RUN` | `false` | Test mode without actual deletion |
| **Max Keys** | `MAX_KEYS_CLEANUP` | `1000` | MinIO list operation page size |
#### Architecture Diagram
```mermaid
flowchart TD
A[Scheduled Trigger] --> B[cleanup_minio_files]
B --> C[cleanup_temp_directories]
B -.-> MinIO[MinIO Storage]
C -.-> FS[Local Filesystem]
```
#### Retry Strategies
| Operation Type | Initial Interval | Max Interval | Backoff | Max Attempts | Use Case |
|---------------|------------------|--------------|---------|--------------|----------|
| **Network** | 1s | 10s | 2.0x | 5 | MinIO operations (transient network errors) |
| **No Retry** | - | - | - | 1 | Local filesystem operations (permanent errors) |
#### Cleanup Patterns
**MinIO Files:**
- Pattern: `{timestamp}-{filename}` where timestamp is milliseconds since epoch
- Example: `1638360000000-training_data.csv`
- Retention: Files older than `CLEANUP_RETENTION_HOURS` are deleted
**Local Directories:**
- Pattern: `{name}_{YYYYMMDD}_{HHMMSS}_{microseconds}`
- Example: `temp_20231201_143052_123456`
- Retention: Directories older than `CLEANUP_RETENTION_HOURS` are deleted
## Installation & Setup
### Prerequisites
- Python 3.11+
- Temporal server/cluster
- PostgreSQL database
- MLFlow server
- MinIO object storage (for MLFlow artifacts)
- MongoDB server (for notifications)
**Note**: External dependencies must be available either through:
- Kubernetes cluster deployment
- Docker Compose setup
- Cloud-managed services
- Local installations
#### MinIO Setup
MinIO is required for MLFlow artifact storage. For detailed installation and configuration instructions, refer to:
📚 **[Install MinIO via Helm Chart on K8s](https://aignosi-wiki.atlassian.net/wiki/spaces/IT1/pages/225214465/Install+Minio+via+Helm+Chart+on+K8s)**
This guide covers:
- Helm chart installation on Kubernetes
- Storage configuration and persistence
- Access credentials setup
- Integration with MLFlow
### Environment Setup
1. **Clone the repository**:
```bash
git clone <repository-url>
cd sientia-dataops-model-manager
```
2. **Create virtual environment**:
```bash
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
```
3. **Install dependencies**:
```bash
pip install -r requirements.txt
```
4. **Configure environment variables** (see [Configuration](#-configuration) section)
5. **Run validation script**:
```bash
./validate.sh
```
### Temporal Namespace Setup
The Model Manager requires a dedicated Temporal namespace to isolate workflows and maintain proper execution history. The namespace must be created **before** starting the application.
#### Why Create a Namespace?
- **Isolation**: Separates Model Manager workflows from other applications
- **Retention Control**: Configures workflow history retention (default: 7 days)
- **Multi-tenancy**: Enables multiple environments (dev, staging, prod) on same cluster
- **Security**: Allows namespace-level access control and permissions
#### When to Create?
- ✅ **Before first deployment** in any environment
- ✅ **Once per environment** (dev, staging, production)
- ✅ **After Temporal cluster setup** or upgrade
#### How to Create the Namespace
**Option 1: Using Temporal Admin Tools Pod (Recommended for Kubernetes)**
```bash
# 1. List Temporal pods
kubectl get pods -n temporal
# 2. Connect to admin tools pod
kubectl exec -it -n temporal <temporal-admin-tools-pod-name> -- bash
# 3. Create namespace
tctl --namespace model-manager namespace register \
--retention 7 \
--description "Model Manager - ML Model Orchestration Namespace"
# 4. Verify creation
tctl --namespace model-manager namespace describe
# 5. Exit pod
exit
```
**Option 2: Using Port Forward (Local Development)**
```bash
# 1. Port forward Temporal frontend
kubectl port-forward -n temporal svc/temporal-frontend 7233:7233
# 2. In another terminal, create namespace
tctl --address localhost:7233 \
--namespace model-manager \
namespace register \
--retention 7 \
--description "Model Manager - ML Model Orchestration Namespace"
# 3. Verify
tctl --address localhost:7233 --namespace model-manager namespace describe
```
**Option 3: Direct kubectl exec (One-liner)**
```bash
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
tctl --namespace model-manager namespace register \
--retention 7 \
--description "Model Manager - ML Model Orchestration Namespace"
```
#### Namespace Configuration
| Parameter | Value | Description |
|-----------|-------|-------------|
| **Name** | `model-manager` | Namespace identifier (configurable via `TEMPORAL_NAMESPACE` env var) |
| **Retention** | `7 days` | Workflow history retention period |
| **Description** | `Model Manager - ML Model Orchestration Namespace` | Human-readable description |
#### Verification
To verify the namespace was created successfully:
```bash
# List all namespaces
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- tctl namespace list
# Describe specific namespace
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
tctl --namespace model-manager namespace describe
```
#### Troubleshooting
**Error: "namespace already exists"**
- ✅ This is fine! The namespace is already configured
- No action needed, proceed with application deployment
**Error: "connection refused"**
- ❌ Temporal server is not accessible
- Verify Temporal cluster is running: `kubectl get pods -n temporal`
- Check network connectivity and port forwarding
**Error: "permission denied"**
- ❌ Insufficient permissions to create namespace
- Contact cluster administrator for namespace creation
- Or request elevated permissions for your service account
### Local Development Setup
1. **Clone the repository**
```bash
git clone <repository-url>
cd sientia-dataops-model-manager
```
2. **Create virtual environment**
```bash
conda create -p ./venv python=3.11
conda activate ./venv
```
3. **Install dependencies**
1. **Install github cli**
```bash
sudo apt update
sudo apt install gh -y
```
2. **Authenticate with github**
```bash
gh auth login
```
3. **Install Python dependencies**
```bash
python -m pip install --upgrade pip
# Install production dependencies
pip install -r requirements.txt
# Install development and testing tools
pip install -r requirements-dev.txt
```
4. **Create environment configuration file**
```bash
cp .env.example .env
# Edit .env with your connection details
```
5. **Configure external dependencies**
The Model Manager requires connections to several external services. For local development, you can use the provided port-forward script to establish connections to services running in your Kubernetes cluster.
**Services and Ports:**
| Local Port | Service | Description | Namespace |
|------------|---------|-------------|-----------|
| `5432` | `paradedb-rw` | PostgreSQL Database | `paradedb` |
| `45249` | `sientia-tracker-mlflow-tracking` | MLflow Tracking Server | `sientia-tracker` |
| `37463` | `temporal-frontend` | Temporal gRPC API | `temporal` |
| `8080` | `temporal-web` | Temporal Web UI | `temporal` |
| `42297` | `my-release-mongodb` | MongoDB Database | `mongodb` |
| `36577` | `minio` | MinIO Object Storage | `minio` |
**Managing Port Forwards:**
```bash
# View active port forwards
jobs -l
# Stop all port forwards
jobs -p | xargs kill
```
**Manual Port Forwarding:**
If you prefer manual control or need different ports:
```bash
# PostgreSQL
kubectl -n paradedb port-forward svc/paradedb-rw 5432:5432 &
# MLflow
kubectl -n sientia-tracker port-forward svc/sientia-tracker-mlflow-tracking 45249:80 &
# Temporal
kubectl -n temporal port-forward svc/temporal-frontend 37463:7233 &
# Temporal UI
kubectl -n temporal port-forward svc/temporal-web 8080:8080 &
# MongoDB
kubectl -n mongodb port-forward svc/my-release-mongodb 42297:27017 &
# MinIO
kubectl -n minio port-forward svc/minio 36577:9000 &
```
## How to Run
### Running the Model Manager Application
Use the provided script to run the application locally:
```bash
# Make script executable (first time only)
chmod +x run_local.sh
# Run the application
./run_local.sh
```
The script will:
- Load environment variables from `.env`
- Start the model-manager worker application
**Note**: Activate your virtual environment before running the script:
```bash
source ./venv/bin/activate # or: conda activate ./venv
./run_local.sh
```
### Running Tests and Coverage
Run tests with coverage using pytest directly:
```bash
# Run all tests with coverage
pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=html
# Open the coverage report in your browser
open htmlcov/index.html # macOS
xdg-open htmlcov/index.html # Linux
```
### Manual Test Execution
You can also run tests manually:
```bash
# Activate virtual environment
source ./venv/bin/activate
# Run all tests
pytest
# Run with coverage
pytest --cov=model_manager --cov-report=html
# Run specific test categories
pytest tests/activities/
pytest tests/workflow/
```
### Manual Application Execution
For manual execution without scripts:
```bash
# Activate virtual environment
source ./venv/bin/activate
# Load environment variables (if using .env file)
if [ -f .env ]; then
export $(cat .env | grep -v '^#' | xargs)
fi
# Start the model-manager worker
python -m model_manager.worker.worker
```
## Code Quality & Validation
### Overview
Since Python is not a compiled language, we use a robust set of tools to validate code quality, security, and correctness before execution. These tools detect errors, style issues, security vulnerabilities, and ensure code consistency.
### Validation Tools
#### 1. **Ruff** - Linting and Formatting ⚡
Modern and extremely fast tool (written in Rust) that replaces multiple tools:
- **Linting**: Detects code errors, style issues (PEP 8), common bugs
- **Formatting**: Automatically formats code consistently
- **Speed**: 10-100x faster than Flake8/Black
#### 2. **mypy** - Type Checking 🏷️
Static type checker that analyzes type hints:
- Detects type errors before execution
- Improves code documentation
- Prevents bugs related to incorrect types
#### 3. **Bandit** - Security Analysis 🔒
Security vulnerability scanner:
- Detects insecure code patterns
- Identifies hardcoded passwords, SQL injection, etc.
- Ensures compliance with security practices
#### 4. **pytest** - Automated Testing 🧪
Testing framework with code coverage:
- Executes unit and integration tests
- Measures code coverage
- Supports asynchronous tests
### Tools Installation
```bash
# Install development dependencies
pip install -r requirements-dev.txt
```
### Complete Validation
#### Option 1: Automated Script (Recommended)
```bash
# Run all validations at once
./validate.sh
```
The `validate.sh` script automatically executes:
1. ✅ Format checking (Ruff)
2. ✅ Code linting (Ruff)
3. ✅ Type checking (mypy)
4. ✅ Security analysis (Bandit)
5. ✅ Unit tests with coverage (pytest)
#### Option 2: Individual Commands
```bash
# 1. Check formatting
ruff format --check model_manager/ tests/
# 2. Check linting
ruff check model_manager/ tests/
# 3. Check types
mypy model_manager/
# 4. Security analysis
bandit -r model_manager/ -ll
# 5. Run tests
pytest tests/ --cov=model_manager --cov-report=term-missing
```
### Automatic Fixes
Some tools can automatically fix issues:
```bash
# Format code automatically
ruff format model_manager/ tests/
# Fix linting issues automatically
ruff check --fix model_manager/ tests/
```
### Configuration
All tools are configured in the `pyproject.toml` file:
- **Ruff**: Linting rules, formatting, complexity
- **mypy**: Type checking settings
- **pytest**: Test and coverage options
- **Bandit**: Security rules
### CI/CD Integration
The project uses **reusable workflows** from `Aignosi/github_workflow_templates` for CI/CD:
#### Quality Gate (`.github/workflows/quality-gate.yml`)
Runs automatically on each Pull Request to `main`:
- ✅ Formatting and linting block merge if they fail
- ⚠️ Type checking and security generate warnings but don't block
- ✅ Tests must pass with minimum 80% coverage
- ✅ SonarQube analysis for code quality metrics
- ✅ Automatic version calculation based on branch pattern
#### Deploy (`.github/workflows/deploy.yml`)
Runs automatically when a PR is merged to `main`:
- ✅ Builds and pushes Docker image to Azure Container Registry
- ✅ Creates GitHub release with calculated version
- ✅ Deploys to Kubernetes using Helm
### Best Practices
1. **Before Commit**: Run `./validate.sh` to ensure quality
2. **During Development**: Use `ruff check --watch` for real-time feedback
3. **Type Hints**: Add type hints to new functions for better validation
4. **Tests**: Maintain coverage above 80%
5. **Security**: Review and fix all Bandit warnings
## Testing
### Test Structure
```
tests/
├── activities/ # Activity implementation tests
├── workflows/ # Workflow orchestration tests
├── utils/ # Utility function tests
├── worker/ # Worker tests
├── schedules/ # Schedule configuration tests
└── sientia/ # Sientia module tests
```
### Test Execution
```bash
# Install test dependencies
pip install pytest pytest-cov pytest-asyncio
# Run tests with coverage
pytest --cov=model_manager --cov-report=html
# Run specific test modules
pytest tests/activities/test_training.py
pytest tests/workflows/test_train_model.py
```
### Integration Tests
The project includes integration tests that validate the complete training workflow against a running Temporal cluster. These tests use JSON-based scenario files for easy configuration and maintenance.
#### Running Integration Tests
```bash
# List available scenarios
python scripts/run_training_test.py --list
# Run a specific test scenario
python scripts/run_training_test.py --scenario 01-linear-regression-basic
# Run with custom CSV data file
python scripts/run_training_test.py --scenario 03-polynomial-regression-degree2 --csv /path/to/data.csv
# Run ALL scenarios sequentially with summary report
python scripts/run_training_test.py --all
# Run all scenarios with custom CSV
python scripts/run_training_test.py --all --csv docs/custom-data.csv
```
#### Batch Execution Output
When running all scenarios with `--all`, the script provides:
- Progress indicators for each scenario (`[1/10] Running scenario: ...`)
- Status symbols (✓ for passed, ✗ for failed)
- Final summary with total/passed/failed counts
- Detailed error messages for failed scenarios
- Exit code 0 if all pass, 1 if any fail
Example output:
```
Running 10 scenarios...
[1/10] Running scenario: 01-linear-regression-basic
Loaded scenario: 01-linear-regression-basic
Uploaded CSV to MinIO: test-model-data-20231219-120000.csv
Created experiment_run with ID: 42
Workflow started: train-model-test-abc123
[1/10] ✓ 01-linear-regression-basic
...
============================================================
SUMMARY
============================================================
Total: 10 | Passed: 9 | Failed: 1
============================================================
✓ PASSED:
- 01-linear-regression-basic
- 02-linear-regression-with-scaler
...
✗ FAILED:
- 05-linear-regression-with-lags
Error: Failed to start Temporal workflow: connection refused
```
#### Test Scenarios
Test scenarios are defined as JSON files in `docs/test-scenarios/`. Each scenario configures a complete training workflow with specific parameters:
| Scenario | Description | Key Features |
|----------|-------------|--------------|
| `01-linear-regression-basic` | Basic linear regression | No scaler, no lags |
| `02-linear-regression-with-scaler` | Linear regression with normalization | Standard Scaler enabled |
| `03-polynomial-regression-degree2` | Polynomial regression (degree 2) | Requires scaler (mandatory) |
| `04-polynomial-regression-degree3` | Polynomial regression (degree 3) | Requires scaler (mandatory) |
| `05-linear-regression-with-lags` | Linear regression with lag features | Lag train/val configuration |
| `06-linear-regression-nan-interpolation` | Linear regression with NaN handling | `nanTreatment: "linear interpolation"` |
| `07-linear-regression-static-window-removal` | Linear regression with static window removal | `remStaticWin: true` |
| `08-linear-regression-with-limits` | Linear regression with variable limits | `lowLim`/`uppLim` configuration |
| `09-polynomial-degree2-with-scaler-and-lags` | Complete polynomial scenario | Scaler + lags + degree 2 |
| `10-linear-regression-with-ar` | Linear regression with autoregressive variable | `includeAr: true` |
| `11-linear-regression-static-threshold-custom` | Linear regression with custom static threshold | `staticThreshold: 100` |
#### Scenario File Structure
```json
{
"_description": "Human-readable description of the scenario",
"experimentName": "test-experiment-name",
"username": "user@example.com",
"modelName": "Linear Regression",
"targetVariable": "target_column_name",
"variableColumns": ["feature1", "feature2"],
"lagTrain": {"feature1": 0, "feature2": 0},
"lagVal": {"feature1": 0, "feature2": 0},
"remStaticWin": false,
"staticThreshold": null,
"lowLim": {},
"uppLim": {},
"window": 0,
"useScaler": false,
"includeAr": false,
"trainSize": 80,
"shuffle": true,
"lineSeparator": ",",
"decimalSeparator": ".",
"removedIntervals": [],
"degree": 1,
"interactionOnly": false,
"nanTreatment": "drop",
"startDate": null,
"endDate": null,
"scalerName": "None",
"supportFilters": {}
}
```
#### Creating New Scenarios
1. Copy an existing scenario file as a template
2. Modify parameters according to your test case
3. Save with a descriptive name: `XX-description.json`
4. Run with: `python scripts/run_training_test.py --scenario XX-description`
#### Important Validations
The training workflow enforces several business rules:
- **Polynomial Regression requires Scaler**: Models with `degree > 1` must have `useScaler: true` and a valid `scalerName` to prevent numerical overflow
- **Static Window Removal requires DatetimeIndex**: Scenarios with `remStaticWin: true` require data with a timestamp column for the `TimeSeriesDiscontinuityAnalyzer`
- **Variable Limits Consistency**: `lowLim` and `uppLim` must have matching keys, and `lowLim[key] < uppLim[key]` for all variables
## Monitoring and Metrics
The Model Manager system exposes comprehensive Prometheus metrics for operational visibility and performance monitoring:
### Application Health Metrics
- `app_up`: Application health status (1=healthy, 0=unhealthy)
- Labels: `pod_id`
### Workflow Execution Metrics
- `workflow_execution_total`: Total workflow executions
- Labels: `workflow_name`, `status` (success/failure)
- `activity_execution_total`: Total activity executions
- Labels: `activity_name`, `status` (success/failure)
### Training Metrics
- Training success/failure rates through notification system
- Model save performance metrics
- Experiment status tracking
- **RCE Drift Metrics**: Reduced Coulomb Energy (RCE) for drift detection
- `silverman_radius`: Optimal bandwidth for kernel density estimation
- `rce_reference`: RCE value for reference data
- `rce_current`: RCE value for current data
- `rce_drift`: Drift score between reference and current distributions
### Cleanup Metrics
- Cleanup execution success/failure rates
- Number of files deleted from MinIO
- Number of directories cleaned from local filesystem
- Cleanup duration and performance
## Configuration
### Environment Variables
| Variable | Description | Default | Required |
|----------|-------------|---------|----------|
| `TEMPORAL_HOST` | Temporal server address | `localhost:7233` | Yes |
| `TEMPORAL_NAMESPACE` | Temporal namespace | `model-manager` | No |
| `TEMPORAL_USE_TLS` | Enable TLS for Temporal connection | `false` | No |
| `TRAIN_TASK_QUEUE` | Task queue for training workflows | `train_model-queue` | No |
| `CLEANUP_TASK_QUEUE` | Task queue for cleanup workflows | `cleanup-queue` | No |
| `POSTGRES_HOST` | PostgreSQL hostname | `localhost` | Yes |
| `POSTGRES_PORT` | PostgreSQL port | `5432` | Yes |
| `POSTGRES_USER` | PostgreSQL username | `sientia` | Yes |
| `POSTGRES_PASSWORD` | PostgreSQL password | `sientia` | Yes |
| `POSTGRES_DBNAME` | PostgreSQL database | `sientia` | Yes |
| `POSTGRES_MIN_CONNECTIONS` | Minimum PostgreSQL connections | `5` | No |
| `POSTGRES_MAX_CONNECTIONS` | Maximum PostgreSQL connections | `20` | No |
| `MLFLOW_URL` | MLFlow server URL (full URL with protocol and port) | `http://localhost:5080` | Yes |
| `MLFLOW_USERNAME` | MLFlow username | `aignosi` | Yes |
| `MLFLOW_PASSWORD` | MLFlow password | `aignosi` | Yes |
| `MINIO_ENDPOINT_URL` | MinIO server endpoint | `http://minio.minio.svc.cluster.local:9000` | Yes |
| `MINIO_ACCESS_KEY` | MinIO access key | `minioadmin` | Yes |
| `MINIO_SECRET_KEY` | MinIO secret key | `minioadmin` | Yes |
| `MINIO_REGION` | MinIO region | `us-east-1` | No |
| `MINIO_USE_SSL` | Enable SSL for MinIO | `false` | No |
| `MINIO_MAX_RETRY_ATTEMPTS` | Maximum retry attempts | `3` | No |
| `MINIO_RETRY_MODE` | Retry mode (standard/adaptive) | `adaptive` | No |
| `MINIO_CONNECT_TIMEOUT` | Connection timeout (seconds) | `10` | No |
| `MINIO_READ_TIMEOUT` | Read timeout (seconds) | `60` | No |
| `MONGODB_URL` | MongoDB connection URI | `localhost:27018` | Yes |
| `MONGODB_USERNAME` | MongoDB username | `root` | Yes |
| `MONGODB_PASSWORD` | MongoDB password | `wKZDbMNU1c` | Yes |
| `MONGODB_DATABASE` | MongoDB database name | `sientia` | Yes |
| `MONGODB_TTL_INDEX_HOURS` | MongoDB TTL index hours | `1` | No |
| `CLEANUP_SCHEDULE_ID` | Cleanup schedule identifier | `cleanup-files-daily` | No |
| `CLEANUP_CRON` | Cleanup cron expression | `0 0 * * *` | No |
| `CLEANUP_TIMEZONE` | Cleanup schedule timezone | `UTC` | No |
| `CLEANUP_EXECUTION_TIMEOUT_HOURS` | Cleanup execution timeout | `1` | No |
| `CLEANUP_RETENTION_HOURS` | File retention period (hours) | `24` | No |
| `CLEANUP_DRY_RUN` | Dry-run mode (no actual deletion) | `false` | No |
| `MAX_KEYS_CLEANUP` | MinIO list operation page size | `1000` | No |
| `DEFAULT_CLEANUP_BUCKET` | Default bucket for cleanup | `model-training` | No |
| `LOG_LEVEL` | Application log level | `INFO` | No |
| `PROJECT_NAME` | Project name for metrics | `model-manager` | No |
| `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
| `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
| `POD_ID` | Kubernetes pod identifier | `None` | No |
| `EXTRA_PIP_REQUIREMENTS` | Extra pip requirements for MLFlow model serving | `None` | No |
#### Workflow Activity Timeouts
These timeouts control how long each activity in workflows can run before timing out. All values are in seconds.
**Training Workflow Timeouts:**
| Variable | Description | Default | Calculation Basis |
|----------|-------------|---------|-------------------|
| `TIMEOUT_VALIDATE_PARAMS` | Parameter validation timeout | `30` | Fast operation, no I/O |
| `TIMEOUT_TRAIN_MODEL` | Model training timeout | `2700` | Large dataset processing (45 min) |
| `TIMEOUT_DELETE_FILE` | Delete file from MinIO timeout | `120` | MinIO delete operation (2 min) |
| `TIMEOUT_UPDATE_DATABASE` | Database update timeout | `30` | PostgreSQL update query (30 sec) |
**Cleanup Workflow Timeouts:**
| Variable | Description | Default | Calculation Basis |
|----------|-------------|---------|-------------------|
| `TIMEOUT_CLEANUP_MINIO` | MinIO cleanup timeout | `300` | Scan and delete multiple files (5 min) |
| `TIMEOUT_CLEANUP_LOCAL` | Local cleanup timeout | `120` | Scan and delete directories (2 min) |
**Note**: These timeouts can be adjusted based on your infrastructure performance and file sizes. If you're processing files larger than 200MB or have slower network/compute resources, increase these values accordingly.
## Development
### Code Quality & Testing
The project maintains **100% code coverage** with comprehensive unit and integration tests.
#### Running Tests
```bash
# Run all tests with coverage
pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=xml --cov-report=html
# Run specific test file
pytest tests/activities/test_training.py -v
# Run with coverage visualization
pytest tests/ --cov=model_manager --cov-report=xml
# Then open htmlcov/index.html in browser
```
#### Validation Script
The `validate.sh` script runs all quality checks before commit:
```bash
./validate.sh
```
This script performs:
1. ✅ **Code Formatting** (Ruff)
2. ✅ **Code Linting** (Ruff)
3. ✅ **Type Checking** (mypy)
4. ✅ **Security Analysis** (Bandit)
5. ✅ **Unit Tests** (pytest with 80%+ coverage requirement)
#### Coverage Visualization
For real-time coverage feedback in VS Code/Windsurf:
1. **Install Coverage Gutters extension**
2. **Configure `.vscode/settings.json`**:
```json
{
"coverage-gutters.coverageBaseDir": "${workspaceFolder}",
"coverage-gutters.coverageFileNames": ["coverage.xml"],
"coverage-gutters.showLineCoverage": true,
"coverage-gutters.showRulerCoverage": true
}
```
3. **Run tests to generate coverage**:
```bash
pytest tests/ --cov=model_manager --cov-report=xml
```
4. **Activate Coverage Gutters**: Press `Ctrl+Shift+7` (or `Cmd+Shift+7` on Mac)
#### Automated Versioning
The project uses semantic versioning based on branch patterns:
| Branch Pattern | Version Change | Example |
|---------------|----------------|---------|
| `release/*` | Major version bump | `2.0.0` |
| `feature/*` | Minor version bump | `1.2.0` |
| `fix/*` | Patch version bump | `1.1.3` |
| `rc/*` | Release candidate | `1.1.2-rc2` |
Version is calculated automatically in the CI/CD pipeline and passed to SonarQube.
### Project Structure
```
sientia-dataops-model-manager/
├── model_manager/ # Main application package
│ ├── activities/ # Temporal activity implementations
│ │ ├── __init__.py
│ │ ├── activities.py # Main activities orchestrator (combines all activities)
│ │ ├── experiment_tracking.py # Experiment status tracking and database operations
│ │ ├── training.py # ML model training operations (includes MLFlow & MinIO)
│ │ └── cleanup.py # File and directory cleanup operations
│ ├── workflows/ # Temporal workflow definitions
│ │ ├── __init__.py
│ │ ├── train_model.py # Complete ML model training workflow
│ │ └── cleanup_files.py # Automated file cleanup workflow
│ ├── schedules/ # Temporal schedule configurations
│ │ ├── __init__.py
│ │ └── cleanup_schedule.py # Cleanup schedule creation and management
│ ├── worker/ # Worker implementation
│ │ ├── __init__.py
│ │ └── worker.py # Main worker orchestrator (Temporal client setup)
│ ├── utils/ # Utility functions and helpers
│ │ ├── __init__.py
│ │ ├── connectors_config.py # Environment-based configuration builders
│ │ ├── exceptions.py # Custom exception definitions
│ │ ├── logger_helper.py # Logger initialization utilities
│ │ ├── models/ # Data models and schemas
│ │ │ ├── __init__.py
│ │ │ ├── train_model_params.py # Training parameters model
│ │ │ ├── train_model_result.py # Training result model
│ │ │ └── experiment_status.py # Experiment status enum
│ │ └── repository/ # Data access layer
│ │ ├── training_repository.py # Training business logic
│ │ ├── model_repository.py # MLFlow artifact management
│ │ └── storage_repository.py # MinIO storage operations
│ ├── sientia/ # Sientia-specific implementations
│ │ ├── __init__.py
│ │ ├── exceptions.py # Custom exceptions
│ │ ├── metrics.py # Business metrics (includes RCE drift detection)
│ │ ├── models.py # ML model implementations (Linear & Polynomial Regression)
│ │ ├── model_serving.py # Model serving utilities
│ │ ├── reports.py # Report generation
│ │ └── utils.py # Utility functions
│ ├── reports/ # Report templates and temporary files
│ │ ├── header.html # HTML report header template
│ │ └── temp/ # Temporary report files (cleaned up automatically)
│ ├── metrics.py # Prometheus metrics definitions
│ └── __init__.py
├── scripts/ # Test and utility scripts
│ ├── run_cleanup_test.py # Manual cleanup workflow test
│ └── run_training_test.py # Training test with scenario support (--all for batch)
├── tests/ # Test suite
│ ├── activities/ # Activity tests
│ ├── workflows/ # Workflow tests
│ ├── utils/ # Utility tests
│ ├── worker/ # Worker tests
│ ├── schedules/ # Schedule tests
│ └── sientia/ # Sientia module tests
├── docs/ # Documentation and test data
│ ├── test-scenarios/ # JSON test scenario files for integration tests
│ └── test-model-data.csv # Sample CSV data for testing
├── .github/workflows/ # CI/CD workflows
│ ├── quality-gate.yml # PR quality checks
│ └── deploy.yml # Deployment workflow
├── Dockerfile # Container image definition
├── values.yaml # Helm chart values
├── pyproject.toml # Project configuration
├── requirements.txt # Production dependencies
├── requirements-dev.txt # Development dependencies
├── validate.sh # Code quality validation script
├── run_local.sh # Local execution script
└── README.md # This file
```
### Adding New Features
1. **Follow Temporal patterns** for new workflows and activities
2. **Add comprehensive docstrings** for all public methods
3. **Include Prometheus metrics** for monitoring
4. **Add unit tests** for new functionality (maintain 80%+ coverage)
5. **Run validation script** (`./validate.sh`) before committing
6. **Update this README** with new features and configuration
### Test Coverage Guidelines
- **Minimum coverage**: 80% (enforced by CI/CD)
- **Current coverage**: 100% 🎯
- **Test all branches**: Use Coverage Gutters to identify uncovered lines
- **Mock external dependencies**: Use `unittest.mock` for external services
- **Async testing**: Use `pytest-asyncio` for async activities and workflows
- **Test structure**:
```
tests/
├── activities/ # Activity tests
├── workflows/ # Workflow tests
├── utils/ # Utility tests
├── worker/ # Worker tests
├── schedules/ # Schedule tests
└── sientia/ # Sientia module tests
```
## Troubleshooting
### Common Issues
1. **Temporal Connection Failures**
- Verify Temporal server is running and accessible
- Check namespace configuration and permissions
- Review server logs for connection issues
2. **MLFlow Connection Issues**
- Verify MLFlow server is running and accessible
- Check authentication credentials and permissions
- Ensure model names and versions exist
3. **Database Connection Issues**
- Verify PostgreSQL service is running
- Check connection credentials and network access
- Ensure proper connection pool configuration
4. **Workflow Execution Failures**
- Review activity error logs and notifications
- Check training parameter validation errors
- Verify input data format and required fields
### Debug Mode
Enable debug logging by setting the log level:
```bash
export LOG_LEVEL=DEBUG
```
## Performance Tuning
### Key Parameters
- **Worker Concurrency**: Adjust `max_concurrent_workflow_tasks` and `max_concurrent_activities`
- Training workflows: 10 concurrent tasks/activities
- Cleanup workflows: 20 concurrent tasks/activities
- **Connection Pools**: Optimize database connection pool sizes
- **Model Retention**: Configure MLFlow model retention based on requirements
- **Batch Sizes**: Adjust data processing batch sizes for optimal throughput
- **Cleanup Performance**: Tune `MAX_KEYS_CLEANUP` for MinIO list operation page size
### Scaling Considerations
- **Horizontal Scaling**: Deploy multiple worker instances
- **Task Queue Distribution**: Two dedicated task queues for workflow isolation
- `train_model-queue`: Training workflows
- `cleanup-queue`: Cleanup workflows
- **Database Performance**: Optimize indexes and connection pooling
- **MLFlow Performance**: Configure appropriate model serving resources
- **Storage Management**: Adjust cleanup retention period based on storage capacity and costs
## Contributing
1. Fork the repository
2. Create a feature branch
3. Make your changes with comprehensive testing
4. Update documentation and docstrings
5. Submit a pull request
### Code Quality Standards
- Follow PEP 8 style guidelines
- Include comprehensive docstrings for all public methods
- Maintain test coverage above 80%
- Use type hints where appropriate
- Follow Temporal.io best practices
## License
This project is licensed under the terms specified in the LICENSE file.
## Support
For support and questions:
- Check the troubleshooting section above
- Review the metrics and logs for error patterns
- Open an issue in the project repository
- Contact the development team
## Local GitHub Actions Testing (act)
### What is act?
[act](https://nektosact.com/) is a tool that allows you to run GitHub Actions locally using Docker. This is useful for:
- **Testing workflows** before pushing to the repository
- **Debugging issues** in workflows without creating commits
- **Speeding up development** by avoiding push/wait/check cycles
- **Saving GitHub Actions minutes** during development
### Installation
#### Prerequisites
- Docker installed and running
- Go (for installation via `go install`)
#### Installation Steps
```bash
# 1. Update packages
sudo apt-get update
# 2. Install Go (if not already installed)
sudo apt-get install golang
# 3. Install act
go install github.com/nektos/act@latest
# 4. Add Go bin to PATH
echo 'export PATH="$PATH:$HOME/go/bin"' >> ~/.bashrc
source ~/.bashrc
# 5. Verify installation
act --version
```
On first run, `act` will ask which Docker image to use:
- **Large** (~17GB): Full image, compatible with almost all actions
- **Medium** (~500MB): Balanced image, compatible with most actions ✅ Recommended
- **Micro** (<200MB): Minimal image, Node.js only
### Configuration
#### `.secrets` File
Create a `.secrets` file in the project root to store tokens and credentials:
```bash
# .secrets
GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxx
SONAR_TOKEN=sqp_xxxxxxxxxxxxxxxxxxxx
SONAR_HOST_URL=https://sonarqube.example.com
CI_DEPS_APP_ID=123456
CI_DEPS_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"
```
> ⚠️ **Important**: The `.secrets` file is already in `.gitignore`. Never commit this file!
#### `.event.json` File
Create a `.event.json` file to simulate GitHub events (e.g., pull request):
```json
{
"pull_request": {
"head": {
"ref": "feature/my-feature"
},
"number": 1
}
}
```
> ⚠️ **Important**: The `.event.json` file is already in `.gitignore`. Never commit this file!
### Usage
#### List Available Jobs
```bash
act -l
```
This command lists all available workflows and jobs in the repository.
#### Run Quality Gate Locally
```bash
act pull_request -j quality-gate \
--secret-file .secrets \
--env SONAR_SCANNER_OPTS="-Dsonar.ci.autoconfig.disabled=true" \
--eventpath .event.json
```
#### Run Deploy Workflow with Local Repository
If you have workflows that reference external repositories (e.g., reusable workflows), you can map them locally:
```bash
act pull_request \
-e .event.json \
--secret-file .secrets \
-W .github/workflows/deploy.yml \
--local-repository Aignosi/github_workflow_templates=/path/to/local/github_workflow_templates \
--container-daemon-socket /var/run/docker.sock \
--container-options "--user $(id -u):$(id -g)"
```
#### Docker Socket Permissions
If you encounter permission issues with Docker socket:
```bash
# Grant temporary access to Docker socket
sudo chmod 666 /var/run/docker.sock
# Fix file ownership after running act (if needed)
sudo chown -R $USER:$USER /path/to/project
```
### Command Reference
#### Command Parameters
| Parameter | Description |
|-----------|-------------|
| `pull_request` | Event type to simulate (can be `push`, `pull_request`, `workflow_dispatch`, etc.) |
| `-j quality-gate` | Specific job name to execute (use `act -l` to see available jobs) |
| `--secret-file .secrets` | File containing secrets (tokens, credentials) |
| `--env VAR=value` | Sets environment variables for execution |
| `--eventpath .event.json` | JSON file with the simulated event payload |
#### Special Parameter: `SONAR_SCANNER_OPTS`
```bash
--env SONAR_SCANNER_OPTS="-Dsonar.ci.autoconfig.disabled=true"
```
This parameter is required because SonarQube tries to automatically detect the CI environment. When running locally with `act`, the complete GitHub Actions context is not available, causing errors. The `-Dsonar.ci.autoconfig.disabled=true` flag disables this automatic detection.
#### Other Useful Commands
```bash
# List all jobs
act -l
# Run with verbose output
act pull_request -j quality-gate --secret-file .secrets -v
# Run a push event
act push -j build --secret-file .secrets
# Use a specific Docker image
act -P ubuntu-latest=catthehacker/ubuntu:act-latest
# Dry-run (doesn't execute, only shows what would be done)
act -n
```
#### Troubleshooting
| Problem | Solution |
|---------|----------|
| `SyntaxError: Unexpected end of JSON input` | Check if `.event.json` is properly formatted |
| `NullPointerException` in SonarQube | Add `--env SONAR_SCANNER_OPTS="-Dsonar.ci.autoconfig.disabled=true"` |
| `Not Found` when accessing GitHub API | Check if `GITHUB_TOKEN` in `.secrets` is valid |
| Job not found | Use `act -l` to see the correct job names |
## Docker
### Create image
```bash
$ docker build --ssh default --no-cache --progress=plain -t aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0 .
```
### Create container
```bash
$ docker run --env-file .env --network="host" --name sientia-dataops-model-manager -d aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0
$ docker logs -f sientia-dataops-model-manager
```
### Login using access token
```bash
$ docker login -u <username> -p <access-token> aignosi.azurecr.io
```
### Push image to repository
```bash
$ docker push aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0
```
## Helm Chart
### Reference
https://aignosi-wiki.atlassian.net/wiki/spaces/IT1/pages/274563074/Como+utilizar+o+Helm+Repo+Privado
### Add Helm Chart repository
```bash
$ helm repo add sientia \
https://raw.githubusercontent.com/Aignosi/sientia-dataops-helm-repo/refs/heads/main/ \
--username $GITHUB_USER \
--password $GITHUB_PASS
# Update repository
$ helm repo update
# List repositories
$ helm repo list
# List versions of a specific chart
$ helm search repo sientia --versions
# List all charts available
$ helm search repo sientia
# List chart details
$ helm show all sientia/sientia-module
# Download chart to current directory
$ helm pull sientia/sientia-module --version 0.6.0 --untar
# Remove chart directory
$ rm -rf sientia-module
```
### Helm Install
```shell
$ helm upgrade --install sientia-dataops-model-manager sientia/sientia-module -n sientia --create-namespace -f ./values.yaml --version 0.6.0
```
### Uninstall Helm Chart
```shell
$ helm uninstall sientia-dataops-model-manager -n sientia
```
---
**Note**: The Model Manager system is designed for production use in industrial ML environments. Ensure proper security configuration and network isolation for production deployments.
Reference in New Issue View Git Blame Copy Permalink